> For the complete documentation index, see [llms.txt](https://docs.zus.network/zus-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zus.network/zus-docs/webapps/blimp.md). # Blimp ## Introduction Blimp sits as a sidecar to your pipeline and provides a smart, ACID cache with an autonomous materialized view — keeping your AI/ML context fresh, without you migrating data or changing your engines. Point your engines at it, simplify your pipeline, and lower cost. Point your existing tools (Spark, Trino, Snowflake, Ray, PyTorch, …) and Iceberg catalog at a Blimp node. Blimp takes your query, materializes the right Iceberg views automatically, and serves reads from a high-throughput NVMe cache. * **Open & standard** — Iceberg tables, S3-compatible endpoints, Glue / Polaris / Nessie-compatible catalog. * **Fully ACID** — every write commits as an atomic data entity; no orphaned files, no partial reads. * **Yours end-to-end** — deploys on any server or cloud VM; you keep ownership of the data, keys, and bucket. ## Features * **Autonomous materialized views** — Blimp takes your query, then authors and materializes Iceberg materialized views (MVs) for you and keeps them fresh via CDC delta-merges in a few seconds (3–4s). Materialized queries return **sub-second** — and because the MVs are standard Iceberg tables, any existing pipeline gets the same sub-second reads through its **current query engine** (Trino, Spark, Snowflake, DuckDB, …). No SQL rewrites, no schema changes. * **Storage Cache** — an **efficient cache per core**: an NVMe cache exposed over fast S3, NFS and Fuse/Mountpoint-S3 endpoints, serving fresh data for inference and keeping GPUs fed for training with no cross-region latency. On a miss the Gateway pulls the object from your origin once, stores it on the node blobbers, and serves it; every repeat read is then served from the blobbers. Fewer cores for the same throughput means lower cost and a greener footprint, and it scales up or down with any workload — analytics, inference, or training. * **CDC engine** — a change-data-capture pipeline that tracks Iceberg snapshots and keeps your MVs and cached data consistent with the source tables, so optimizer MVs refresh automatically as your data changes. * **Storage benchmarks** — built-in warp and MLPerf benchmarks let you measure a node's throughput. * **Unlimited nodes** — spin up as many independent nodes as you need, each on your own infrastructure, and add more to scale throughput. #### Blimp Fleet Storage **One S3 endpoint for your whole fleet** Every node in your account is reachable behind a single hostname — `https://fleet-.blimp.software:9443` — with one shared access/secret key. Point any S3 tool (`aws-cli`, `mc`, `boto3`, `warp`) or a `mount-s3` FUSE client at it and you get your entire namespace, no matter which node stores each object. Data is deduplicated fleet-wide (identical content kept once across all nodes), and the endpoint is round-robin + health-checked, so if a node goes down traffic moves to a healthy one automatically. **Fleet: aggregate bandwidth behind one URL** A fleet is a set of gateway nodes that act as one storage service. Every node is stateless and holds no exclusive control plane — you get one credential and one URL (`fleet-.blimp.software`) that every node honors. The URL is a round-robin DNS name over all live nodes, so concurrent clients spread across the fleet and their throughput adds up — each node you add contributes its full NIC and disk bandwidth to the same namespace. Add a node and aggregate bandwidth grows linearly; destroy one and it drops out of the URL within \~2 minutes automatically, no reconfiguration. **Dedup: store once, across the whole fleet** Objects are content-addressed by a hash of their bytes. Before storing, the fleet checks a central index: if those exact bytes already live on any node, the new upload is bound to the existing copy instead of re-stored — identical content is kept once, fleet-wide, regardless of which node or object name it arrives under. A node that holds a name but not the bytes fetches them transparently from the node that does, so reads work from any node. **Encryption (optional — default OFF)** Turn on AES-256-GCM to encrypt object bytes at rest. Keys are derived per object; enabling it costs roughly 9% on GET throughput and is off by default so unencrypted workloads pay nothing. **ACID strong consistency (optional — default OFF)** Turn on ACID mode for torn-read protection from orphan and corrupted files: concurrent readers and writers see a whole, consistent object (no mix of old and new erasure-coded shards). It costs roughly 6% on GET throughput and is off by default — enable it when strict read-after-write consistency matters more than peak read speed. **Immutability (optional — default OFF)** Turn on to lock an allocation's object version history on-chain — users can still upload, list, download, and share files and folders, but can't delete, rename, move, copy, or update an existing file. Unlike other vendors' immutability, Blimp anchors object version history and access grants to a distributed ledger — a retroactive modification to a version's history is cryptographically detectable, rather than relying solely on access-control enforcement. Off by default; enable per allocation when compliance requires tamper-evident history. **Unified view + per-node view** Files are browsable two ways: Per-node view (default) shows only the objects whose bytes physically live on the node you're looking at; the Global View link on the nav bar switches to the unified fleet namespace — every object across all nodes, deduped, as one list. **Carousel + broad file rendering** The browser file viewer previews most common types inline — images, PDFs, text/code, audio and video — and a carousel lets you swipe through a folder's files without leaving the viewer. Because bytes are served directly from the gateway over HTTPS, previews render with the browser's own players/viewers (no external path). ## Architecture A Blimp deployment has two planes. **Control plane (Manager)** — a multi-tenant, highly-available service that authenticates users, provisions nodes onto the server or cloud instance you point it at (cloud or on-prem), and proxies privileged operations. Application calls carry your login token in an `x-app-id-token` header; the node's gateway admin token never leaves the server side (browsers use short-lived SigV4 credentials for data operations). The Manager also owns the content-dedup index and exposes a REST control-plane API used by both the web app and your own automation. **Data plane (the node)** — each node runs entirely on your infrastructure: | Component | Role | Endpoint | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | **Gateway** | S3-compatible gateway + DuckDB query optimizer; authors/materializes MVs; erasure-codes writes and reconstructs reads | S3 `:9000`, TLS S3 `:9443` | | **Blobbers** | Erasure-coded storage on local NVMe; hold content-addressed, deduplicated shards — MVs, and query results | internal (`:5051`) | | **Iceberg catalog (Nessie)** | Catalog for the materialized views; reached read-only through the gateway's Iceberg proxy | proxy `:19123` → Nessie `:19120` | | **CDC / refresh** | Tracks Iceberg snapshots and refreshes dependent MVs and cached data (snapshot tracker + optimizer cycle) | internal | On write (PUT) the gateway erasure-codes the object and fans out shards to every blobber; identical content deduplicates to a single physical copy across the namespace (claim/commit against the Manager's dedup index). On read (GET) it reconstructs from any D of the N shards — issuing D+1 requests and using the fastest D — so the gateway's vCPU and network scale with the data-shard count while blobbers stay uniform. #### Auto-configuration — any hardware, no manual shard math Blimp deploys on any server or cloud instance you point it at — bare-metal, VPC, or cloud VM — and configures itself from what it finds: * **Disks → blobbers.** Each blobber is matched to one disk, up to 12 disks (blobbers) per node. * **Cores.** The deployer reserves 1–2 cores per blobber; the rest goes to the gateway, so gateway throughput scales with whatever's left over on the box. * **Erasure coding.** From the resulting blobber count, Blimp picks a parity width between 1 and 2 — the rest of the disks become data shards — chosen to maximize throughput for that disk count. * **Mixed hardware.** Servers in the same fleet don't need to match — mix and match freely; each node configures independently for its own hardware. No manual shard math, no fixed instance shape — the node fits whatever hardware you gave it. ![Auto-configuration — from raw disks and cores to a running node: reserve cores per blobber, match blobbers to disks, pick an erasure-coding ratio](/files/CuVz13N30R2i4KJhJYUi) #### Query Optimizer — Iceberg MVs authored on demand A query registers its source table into the Iceberg catalog, Harness/LLM authors the matching MV shape, and the Gateway materializes it by pulling straight from S3 and caching the result on the blobbers — Blimp then returns a link to the result. CDC watches the source afterward and keeps the MV fresh as it changes. ![Query Optimizer — Harness/LLM-authored Iceberg MV flow, kept fresh by CDC](/files/SvbKQHwWnnhNWZhwF9GN) ## ACID — consistent under load Every write to a Blimp node is erasure-coded across independent blobbers and commits as a single atomic unit. If a write is interrupted, the write either lands completely or not at all. Reads stay consistent under concurrent load too: a read racing an overwrite never returns a stale or torn mix of old and new bytes — whether you reach the node through the raw gateway S3 API or the mounted filesystem. This is validated with the same linearizability model-checker used in Jepsen-style distributed-systems testing, run against both read paths under both single-writer and multi-writer load — see run-blimp's [`blimp --acid`](https://github.com/0chain/run-blimp#d--testing-acid-blimp---acid). The result: you don't have to reason about orphaned files, torn writes, silent corruption, or lost transactions — the node guarantees them away. ## Security — blockchain zero-trust Blimp is **breach-resistant by architecture**. It runs on **zero standing trust** — nothing in the pipeline is trusted by default. Most breaches start with a spoofed key or a stolen identity, an attacker impersonating a trusted entity; Blimp anchors every identity on the blockchain, so a leaked key alone can't be replayed to forge an identity. Every message between entities is also cryptographically signed with a nonce, closing off replay and man-in-the-middle attacks. * **Blockchain-anchored, zero-trust identity** — every entity (gateway, blobbers, clients) has an identity anchored on the blockchain, so a leaked or stolen key can't on its own be used to impersonate it. No one can spoof a key or steal an identity to impersonate an entity in your pipeline. * **Signed messages between entities** — every message between the gateway, blobbers, and clients is cryptographically signed. No unsigned traffic, no shared secrets, no implicit trust anywhere in the system. Each signature is nonce-bound, so a captured message can't be replayed and an attacker on the network path can't tamper with or inject traffic — no replay, no man-in-the-middle. * **ACID data integrity** — every write commits as an atomic data entity: no partial writes, no torn reads, no orphaned files. Your AI inference always reads complete, consistent data. * **Tamper-evident immutability (optional)** — lock an allocation's object version history on-chain; a retroactive modification to a version's history is cryptographically detectable rather than relying solely on access-control enforcement. The permission state itself — e.g. an allocation locked to upload, share, and download only, with delete, rename, move, copy, and update disabled — is stamped on the ledger, not just enforced client-side. Unlike other vendors' immutability, Blimp anchors both the version history and access grants to a distributed ledger. * **Customer-owned infrastructure** — every node runs on your infrastructure, any server or cloud; your engines, catalog, and data never leave it. ## Quickstart — try Blimp end to end in \~15 minutes The fastest way to see what Blimp does: sign up, launch a node on a server or cloud instance. 1. **Sign up with Gmail.** Go to [blimp.software](https://blimp.software) and continue with Google — a key is created for you automatically. 2. **Launch a node.** On the empty Home screen click **Launch Node**, enter SSH credentials of your server or cloud VM (with volumes attached) — you can get started with as little as 4 vCPU and a single 100 GB attached disk. The deployer inspects the box and provisions a storage node automatically — no manual shard math, no fixed instance shape. 3. **Wait \~4–5 minutes** while the node forms — watch the deployment progress bar. When it turns **green**, you'll see two demo areas, the **Query Optimizer** and the **Storage Cache**. 4. **Run the SF1 query.** In **Query Optimizer → Benchmark (TPC-DS)**, click **Run TPC-DS query** against the self-seeded SF1 (\~1 GB) dataset — note the author / materialize times; a warm re-run is sub-second. Try with a different variant or query in the TPC-DS test profile. 5. **Check the Storage Cache.** Open **Storage Cache → Benchmarks** and run **warp** or **MLPerf** to measure the node's S3 and AI data-loader throughput tested internally on the gateway — each test takes 3-6 minutes, with MLPerf the longest as it builds and writes the data first to conduct the test. ## Storage Performance Measure how much data the **gateway** can pull from the **blobbers** and serve. Open a node → **Storage Cache** → **Benchmarks**, choose a tool + concurrency / object size, and run — each streams a live log and a summary: * **warp** — S3 throughput against the gateway's S3 endpoint (S3:9000). Parallel GET/PUT, reporting aggregate MB/s + ops/s — the S3-protocol data rate served from the blobbers. * **MLPerf Storage** — an ML training data-loader benchmark against the node (samples/s, achieved utilization) — an end-to-end measure of whether the cache keeps GPUs fed under an AI workload. The cache product can be used for **high-compute analytics, datalake compaction, and AI inference and training** — anywhere you need speed of execution while lowering CPU and GPU compute cost. It runs on commodity network-optimized instances and delivers an efficient cache per core — fewer cores for the same throughput, which is both cheaper and greener. Expected performance (measured on live nodes): | | **6/2 node** | | -------------------------------- | --------------------------- | | Gateway | 8 vCPU | | Blobber nodes (8) | 1 vCPU | | Fault tolerance | 6 data + 2 parity | | Capacity | 6 x disk size | | TTFB (p50 / p99) | 1-2 ms | | S3 — warp | 1.3 GB/s PUT · 4.4 GB/s GET | | AI inference & training — MLPerf | 1.1 GB/s at 97.5% AU | | GET throughput / vCPU | \~280 MB/s | Results persist on the gateway with hosted log links, so a run survives a browser refresh — compare tools and concurrency levels to find the node's ceiling. ## Query Mechanics ### How it works We take the original query, strip its literals, and generalise the grain so the MV isn't tied to one set of filter values — `d_year = 2000` becomes `d_year` in the `GROUP BY`. That materialised view holds pre-aggregated rows at that grain. The rewrite then re-applies the literals on top of the MV and joins whatever dimensions it still needs, producing the original query's exact answer — verified by row-hashing the original against the rewrite so a wrong MV is rejected rather than served. Three constraints make it actually work: * **Measures must be re-aggregatable.** SUM and COUNT roll up; AVG doesn't. An MV storing an average is unusable, so we store `SUM(x)` and `COUNT(x)` per measure and let the rewrite divide. * **Refresh is O(delta), not O(fact).** When new rows land, we aggregate only the delta at the same grain and write it as a separate parquet part beside the MV; readers fold base + delta at query time. The base is never rewritten. This holds only when each UNION branch of the recipe reads exactly one fact — with k facts in a branch, the algebra needs 2^k−1 inclusion-exclusion terms and the unchanged ones bind to the full table, so the "merge" silently becomes a full scan. * **Grain must be scale-invariant.** A grain on a bounded column (`ss_quantity`, 100 values) gives the same MV at SF1 and SF1000. A grain on a surrogate key (`ss_customer_sk`) is 100k rows at SF1 and 12M at SF1000 — which is how q4 ends up with a 53M-row MV that costs more to read than the query it replaces. Example — TPC-DS **q9**, removing non-linearity from a query so its MV can delta-merge. Measured on a single 16 vCPU node against TPC-DS SF1000 — `store_sales` 2,879,987,999 rows. This is the clearest example of the technique, because q9 packs three different kinds of non-linearity into one query and all three have to come out before the view can be maintained incrementally. It asks five independent questions of the same table: each is a `CASE` whose branch is chosen by a count threshold, and whose value is an average over a quantity band. **1 · Original query** ```sql CASE WHEN (SELECT count(*) FROM store_sales WHERE ss_quantity BETWEEN 1 AND 20) > 74129 THEN (SELECT avg(ss_ext_discount_amt) FROM store_sales WHERE ss_quantity BETWEEN 1 AND 20) ELSE (SELECT avg(ss_net_paid) FROM store_sales WHERE ss_quantity BETWEEN 1 AND 20) END bucket1, ... the same shape for buckets 2-5, bands 21-40, 41-60, 61-80, 81-100 ... FROM reason WHERE r_reason_sk = 1 ``` Three things here cannot live in a materialized view: | construct | why it breaks incremental maintenance | | ------------------ | ----------------------------------------------------------------------- | | `avg(x)` | not additive — `avg(A ∪ B)` is not derivable from `avg(A)` and `avg(B)` | | `CASE ... > 74129` | a literal threshold; baking it in fixes the view to one binding | | `BETWEEN 1 AND 20` | a literal band; same problem, five times over | **2 · The linear core** — MV definition Strip all three out and what remains is a plain SPJG aggregate — one table, one grouping column, additive measures only: ```sql SELECT ss_quantity, COUNT(*) AS cnt, SUM(ss_ext_discount_amt) AS sum_disc, COUNT(ss_ext_discount_amt) AS cnt_disc, SUM(ss_net_paid) AS sum_net, COUNT(ss_net_paid) AS cnt_net FROM store_sales GROUP BY ss_quantity ``` 101 rows. Every non-linear construct is gone. Note what replaced `avg`: a **SUM and a COUNT stored separately**, so the division can happen later. The `COUNT` is of the measure expression, not `COUNT(*)`. That distinction is not cosmetic — on q14 the same recipe used `COUNT(*)` as an AVG denominator when the true one excluded NULL factors, and the two differed by **129,390 rows** at SF1. The row-hash verifier caught it; a stored average would have hidden it. **3 · The rewrite** — where the non-linearity goes ```sql SELECT CASE WHEN (SELECT SUM(cnt) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 1 AND 20) > 74129 THEN (SELECT SUM(sum_disc)/SUM(cnt_disc) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 1 AND 20) ELSE (SELECT SUM(sum_net)/SUM(cnt_net) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 1 AND 20) END bucket1, CASE WHEN (SELECT SUM(cnt) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 21 AND 40) > 122840 THEN (SELECT SUM(sum_disc)/SUM(cnt_disc) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 21 AND 40) ELSE (SELECT SUM(sum_net)/SUM(cnt_net) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 21 AND 40) END bucket2, CASE WHEN (SELECT SUM(cnt) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 41 AND 60) > 56580 THEN (SELECT SUM(sum_disc)/SUM(cnt_disc) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 41 AND 60) ELSE (SELECT SUM(sum_net)/SUM(cnt_net) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 41 AND 60) END bucket3, CASE WHEN (SELECT SUM(cnt) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 61 AND 80) > 10097 THEN (SELECT SUM(sum_disc)/SUM(cnt_disc) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 61 AND 80) ELSE (SELECT SUM(sum_net)/SUM(cnt_net) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 61 AND 80) END bucket4, CASE WHEN (SELECT SUM(cnt) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 81 AND 100) > 165306 THEN (SELECT SUM(sum_disc)/SUM(cnt_disc) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 81 AND 100) ELSE (SELECT SUM(sum_net)/SUM(cnt_net) FROM tpcds_mv.store_sales_qty_rollup WHERE ss_quantity BETWEEN 81 AND 100) END bucket5 FROM (SELECT 1) dummy WHERE 1=1 AND (SELECT r_reason_sk FROM reason WHERE r_reason_sk = 1) = 1 ``` ## Performance Comparison A Blimp node is in its own category — a query *and* cache optimizer with unparalleled cost efficiency through liquid scaling of data acceleration based on demand, a uniquely secure data environment, easy scalability on the cloud, and full ACID compliance. It also holds up directly against pure-play systems: more than 3× the cache throughput per core of VAST Data, and faster query performance than StarRocks on TPC-DS (Iceberg). | Cache throughput | Cores | GB/s | GB/s per core | | ------------------------------------------------------------------------------------------------------------- | ----- | ---- | ------------- | | Blimp | 8 | 4.4 | 0.55 | | [VAST Data](https://kb.vastdata.com/docs/s3-benchmarking-using-elbencho#choosing-the-right-number-of-clients) | 64 | 10 | 0.16 | | [Alluxio](https://documentation.alluxio.io/ee-ai-en/benchmark/s3-api#id-6-node-httpbench-on-aws) | 72 | 11 | 0.15 | Sources: Alluxio 6-node HTTPBench on AWS (single client/worker pair, peak throughput, 72-vCPU c5n.18xlarge worker) · VAST Data S3 benchmarking guide (per-CNode figure). TPC-DS — StarRocks (Iceberg catalog) vs Blimp, total query time: | Query | StarRocks — Iceberg | Blimp | | ----- | ------------------- | ------ | | Q09 | 23.2 s | 0.83 s | | Q88 | 19.5 s | 1.14 s | | Q04 | 16.7 s | 8.44 s | | Q14 | 15.3 s | 2.71 s | | Q24 | 11.9 s | 5.74 s | | Q59 | 3.2 s | 0.90 s | | Q47 | 3.1 s | 0.93 s | | Q13 | 2.2 s | 1.08 s | Source: [StarRocks TPC-DS benchmark](https://docs.starrocks.io/docs/benchmarking/TPC_DS_Benchmark/#test-results) — Iceberg catalog results, 1 FE + 4 BE × m6id.4xlarge (16 vCPU each). Blimp — measured on the 6/2 node above. ## Production setup - use run-blimp repo The production paths mirror the [**run-blimp**](https://github.com/0chain/run-blimp) client, which is a repo to automate testing out the Blimp node for cache and query performance from your app hosted on a machine that's close to the node. Test the node end-to-end with the **run-blimp** repo's `blimp` CLI: `blimp --setup` wires the kit to your data in about 10-15 minutes, `blimp --query` runs the query-optimizer + CDC suite (author → append → delta-merge) in about 5 minutes, and `blimp --storage` runs the storage/cache suite in about 30 minutes. Add `--verify` to `blimp --query` to row-hash verify each result against a full DuckDB recompute of the source. You can use the scripts in **run-blimp** to integrate with your app or orchestrator and scale your data pipeline's performance. The node is a **liquid** resource: add TB/s of throughput and thousands of queries per second by spawning nodes on demand. #### Production query — your Iceberg 1. **Connect your Iceberg and S3.** **run-blimp** interactively automates this setup — run `blimp --setup`. 2. **Run a query.** Enter SQL via node panel UI or script and click **▶ Run Production Query**. Blimp pulls the source into the node's blobbers, materializes the MV + result, and returns hosted viewer URLs. After that, your Iceberg commit trigger drives CDC and auto-refreshes the MV. 3. **Govern access.** Each node has CIDR allow-lists for its Iceberg catalog and S3 gateway endpoints — restrict them to the networks that should reach the node. Drive the same prod query from scripts and dashboards with the control-plane API — see **API & integration** below. Query performance is largely dependent on the size of the dataset, so the gateway needs to scale with capacity demand. For example, 8 vCPU is adequate for SF1 (\~1 GB); SF10 and SF100 need 16 vCPU or larger; and a SF1000 dataset starts at 32 vCPU. #### Production cache — your S3 1. **Point the cache at your S3.** Open a node → **Storage Cache** → **Production (your S3 endpoint)** and supply your endpoint + bucket. 2. **Govern access.** Lock every endpoint (S3 `:9000`) to your networks with the node CIDR allow-lists. Wire the same data staging into your own scripts and data loaders — see **API & integration** below. ## API & integration Two things are worth wiring into your own scripts, dashboards, and ML jobs: **running optimized queries on your data**, and **reading your data through the node cache**. (The demo and benchmark buttons are app-only — nothing to integrate there.) ### 1 · Run optimized queries on your Iceberg ![Query integration — A: the managed prod-query API authors + materializes an MV and returns hosted URLs; B: your own engine reads the MVs directly via the Iceberg MV endpoint (:19123).](/files/yK1XHjTtpPwtDDQxLkCL) POST your SQL to the control-plane API with your login token in the `x-app-id-token` header. The gateway ingests only the tables your SQL touches (in parallel), authors + materializes a view, and returns hosted result/MV URLs; a warm re-run reuses the MV and is sub-second. ```bash # enable once (one-way) curl -X POST https://datalake.blimp.software/api/v1/clusters/optimizer/prod-query/enable \ -H 'x-app-id-token: ' -H 'Content-Type: application/json' \ -d '{"cluster_id":""}' # run a query — scope the SQL to the tables you actually need curl -X POST https://datalake.blimp.software/api/v1/clusters/optimizer/prod-query \ -H 'x-app-id-token: ' -H 'Content-Type: application/json' \ -d '{ "cluster_id":"", "iceberg_url":"https://", "warehouse":"s3://", "s3_endpoint":"https://", "s3_key":"", "s3_secret":"", "namespace":"", "sql":"SELECT d_year, COUNT(*) FROM date_dim GROUP BY d_year" }' ``` Prefer your own engine? Point Trino / Spark / DuckDB at the read-only **Iceberg MV endpoint** `http://:19123/iceberg` (warehouse `mv`) and query the materialized views directly. ### 2 · Stage your data on the node for AI inference & training ![Data staging — your app writes a dataset onto the node's blobbers once via the Gateway; every read for inference or training is served locally from NVMe.](/files/LR3Yp1D09JN8oB5VxI0q) Set it up once from the app — **Storage Cache → Production** — to point the node at your data source. Stage the dataset onto the node's blobbers ahead of your run: an initial write pays the one-time transfer cost, and every read afterward — for inference or training — is served locally from NVMe, with no repeated fetch. Throughput **scales with concurrency**; use parallel writers to stage a large dataset quickly, and parallel readers across many files once it's staged. ```bash # Mountpoint-S3: mount the node as a filesystem to stage and access data (inference / training) mount-s3 /mnt/blimp --endpoint-url http://:9000 # S3-style: read staged data from the node's Gateway curl http://:9000// -o out ``` Lock every endpoint (`S3:9000`, Iceberg `:19123`) to your networks with the node **CIDR allow-lists**. Tune the cache size with the eviction-% (share of free disk) and validate the speed-up under **Storage Cache → Production** before you cut a workload over. ### 3 · Connect to your fleet (single namespace) * **Endpoint:** `https://fleet-.blimp.software:9443` (TLS, path-style) * **Credentials:** your account's shared fleet S3 key * **S3:** `mc alias set fleet https://fleet-.blimp.software:9443 ` · `aws --endpoint-url … s3 ls` * **FUSE (mp-s3):** `mount-s3 --endpoint-url https://fleet-.blimp.software:9443 /mnt/fleet` (same key) The hostname round-robins across all your live nodes; any node resolves the full namespace (content-addressed dedup index + cross-node fetch). Prefer the fleet URL over per-node `blimp--0…:9443` for anything user-facing. Note: one client connection lands on one node — use several clients (or many mounts) for aggregate throughput. ## Deployment & sizing **Blimp — deploy anywhere** Deploy on any server or any cloud. Point Blimp at any machine — a bare-metal server, a VPC instance, any cloud VM — with volumes attached. The deployer inspects the box and provisions a storage node automatically: it matches each blobber to a disk (up to 12 disks per node), reserves 1–2 cores per blobber with the remainder going to the gateway, and picks an erasure-coding parity width between 1 and 2 from the resulting blobber count (the rest of the disks become data shards) to maximize throughput for that hardware. No manual shard math, no fixed instance shape — the node fits the hardware you gave it. See [Auto-configuration](#auto-configuration-any-hardware-no-manual-shard-math) above for the full breakdown. Fast to stand up, fast to tear down. A node deploys in \~4–5 minutes and is ready to serve S3/NFS; destroy takes \~1–2 minutes. Add nodes to grow aggregate bandwidth and capacity; destroy any node when you're done. ## Scaling with nodes Blimp scales horizontally: * **Add unlimited nodes** from **Manage Nodes** at any time. Each node is independent and runs on your infrastructure. * **Scale throughput** by adding nodes — more nodes mean more aggregate cache and query bandwidth to serve more engines and workloads in parallel. * **Manage a demand-based lifecycle** — launch and destroy nodes from the same screen; the deployment status and disk-usage gauges keep you informed. *** Need help? Reach the team via **Help → Discord** in the settings menu, email , or [book a meeting](https://calendly.com/saswata_basu/15min).