> For the complete documentation index, see [llms.txt](https://docs.zus.network/zus-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zus.network/zus-docs/webapps/blimp.md). # Blimp Blimp is an ACID cache and autonomous materialized view (MV) on Iceberg — an efficient per-core cache (2–4 GB/s per node) that keeps your AI/ML context fresh with CDC delta-merges in seconds. One small, simple, scalable node runs in your own AWS account and VPC — your engines, your catalog, and your data never leave your cloud. ## Introduction Blimp sits between your compute engines and S3 as a smart, ACID cache with an autonomous materialized view — keeping your AI/ML context fresh, without you migrating data or changing your engines. Point your engines at it, simplify your pipeline, and lower cost. You point your existing tools (Spark, Trino, Snowflake, Ray, PyTorch, …) and your Iceberg catalog at a Blimp node. Blimp takes your query, materializes the right Iceberg views automatically, and serves reads from a high-throughput NVMe cache in front of S3. It is **non-invasive**: the node is provisioned into *your* AWS account, your data stays in *your* S3 bucket, and nothing leaves your cloud. * **Open & standard** — Iceberg tables, S3-compatible and NFS endpoints, Glue / Polaris / Nessie-compatible catalog. * **Fully ACID** — every write commits as an atomic data entity; no orphaned files, no partial reads. * **Yours end-to-end** — runs on Amazon EC2 in your VPC; you keep ownership of the data, keys, and bucket. ## Features * **Autonomous materialized views** — Blimp takes your query, then authors and materializes Iceberg materialized views (MVs) for you and keeps them fresh via CDC delta-merges in a few seconds (3–4s). Materialized queries return **sub-second** — and because the MVs are standard Iceberg tables, any existing pipeline gets the same sub-second reads through its **current query engine** (Trino, Spark, Snowflake, DuckDB, …). No SQL rewrites, no schema changes. * **Storage Cache** — an **efficient cache per core (2–4 GB/s per node)**: an NVMe cache in front of S3, exposed over fast S3 and NFS endpoints, serving fresh data for inference and keeping GPUs fed for training with no cross-region latency. Fewer cores for the same throughput means lower cost and a greener footprint, and it scales up or down with any workload — analytics, inference, or training. * **Cache Router** — a transparent read-through cache (the blimp-router, `:8088`) in front of your S3. On a miss it pulls the object from your origin once, stores it on the node blobbers, and serves it; every repeat read is then served from the blobbers. Cached data is reachable two ways: an **S3 path** (gateway `S3:9000`) for objects and an **NFS mount** (`:2049`) for files. **Mountpoint for Amazon S3** is also supported as a filesystem mount over the same S3 endpoint, with slightly better read performance than NFS. * **CDC engine** — a change-data-capture pipeline that tracks Iceberg snapshots and keeps your MVs and cached data consistent with the source tables, so optimizer MVs refresh automatically as your data changes. * **Storage benchmarks** — built-in fio, warp, and MLPerf benchmarks let you measure a node's throughput before you commit a workload. * **Unlimited nodes** — spin up as many independent nodes as you need, each in your VPC, and add more to scale throughput. ## Architecture A Blimp deployment has two planes. **Control plane (Manager)** — a multi-tenant, highly-available control plane authenticates users, provisions nodes into your AWS account, and proxies privileged operations. Application calls carry your login token in an `x-app-id-token` header; the node's gateway admin token never leaves the server side. The Manager exposes a REST control-plane API used by both the web app and your own automation. **Data plane (the node)** — each node runs entirely inside your VPC: | Component | Role | Endpoint | | ------------------- | ------------------------------------------------------------------------------------------------------------ | -------------------------------------------- | | **Gateway** | S3-compatible gateway + DuckDB query optimizer; authors/materializes MVs, encodes/decodes erasure-coded data | S3 `:9000`, NFS `:2049`, Iceberg MV `:19123` | | **blimp-router** | Transparent read-through cache in front of origin S3 | `:8088` | | **Blobbers** | Erasure-coded storage backed by local NVMe; hold cached objects, MVs, and results | internal | | **Iceberg catalog** | Catalog for the materialized views (Nessie / Polaris / Glue-compatible) | via gateway | | **CDC engine** | Tracks Iceberg snapshots and refreshes dependent MVs and cached data | internal | The gateway fans out to every blobber on write (PUT) and reconstructs from any available shard on read (GET), so its vCPU and network scale with the data-shard count while blobbers stay uniform. ## Security — blockchain zero-trust Blimp is **breach-resistant by architecture**. It runs on **zero standing trust** — nothing in the pipeline is trusted by default. Most breaches start with a spoofed key or a stolen identity, an attacker impersonating a trusted entity; Blimp anchors every identity on the blockchain, so a leaked key alone can't be replayed to forge an identity. * **Blockchain-anchored, zero-trust identity** — every entity (gateway, blobbers, clients) has an identity anchored on the blockchain, so a leaked or stolen key can't on its own be used to impersonate it. * **Signed messages between entities** — every message between the gateway, blobbers, and clients is cryptographically signed. No unsigned traffic, no shared secrets, no implicit trust anywhere in the system. * **ACID data integrity** — every write commits as an atomic data entity: no partial writes, no torn reads, no orphaned files. Your AI inference and training always read complete, consistent data. * **Customer-owned infrastructure** — every node runs inside your own AWS account and VPC; your engines, catalog, and data never leave your cloud. ## Quickstart — try Blimp end to end in \~8 minutes The fastest way to see what Blimp does: sign up, launch the prefilled dev node, and run the built-in demos — then point it at your own data. No data migration, no SQL rewrites. 1. **Sign up with Gmail.** Go to [blimp.software](https://blimp.software) and continue with Google — a key is created for you automatically. 2. **Launch the 2/1 node.** On the empty Home screen click **Launch a node**. The form comes **prefilled** for a **2/1** node (region, data/parity ratio, and storage already set) — just enter your **wallet password** and submit. The first time, connect the AWS account Blimp provisions into; after that it's remembered. You'd need to click **Create ARN**, which takes you to the AWS login page and then to a Quick Create Stack template — tick the **Acknowledge** box, click **Create Stack**, and on the resulting page select the **Output** tab and click the **Blimp** link to return to the app (or simply copy & paste the ARN value). 3. **Wait \~8 minutes** while the node forms — watch the deployment progress bar. When it turns **green**, you'll see two demo areas, the **Query Optimizer** and the **Storage Cache**. 4. **Run the SF1 query + CDC.** In **Query Optimizer → Benchmark (TPC-DS)**, click **Run TPC-DS q1** against the self-seeded SF1 (\~1 GB) dataset — note the author / materialize times; a warm re-run is sub-second. Then click **Simulate CDC (stream + re-run)**: Blimp streams new rows, marks the materialized view stale on Iceberg, and the re-run reflects the fresh data. This is exactly how Blimp serves always-fresh **dashboards and AI feature stores**. 5. **Scale up to 1 TB.** Run the **SF1000** query — the public 1 TB TPC-DS dataset, read through the node cache. A **single** query (or the **batched** suite) returns in about a second, showing how large-dataset queries perform once Blimp has materialized them. 6. **Check the Storage Cache.** Open **Storage Cache → Benchmarks** and run **warp**, **fio**, or **MLPerf** to measure the node's S3, NFS, and AI data-loader throughput. 7. **Point it at your own data.** Switch to **Production (your Iceberg)** — from the UI or the **API** — and connect your dataset in S3 through an Iceberg catalog. Iceberg gives you **time travel** and a commit-driven **CDC refresh**, and with Blimp you keep the cached data and MVs fresh automatically. You're running on your own data in minutes. ## Demos you can run ![Query Optimizer — Harness/LLM-authored Iceberg MV flow, kept fresh by CDC](/files/SvbKQHwWnnhNWZhwF9GN) Every demo proxies through the control-plane **Manager** to the node's **Gateway**. The optimizer authors a materialized view (MV), row-hash-verifies it, materializes it onto the node blobbers, and serves the result — reporting the MV authoring time, the materialize time (pull source + write MV), and the result's **rows × cols**. Each run button has an **ⓘ** info icon that pops the same architecture diagram. ### 1 · TPC-DS SF1 — internal (query + CDC) ``` Blimp → Manager → Gateway · DuckDB → local MinIO (S3) / Nessie (self-seeded SF1, 1 GB) ``` * **Run TPC-DS q1 / Run query** — authors + materializes a verified MV against the node's self-seeded SF1 data and returns hosted viewer links. A warm re-run reuses the MV and is sub-second. * **Simulate CDC** — streams new rows into the source table; the dependent MV is marked stale and the re-run reflects the new data. CDC is the internal-SF1 demo only. ### 2 · TPC-DS SF1000 — external (query-only) ``` Blimp → Manager → Gateway · DuckDB → blimp-router (read-through cache) → public S3 (blimp-tpcds1000, 1 TB) ``` Same optimizer path, but the source is the public 1 TB TPC-DS dataset read through the blimp-router cache — no local seeding and **no CDC** (query-only). Use it to show author/materialize timings at 1 TB scale. ### 3 · Throughput benchmarks — warp · fio · MLPerf Measure how much data the **gateway** can pull from the **blobbers** and serve. Open a node → **Storage Cache** → **Benchmarks**, choose a tool + concurrency / object size, and run — each streams a live log and a summary: * **warp** — S3 throughput against the gateway's S3 endpoint (`S3:9000`). Parallel GET/PUT, reporting aggregate MB/s + ops/s. * **fio** — NFS throughput against the gateway's mount (`:2049`). Sequential / random read + write bandwidth and IOPS. * **MLPerf Storage** — an ML training data-loader benchmark against the node (samples/s, achieved utilization) — an end-to-end measure of whether the cache keeps GPUs fed under an AI workload. Expected results (measured on live AWS nodes — see [Cache product and performance](#cache-product-and-performance) for the full configuration): | Benchmark | **2/1 node** (28 vCPUs, up to 128 TB) | | -------------------- | ------------------------------------- | | warp — S3 | 0.9 GB/s PUT · 4.3 GB/s GET | | warp — GET TTFB | 2.9 / 5.4 ms (p50 / p99) | | fio — NFS | 1.3 GB/s write · 1.9 GB/s read | | MLPerf — AI training | 1.6-2 GB/s at 95% AU | The numbers are measured from a separate client instance in the same VPC — an external machine, the same path a production workload takes — not from the node itself, so they are what a production deployment should expect: Results persist on the gateway with hosted log links, so a run survives a browser refresh — compare tools and concurrency levels to find the node's ceiling before you commit a workload. ## Production setup ![blimp-router read paths — ① direct cross-region S3 baseline vs ② warm S3 and ③ warm NFS served from the local blobbers](/files/atFJXbbgsiXnukZwqtEl) The production paths mirror the demos exactly, but pointed at *your* endpoints through the blimp-router read-through cache — your data is pulled into the node's blobbers and the MV + result are hosted on the node. **Production query — your Iceberg** ``` Blimp → Manager → Gateway · DuckDB → blimp-router → your S3 + Iceberg REST ``` **Production cache — your S3** ``` Router (pointed at your S3) → read-through cache → your S3 bucket ``` Open a node → **Storage Cache** → **Production (your S3 endpoint)**, supply your endpoint + bucket, and **▶ Measure PUT/GET** reports the observed last-60s data rate through the cache. Open a node → **Query Optimizer** → **Production (your Iceberg)** tab to point Blimp at your own data: 1. **Enable Prod Query.** Click **⚡ Enable Prod Query** and approve the confirmation. This turns on a usage charge billed per Gateway vCPU-hour for as long as the node runs. It is **one-way** — to stop the charge you destroy the node. 2. **Connect your Iceberg.** Fill in the Iceberg REST URL, Warehouse, S3 endpoint / access key / secret, and Namespace. 3. **Run a query.** Enter SQL and click **▶ Run Production Query**. Blimp pulls the source into the node's blobbers, materializes the MV + result, and returns hosted viewer URLs. After that, your Iceberg commit trigger drives CDC and auto-refreshes the MV. 4. **Govern access.** Each node has CIDR allow-lists for its Iceberg catalog, S3 gateway, and NFS endpoints — restrict them to the networks that should reach the node. ## Incremental materialized views (CDC) When a source table commits new data, Blimp **delta-merges** every dependent MV — replaying the MV body over only the appended files and re-aggregating into the existing view — instead of rebuilding it. Measured on a 10-MV fleet over TPC-DS SF1000, one source change refreshed every MV, each row-hash verified: | query | MV | fact | MV rows | delta-merge | verified | | ----------------- | ------------- | -------------- | ---------- | ----------- | -------- | | q1 | `mv_9dea9e27` | store\_returns | 54,446,236 | 11.1 s | ✓ | | q3 | `mv_c6b05298` | store\_sales | 541,666 | 3.9 s | ✓ | | q3 (variant) | `mv_96deadde` | store\_sales | 862 | 3.8 s | ✓ | | q19 | `mv_9154eb26` | store\_sales | 1,564,783 | 4.6 s | ✓ | | q19 | `mv_db28a568` | store\_sales | 549,528 | 4.0 s | ✓ | | q19 (net\_profit) | `mv_a1509238` | store\_sales | 65,715 | 3.7 s | ✓ | | q43 | `mv_b2ee1155` | store\_sales | 18,663 | 3.9 s | ✓ | | q52 | `mv_b0fa0382` | store\_sales | 319,900 | 4.0 s | ✓ | | q52 (narrow) | `mv_4f4bed59` | store\_sales | 1,057 | 4.0 s | ✓ | | q55 | `mv_3afa058c` | store\_sales | 917,663 | 4.0 s | ✓ | 9 of 10 MVs delta-merge in **3.7–4.6 s**, flat from 862-row to 1.5 M-row MVs — the merge cost tracks the source delta, not the MV size. The 54 M-row `store_returns` aggregate (q1) merges in 11.1 s. Every refresh is row-hash verified against a full recompute; cold author — the one-time build per query shape — runs \~3–6 min. See [Incremental materialized views (CDC)](https://github.com/0chain/gitbookdocs/tree/ebrahim/refactor/zus-docs-public/webapps/blimp/incremental-mv-architecture.md) for the full architecture. ## API & integration Two things are worth wiring into your own scripts, dashboards, and ML jobs: **running optimized queries on your data**, and **reading your data through the node cache**. (The demo and benchmark buttons are app-only — nothing to integrate there.) ### 1 · Run optimized queries on your Iceberg ![Query integration — A: the managed prod-query API authors + materializes an MV and returns hosted URLs; B: your own engine reads the MVs directly via the Iceberg MV endpoint (:19123).](/files/yK1XHjTtpPwtDDQxLkCL) POST your SQL to the control-plane API with your login token in the `x-app-id-token` header. The gateway ingests only the tables your SQL touches (in parallel), authors + materializes a view, and returns hosted result/MV URLs; a warm re-run reuses the MV and is sub-second. Enabling is one-way — it starts the per-vCPU-hour charge, and the only off is destroying the node. ```bash # enable once (one-way) curl -X POST https://datalake.blimp.software/api/v1/clusters/optimizer/prod-query/enable \ -H 'x-app-id-token: ' -H 'Content-Type: application/json' \ -d '{"cluster_id":""}' # run a query — scope the SQL to the tables you actually need curl -X POST https://datalake.blimp.software/api/v1/clusters/optimizer/prod-query \ -H 'x-app-id-token: ' -H 'Content-Type: application/json' \ -d '{ "cluster_id":"", "iceberg_url":"https://", "warehouse":"s3://", "s3_endpoint":"https://s3..amazonaws.com", "s3_key":"", "s3_secret":"", "namespace":"", "sql":"SELECT d_year, COUNT(*) FROM date_dim GROUP BY d_year" }' ``` Prefer your own engine? Point Trino / Spark / DuckDB at the read-only **Iceberg MV endpoint** `http://:19123/iceberg` (warehouse `mv`) and query the materialized views directly. ### 2 · Read your data through the node cache ![Cache integration — your app reads over NFS (:2049) or the blimp-router (:8088); warm reads are served from the local blobber cache, and a miss pulls once from your origin S3.](/files/LR3Yp1D09JN8oB5VxI0q) Set it up once from the app — **Storage Cache → Production** — which points the node's read-through cache at your S3. After that, read *through the cluster* instead of origin S3: on a miss it pulls the object from your bucket into the local blobbers, and every warm read is served locally — never leaving the region. Throughput **scales with concurrency**; a single stream is capped by your origin S3 and the gateway instance network, so use parallel readers across many files. ```bash # NFS: mount the read-through cache as a filesystem (inference / data access) mount -t nfs -o vers=4.2,nconnect=16,rsize=1048576,wsize=1048576 \ :/ /mnt/zus # S3-style: read through the node router (miss -> origin S3 -> cached) curl http://:8088// -o out ``` Lock every endpoint (`S3:9000`, NFS `:2049`, Iceberg `:19123`, router `:8088`) to your networks with the node **CIDR allow-lists**. Tune the cache size with the eviction-% (share of free disk) and validate the speed-up under **Storage Cache → Production** before you cut a workload over. ## Deployment & sizing Nodes are provisioned into your AWS account on the EC2 `c6in` family. A one-time setup connects the AWS account (via a CloudFormation Quick Create Stack that returns an IAM role ARN); subsequent launches reuse it. Capacity is described by an **erasure-coding ratio** (data shards / parity shards). The gateway scales with the data-shard fan-out; blobbers are uniform. The node tier offered is: | Tier | Erasure coding | Gateway | Blobbers | Capacity | | ---- | ---------------- | ------------------------ | ----------------- | ------------ | | 2/1 | 2 data, 1 parity | `c6in.4xlarge` (16 vCPU) | 3 × `c6in.xlarge` | Up to 128 TB | Each blobber's EBS data volume is provisioned for high throughput (gp3, 8000 IOPS / 2000 MiB/s) independent of instance size. Total usable storage is split across the data shards at deploy time. ## Cache product and performance The cache product can be used for **high-compute analytics, datalake compaction, and AI inference and training** — anywhere you need speed of execution while lowering CPU and GPU compute cost. It runs on commodity network-optimized instances and delivers the **most GB/s per core of any cache** — fewer cores for the same throughput, which is both cheaper and greener — and scales up or down with the workload. The node is scalable in either direction and usable interchangeably over the **S3 API, an NFS mount, or Mountpoint for Amazon S3**: | | **2/1 node** | | --------------------- | ------------------------------ | | Gateway | c6in.4xlarge (16 vCPU) | | Blobber nodes | 3 × c6in.xlarge | | Fault tolerance | 2 data + 1 parity | | Capacity | Up to 128 TB | | Total vCPUs | 28 | | TTFB (p50 / p99) | 2.9 / 5.4 ms | | S3 (Warp) | 0.9 GB/s PUT · 4.3 GB/s GET | | NFS (Fio) | 1.3 GB/s write · 1.9 GB/s read | | AI training (MLPerf) | 1.6-2 GB/s at 95% AU | | GET throughput / core | \~153 MB/s | The 2/1 is the most efficient option on cost/cache and GB/s per core, and it scales up or down with the workload — a demo in minutes, a pilot within an hour, a production sidecar within a day. A node spins up in about 8 minutes and destroys in about 5, so capacity follows demand instead of being provisioned for peak. ## Scaling with nodes Blimp scales horizontally — you are not limited to one node: * **Add unlimited nodes** from **Manage Nodes** at any time. Each node is independent and runs in your VPC. * **Scale throughput** by adding nodes — more nodes mean more aggregate cache and query bandwidth to serve more engines and workloads in parallel. * **Manage the lifecycle** — start, stop, resize storage and instances, and destroy nodes from the same screen; the deployment status and disk-usage gauges keep you informed. * **Pricing** follows a simple per-GB-provisioned-hour model of node storage plus the per-vCPU-hour based on Gateway instance. We do not charge during stops, or for blobber vCPUs, and do not charge for Gateway storage which is used by DuckDB for large dataset spills. You pay AWS directly for the underlying infrastructure (it runs in your account); Blimp meters its software through the [AWS Marketplace listing](https://aws.amazon.com/marketplace/pp/prodview-zn364bubzupaq). *** Need help? Reach the team via **Help → Discord** in the settings menu, or email .