Local CAVE Stack — Setup & Demo

A self-hosted, single-user, $0 deployment of the real CAVEconnectome backend running on one Windows box — queryable by the official caveclient, indistinguishable from production. This page is the guide to standing it up and walking a team through it.

1. What this is

CAVE — the Connectome Annotation Versioning Engine — is the backend that large EM connectomics projects (FlyWire, MICrONS) use to store annotations tied to a segmentation, resolve supervoxels to root IDs through a dynamic proofreading graph, and take immutable, versioned snapshots you can time-travel query. In production it is a fleet of microservices behind global.daf-apis.com.

This project runs that entire fleet locally, for $0, with no cloud account, on a single Windows machine. These are the genuine production container images — InfoService, SchemaService, AnnotationEngine, PyChunkedGraph, and MaterializationEngine (plus its Celery worker) — behind an nginx gateway, backed by Postgres/PostGIS, Redis, and an in-memory BigTable emulator. The official Python client connects to http://127.0.0.1:8080 and cannot tell the difference from real CAVE.

The dummy fixtures it serves:

Datastackfake_datastack → aligned volume fake_volume → PCG table test
Annotation tablemy_cave_table, schema bound_tag (a point + a string tag), voxel resolution [8, 8, 40]
Dummy graphtwo supervoxels 72127962782105600 / ...601, both resolving to root 288230376151711745
Demo annotationspoints [600,100,10] and [900,100,10], landing on the two dummy supervoxels

Why it matters

It lets us demo and train against a fully faithful CAVE — discovery, annotation, live supervoxel→root resolution, materialization, and time-travel queries — with no cloud costs and no credentials. Anyone learning the client, or testing tooling, gets the real API surface on a laptop. It also proves out the hardest-to-host piece, MaterializationEngine, entirely offline.

Where it lives. The working stack is on this box at Desktop\_scratch\cave-local\realcave. Everything below is run from that realcave directory.

2. Windows setup, step by step

Prerequisites: Docker Desktop with the WSL2 backend enabled. Confirm docker version and docker compose version both work.

Run from Git Bash driving docker.exe — not from inside WSL.

Docker Desktop's WSL integration registers Ubuntu but never injects /usr/bin/docker or the socket, and Windows-path bind mounts do not resolve over a raw TCP DOCKER_HOST from WSL. Native Windows-path bind mounts do work through docker.exe. Git Bash + docker.exe is the combination that works.

  1. Open Git Bash and go to the stack directory:

    copycd realcave

  2. Bring the whole stack up. MSYS_NO_PATHCONV=1 stops Git Bash from mangling in-container path arguments like /cfg/...:

    copyMSYS_NO_PATHCONV=1 bash ./bootstrap.sh

    bootstrap.sh runs 8 idempotent-ish steps, in order:

    1. Backing services + gateway + schema — postgres redis bigtable gateway schema.
    2. Wait for Postgres to report healthy.
    3. Create databases (datasets, annotation, materialize, + PostGIS). The per-volume DB fake_volume is auto-created later by AnnotationEngine.
    4. InfoService + DB migrations (flask db upgrade).
    5. Register the datastack — inserts the aligned_volume and datastack rows the client discovers (there is no POST API for this).
    6. AnnotationEngine + PyChunkedGraph.
    7. Build the dummy graph in the emulator — two supervoxels joined by one edge, both resolving to one root.
    8. MaterializationEngine web + worker.

    If step 3 fails once on a fresh PostGIS init race (database system is shutting down), just re-run bootstrap.sh.

  3. Create a host virtualenv (Windows Python 3.11) with the official client and its deps:

    copypython -m venv .venv-host ./.venv-host/Scripts/python.exe -m pip install caveclient pandas numpy

    cloudvolume from requirements-host.txt has no Windows wheel and the demo scripts do not need it — caveclient + pandas + numpy is enough.

  4. Create the annotation table and its two annotations:

    copy./.venv-host/Scripts/python.exe scripts/setup_demo_data.py

  5. Run the materialization workflow to create version 1 (annotations joined to root IDs in an immutable, queryable snapshot):

    copycurl -s -X POST "http://localhost:8080/materialize/api/v2/materialize/run/complete_workflow/datastack/fake_datastack?days_to_expire=5&merge_tables=true"

    Watch it with docker logs -f realcave-mat-worker-1 — look for supervoxel + root lookups and set_version_status ... AVAILABLE. It creates DB fake_datastack__mat1 and an AnalysisVersion row.

  6. Open the live dashboard in a browser:

    copyhttp://localhost:8080/

Daily use

Once it is set up, you do not repeat the whole bootstrap. Two scripts cover day-to-day demoing, both run from realcave/ in Git Bash:

Get demo-ready and run the proof — ensures the stack is up, rebuilds the in-memory dummy graph (needed after any reboot), and runs the 5-part narrated proof:

copycd realcave && ./run_demo.sh

Reset before a fresh demo — wipes the live table back to 2 clean annotations with empty edit history, then materializes a fresh version:

copy./reset_demo.sh

Cold boot (Docker Desktop not running): launch Docker Desktop, wait for the engine, then ./run_demo.sh. It handles the rebooted-box case where the in-memory graph was lost.

3. Gotchas

These are the hard-won, non-obvious things — the reasons a naive docker compose up of the real images does not just work. Don't re-discover them.

Read before you demo

  • Use 127.0.0.1, never localhost, in any host-side client config. On Windows, Python requests resolves localhost to IPv6 ::1 first and eats a ~21s SYN timeout per new connection before falling back to IPv4. This turned a ~3s demo into ~150s.
  • Rebuild the dummy graph after any BigTable restart or reboot. The emulator is in-memory, so the graph is wiped and live queries break. run_demo.sh / reset_demo.sh do this for you; manually: docker exec realcave-pcg-1 python /cfg/create_dummy_graph.py. It must run inside the pcg container so the written graph version matches the server.
  • The three MaterializationEngine compose fixes (in docker-compose.cave.yml) that got the workflow to AVAILABLE: (a) worker pool prefork --concurrency=4, not solo (solo deadlocks on the workflow's Celery chords); (b) LOCAL_SERVER_URL/GLOBAL_SERVER_URL=http://gateway as real env vars (the worker reads LOCAL_SERVER_URL from os.environ, not flask config, else defaults to a bogus host); (c) mount cfg/cave-secret.json at /root/.cloudvolume/secrets/ (CloudVolume's graphene driver demands a token file even with auth disabled).
  • AVX is required. Polars (a MaterializationEngine dependency) needs AVX instructions. Fine on this Intel i7 (native AVX2); it breaks on Apple-Silicon Macs because the x86 images run under Rosetta, which has no AVX. Fallback on a no-AVX host: build the worker with the polars-lts-cpu wheel and set POLARS_SKIP_CPU_CHECK=1.
  • Annotation POST returns 500 but the row still persists. On insert, AnnotationEngine fires a "supervoxel notify" to local_server, which is unreachable from inside the container, so it 500s — but the write already happened, and materialization does its own supervoxel lookup anyway. Optional clean fix: add 127.0.0.1 host.docker.internal to the hosts file and point local_server there.
  • AUTH_DISABLED=true but the client still needs a token. Auth is off on info/annotation/pcg/materialize, but caveclient still insists on some token — pass auth_token="dummy".
  • The materialized version counter only resets on a full down -v. Deleting AnalysisVersion rows or dropping mat DBs does not reset it; old versions accumulate but stay valid and queryable. (This is why team_demo.py's versions == [1] assertion may report a cosmetic "CHECK FAILED" once more than one version exists — the data checks still pass.)

One more intentional-looking oddity: segmentation_source host is gateway (container-reachable over the docker network) while local_server is http://127.0.0.1:8080 (host caveclient reaches it). Different fields, split on purpose.

4. Demo walkthrough — the caveclient command list

This mirrors the click-to-copy cheat-sheet on the dashboard. Start a Python shell on the host (./.venv-host/Scripts/python.exe) and paste these in order. Every command is the same caveclient API the team uses against production — only the server_address differs. Click any command to copy it.

Connect (run these first)

copyfrom caveclient import CAVEclient import datetime client = CAVEclient("fake_datastack", server_address="http://127.0.0.1:8080", auth_token="dummy")
the official client attaches to the self-hosted stack exactly as it would to real CAVE.

Discover what's on the server

copyclient.info.get_datastacks()
the client auto-discovers the datastack, like it does for production.
copyclient.info.get_datastack_info()
the datastack advertises its aligned volume and segmentation source.
copyclient.annotation.get_tables()
a real AnnotationEngine is serving a real table.
copyclient.schema.schema_definition("bound_tag")
SchemaService returns the genuine schema for the table's type.

Read the annotation table

copyclient.annotation.get_annotation("my_cave_table", [1, 2])
the two annotations come back with their positions and tags.

Materialized data — versions + time travel

copyclient.materialize.get_versions()
materialization ran; there is at least one immutable version.
copyclient.materialize.query_table("my_cave_table")
the current materialized snapshot joins each annotation to its root ID.
copyclient.materialize.query_table("my_cave_table", materialization_version=1)
any prior version is still queryable exactly as it was.
copyclient.materialize.live_query("my_cave_table", timestamp=datetime.datetime.now(datetime.timezone.utc))
time-travel: roots are reconstructed as of any given moment.

Segmentation — live supervoxel → root

copyclient.chunkedgraph.get_root_id(72127962782105600)
the real PyChunkedGraph resolves a supervoxel to its current root in real time.
copyclient.chunkedgraph.get_roots([72127962782105600, 72127962782105601])
both dummy supervoxels resolve to the same root 288230376151711745.

Edit — every change is logged with full history

copyclient.annotation.update_annotation("my_cave_table", {"id": 1, "pt": {"position": [600, 100, 10]}, "tag": "my new label"})
editing produces an audit trail — the old row is superseded, not overwritten (see next section).

Note the schema-native body {"id":N,"pt":{"position":[...]},"tag":...} — not a flattened pt_position.

5. The versioning / edit-history story

This is the "V" in CAVE — Versioning — and the most compelling thing to show a team. Editing an annotation never overwrites it. The edit_history_demo.py script tells the story end to end:

  1. CAVE logs every change. When you update an annotation, CAVE keeps the old row, stamps it deleted, marks it valid=false, and links it to its successor via superceded_id. A brand-new row holds the new value.
  2. An audit log of old values. Every prior value is still on disk, with the exact timestamp it changed — you can read the whole lineage of a single annotation, superseded rows and all.
  3. Materialize a new version. Snapshotting the current state produces a fresh immutable version N.
  4. Time-travel across versions. query_table("my_cave_table", materialization_version=N) returns the data exactly as it was at version N, and live_query(timestamp=...) reconstructs state as of any moment.

Run the whole story (re-runnable — each run adds one more edit, so the history visibly grows):

copy./.venv-host/Scripts/python.exe edit_history_demo.py

The demo prints the active "cell A" annotation, renames it, then shows the full audit log where every prior value is still present and timestamped, marked superseded and linked to the row that replaced it. It then materializes a new version and prints what "cell A" was at each version — the essence of a versioned annotation store.

6. The dashboard

The live dashboard (web/index.html) is served by the gateway from the ./web bind mount at http://localhost:8080/ — no rebuild to edit it, but it needs the running stack (it polls the services same-origin every 5 seconds). It is the visual companion to the command list, with these panels:

Annotated dashboard screenshots to be added here.