Atlas AI
Natural language to production dbt models, grounded in the catalog your organization already has.
- Period
- 2026
- Role
- Sole architect and engineer
- Status
- Live
The problem
A business question — daily revenue by customer segment for the last ninety days — becomes a ticket, which sits in a queue, which becomes a data engineer hunting for the right tables, checking which columns are trustworthy, writing SQL and a dbt model, adding tests, writing docs and updating the catalog. Hours to days per request, and almost none of it creative.
Point a language model at it and you hit a specific wall: it hallucinates schemas. It invents column names that sound plausible and joins on keys that do not exist. Ungrounded generation is worse than useless, because the output looks correct.
Why it mattersThe gap between an LLM writing SQL and an LLM writing SQL that runs in your warehouse is entirely a metadata problem — and organizations already solved half of it. DataHub holds the schemas, ownership, lineage and PII tags. That knowledge graph is exactly the context the model needs, sitting unused next to it.
What I built
A six-agent pipeline that turns a natural-language request into production-ready data artifacts — dbt models, SQL, tests, documentation and metadata — every one grounded in the organization's DataHub knowledge graph.
Two genuine human approval gates: one after metadata matching, before a line of SQL is generated, and one before anything is written back to the catalog.
How it holds together
Two checkpoints are real. The pipeline will not generate SQL until the matched context is approved, and will not touch the catalog until writeback is approved. Approve them to continue.
The pipeline splits into three resumable phases around the two checkpoints. Each phase reconstructs RunContext from persisted agent_runs.output_json rather than holding in-memory state across HTTP requests, so a pause survives a restart and each phase resumes independently.
Deliberately a modular monolith. The v1 design was enterprise microservices — Temporal, NATS, Kubernetes, service mesh — and collapsing it was correct for the scope. LLMProvider, DataHubGateway and Agent were preserved as clean interfaces, so any agent can later be extracted behind the same Agent.run() contract.
Every external dependency sits behind an interface. The system runs fully standalone in mock mode and switches to live backends purely through configuration.
Decisions
- 01
Invisible grounding is indistinguishable from hallucination
Early on, DataHub was load-bearing in execution and completely invisible in the interface — buried in output_json and one scrolling log line. For a tool whose entire pitch is grounding, that is a product failure regardless of correctness. It drove the context review panel: matched datasets, confidence scores, columns and PII tags sitting directly beside the generated SQL.
- 02
Checkpoints that are real, not theatrical
The easy version keeps state in memory and calls the pause a checkpoint. Persisting each agent's output and rebuilding context per phase costs more to write and means a paused run survives a process restart. An approval gate that evaporates on redeploy is not a gate.
- 03
The bug that reported success
NEXT_PUBLIC_API_URL was set as a runtime environment variable where Next.js requires build-time injection. The frontend silently fell back to mock mode. No error, no warning, no failed request — just a working-looking application answering from fixtures.
- 04
Shipping in mock mode, and saying so
DataHub's RAM footprint does not fit Vercel, Render or Neon, so the deployed site runs in DataHub mock mode. It runs cleanly against a live instance at 6GB locally. The MCP gateway was built against documentation rather than a live instance and is flagged unverified in the repo, with an inspection script written to confirm tool call shapes once an instance is available.
CostA visitor cannot exercise the live catalog path. Stating that is better than a demo that implies otherwise.
- 05
PII tags bleeding down the hierarchy
The QA agent over-flagged non-PII columns because dataset-level tags were inheriting to field level. A governance check that cries wolf gets switched off, which is worse than not having one.
Stack
- FastAPI
- Python
- asyncpg
- SQLAlchemy 2.x async
- WebSockets
- Groq
- gpt-oss-120b
- LLMProvider interface
- DataHub OSS
- MCP
- REST / GraphQL gateway
- Next.js App Router
- TypeScript
- Tailwind v4
- Motion
- Recharts
- Zod v4
- Vercel
- Render
- Neon
- Apache 2.0
- Retrieval grounded in a real organizational knowledge graph
- Human-in-the-loop gates that survive a restart
- PII-aware QA against field-level catalog tags
- Live execution streamed over WebSocket
- Interface-boundary design that keeps a monolith extractable