# Branching AI chat app — backend architecture How the backend works end to end, and the options for each of its components. ## What the backend actually does Three responsibilities: persist the tree, reconstruct conversation context by walking that tree, and proxy the streaming call to the LLM. It's also the only piece that ever touches your Anthropic API key. ## The full request lifecycle Here's precisely what happens when the user sends a message on some branch: ```mermaid flowchart TD A["Client sends message<br/>POST attached to a parent node ID"] --> B["Insert user node<br/>New row, parent_id = target node"] B --> C["Walk the parent chain<br/>Builds ordered context array"] C --> D["Call the LLM adapter<br/>Streams request to Anthropic API"] D --> E["Stream chunks to client<br/>Backend relays tokens over SSE"] E --> F["Insert assistant node<br/>Saved once the stream completes"] ``` A quick but important nuance: **"branch from here" doesn't create a node by itself.** Clicking that button just tells the client "the next message you send should have this node as its parent, not the current leaf." No database write happens until the user actually types something. This keeps the tree free of empty placeholder nodes. Worth also being explicit about the two most fragile edges of this flow: - **If the client disconnects mid-stream** (closed tab, network drop), the backend should cancel its own request to Anthropic too — otherwise you're paying for tokens nobody will see. Both the Fetch API's `AbortController` (Node) and `httpx`'s cancellation (Python) make this straightforward if you wire it to the SSE connection's close event. - **What gets saved if the stream is cut short.** Decide up front whether a partial response gets saved as-is (marked incomplete) or discarded. Saving partials is usually better UX — nobody wants to lose a long answer because their laptop slept. ## Backend components, and the options for each ### 1. API / web framework — the thing that receives requests This is what listens for `POST /branches/:nodeId/messages`, validates the body, and orchestrates the rest of the pipeline. - **Node.js:** Hono (extremely light, edge-deployable, first-class streaming) or Fastify (mature, great plugin ecosystem) are the two I'd pick between. Express still works but its streaming story is clunkier and it's showing its age. NestJS is heavier and dependency-injection-based — worth it at team scale, overkill for a solo project. - **Python:** FastAPI is the clear choice — native async, built-in request validation via Pydantic, and `StreamingResponse` makes SSE straightforward. Flask can do it too but you're fighting the sync-by-default model. - **Rust:** Axum (Tokio-based, plays nicely with `sqlx`) or Actix-web (slightly older, very fast). Given a systems background, Axum's explicit, no-hidden-magic style will feel familiar. - **Go:** Chi or Fiber, or honestly just the standard `net/http` — Go's stdlib streaming support is solid enough that a framework isn't strictly necessary for something this size. ### 2. Data access layer — how the backend talks to the database Here's a case where less is often more. The entire query surface for this app is small: insert a node, fetch a node, fetch a node's full ancestry, fetch a node's children, list all nodes in a tree. That's maybe five queries. A full ORM can add more ceremony than it saves, especially since the one query that matters most (the recursive ancestry walk) usually has to be hand-written raw SQL anyway. - **Node.js:** Drizzle is the best fit here — it's closer to writing SQL than Prisma is, and dropping into a raw recursive CTE isn't fighting the tool. Prisma is more polished for simple CRUD but recursive CTEs need its `$queryRaw` escape hatch. - **Python:** SQLAlchemy/SQLModel work, but for a query set this small, just using `asyncpg` (Postgres) or `aiosqlite` (SQLite) directly with hand-written SQL is genuinely simpler. - **Rust:** `sqlx` is a strong match — it checks your raw SQL against the actual schema at compile time, so you get type safety without an ORM's abstraction layer sitting between you and the CTE. ### 3. Database schema The whole model fits in one table: ```sql CREATE TABLE messages ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), tree_id UUID NOT NULL, -- which conversation this node belongs to parent_id UUID REFERENCES messages(id), -- null = root of the tree role TEXT NOT NULL, -- 'user' or 'assistant' content TEXT NOT NULL, model TEXT, -- which model generated it, if assistant created_at TIMESTAMPTZ DEFAULT now() ); ``` And the ancestry walk — the query that reconstructs context for any branch — is one recursive CTE: ```sql WITH RECURSIVE ancestry AS ( SELECT * FROM messages WHERE id = $1 UNION ALL SELECT m.* FROM messages m JOIN ancestry a ON m.id = a.parent_id ) SELECT * FROM ancestry ORDER BY created_at ASC; ``` This works in Postgres as-is, and in SQLite too (recursive CTEs have been supported since 3.8.3, so any modern SQLite build handles it fine). ### 4. Streaming transport — SSE vs WebSockets Stick with **Server-Sent Events**. Anthropic's own streaming API is already SSE under the hood, so your backend is really just re-emitting events it's already receiving, reshaped slightly if needed. Both FastAPI's `StreamingResponse` and Node's `ReadableStream`/`fetch` handle this natively. WebSockets only earn their extra complexity (bidirectional, connection state to manage) if you later add something like live multi-device sync — not needed for the core loop. ### 5. Authentication — only relevant if you go hosted If you build the **local-first** version (Tauri + SQLite on-device), you can skip this section entirely — there's no network boundary to guard, so there's nothing to authenticate. If you go **hosted**, options in rough order of effort: - A simple shared passphrase or API key if it's just you accessing it remotely — not real auth, but sufficient for a single-user tool. - Session cookies with a lightweight library (Lucia for Node, `fastapi-users` for Python). - Managed auth-as-a-service — Clerk, Supabase Auth, or WorkOS — if you want OAuth/passwordless without building it yourself. ### 6. Background jobs — only needed for the "extras" The core send-a-message loop is entirely synchronous and doesn't need a job queue. Once you add things like auto-titling branches or building a search index, you'll want work that happens *after* the response completes without blocking it: - For an MVP, firing an async task in-process right after the assistant node is saved is enough (e.g., `asyncio.create_task` in Python, or just not `await`-ing a promise in Node). - Once you want retries and backoff — say, the summarization call fails and you don't want to silently drop it — a real queue like BullMQ (Node) or Celery/RQ (Python), backed by Redis, becomes worth the setup. ### 7. File/attachment storage — only if you add images or file uploads later Local disk is fine for the local-first build. For hosted, an S3-compatible object store (Cloudflare R2, Backblaze B2, or AWS S3 itself) keeps large blobs out of your Postgres rows. ### 8. Deployment — for the hosted path only Fly.io or Railway get you Postgres + a long-running server with minimal ops. A plain VPS with Docker Compose is the most "own your stack" option. Be careful with serverless platforms (Vercel/Netlify functions) here — some have real limits on long-lived streaming responses, so check that before committing if you go that route. ## If you want one opinionated pick For the local-first path this plan leans toward: **Axum + `sqlx` + SQLite + Tauri**. No auth layer, no job queue, no object storage — the entire backend is a handful of routes and the one CTE above, running as a sidecar process inside the desktop app. If you'd rather prototype faster and worry about the "systems" layer later, **Hono + Drizzle + SQLite** gets you the same shape in a language with less ceremony, and you can always port the Rust core over once the UX is validated.