I built an AI engineering lab from the frontend

August 17, 2026

Most AI demos stop at the same place: call the API, stream the text, ship the screenshot. That part is a weekend. Everything that separates a demo from a product happens after it — the model calling typed tools, a human approving the dangerous ones, JSON that a UI can actually trust, and some way of knowing whether a change made the thing better or worse. So I built ai-workbench, a lab where each of those is an actual phase with actual code.

The shape of it

Vite 8 + React 19 + Tailwind v4 on the front, a small Hono server on the back, Vercel AI SDK in the middle, Bun running all of it. The split matters more than the names: the browser never holds an API key, every model call happens in server/, and Vite proxies /api/* to it in development.

The piece I'd keep in any project is src/domain/ — pure logic and Zod schemas, no React and no I/O, imported by both the browser and the server. Tool contracts, approval policy, structured-output schemas and the eval scoring all live there. That's what makes "the schema is a single source of truth" true instead of aspirational, and it's why the interesting logic is unit-testable without a model anywhere near it.

Tools, and the ones you don't let run

Tool calling is the first real step up from a chat box: the model emits a typed call, the server executes it, and the UI renders the result as a component instead of a paragraph. Zod defines the inputSchema, so the argument shape is validated before anything executes.

Human-in-the-loop is the second, and it's the one people skip. Some tools shouldn't fire on the model's say-so. The AI SDK's needsApproval takes a function of the input, not just a boolean, so the policy can be conditional — in the lab, sendEmail asks for approval only when the recipient is outside a trusted domain. The UI pauses on the pending call and waits for Approve or Reject. Writing that policy as a pure function in domain/ meant I could test the interesting question ("does this input need a human?") without any of the streaming machinery.

Observability and evals, or: knowing if you broke it

Phases 5 and 6 are where a lab stops being a toy. Langfuse traces every generation through OpenTelemetry — latency, tokens, cost per step — with a thumbs up/down in the UI writing back as a score. Tracing is opt-in: no keys, and the server logs a warning and runs untraced.

The evals are 12 prompts with expected behaviours, scored by code, not a judge model: required tools, forbidden tools, regexes over the answer, whether the run paused for approval. Deterministic, and free beyond the runs themselves. CI fails if the pass rate drops.

That last sentence is where I learned the actual lesson of the project.

A failed case and a case that never ran are different things

The first full run reported 9/12 (75%) — Quality regression. Nothing about quality had changed. The provider's free tier had spent its quota partway through and the last three cases were 429s.

Here's what made that invisible: the chat route answers 200 and reports failures inside the stream, so the transcript came back empty, and the scorer honestly said expected tool getWeather; called [none]. An empty answer and a wrong answer look identical from the outside. A transient 500 on a different provider faked the same "regression" later, so this isn't a free-tier problem.

Three changes fixed it:

• A describeStreamError that maps the status to a fixed phrase — rate limited by the model provider, the configured model is not available. The SDK default masks everything as "An error occurred.", which is indistinguishable from a bad answer. It maps rather than forwards, so quota figures and model ids never reach the client.
• The transcript carries that error through instead of dropping it.
• Case status became pass | fail | error, and an errored case is excluded from the pass rate rather than dragging it down. isRegression answers "did the model get worse?"; isInconclusive answers "did this run happen at all?". Both exit non-zero — a run that didn't happen still can't read as a green build — but only one of them claims a regression.

Every eval harness I've seen conflates those two. It's the same failure mode as a flaky test suite that gets muted: once a red build can mean "the network hiccuped", nobody reads red as information anymore.

What I left out on purpose

The repo has an Out of scope section instead of a wall of open checkboxes, and I think that's the more honest way to close a project. No deploy — picking hosts and holding production secrets are decisions for a real product, not a lab, and bun run build plus a single Bun process is everything a deploy would need. No conversation persistence: the sidebar is static, and building a real one is application work that would exercise none of the six practices the lab exists for. No dark mode, which stays a values-only change because every colour goes through a semantic token and no component carries a raw hex.

Seven phases, all closed. The point was never the app — it was having a place where each of these practices exists as code I've actually written, rather than a pattern I've read about.

Repo: ai-workbench on GitHub