Architecture

One arrow goes to the model and comes back several times. The model reads the context, names a tool and stops; the runtime validates the arguments, runs it, appends the result and calls again. Everything that makes this safe lives on the runtime's side of that boundary — the allow-list, the timeout, the step budget, the refusal to pass a tool result straight into a shell. A model that could call your systems directly would be an architecture with no seams in it at all.

The dashed box holds everything the model is allowed to see: the conversation so far, whatever retrieval returned, and the output of tools it already ran. Prompt engineering is the small part; the large part is deciding what goes into that budget and in what order, because a bigger context window does not fix a retrieval step that returned the wrong fifty chunks. Treat the assembled prompt as a build artefact — versioned, diffable, and worth logging.

The top of the diagram runs in seconds; the bottom runs nightly and in CI. Indexing decides retrieval quality hours before a question is asked, traces explain a slow answer after it was given, and the eval suite is the only thing standing between “the prompt looks better to me” and a change you can defend. Ship the offline half with the online half — a stack that has only the top row cannot tell you whether it got worse.