I Moved the LLM Out of the Scan's Critical Path

The system became easier to trust when the model was allowed to be helpful without being allowed to define reality.

By July, RedBlue had accumulated several intelligent-looking paths. That flexibility created a harder operational problem: a provider outage or inconsistent model response could change whether work completed, not just how it was explained.

Question

Where should model assistance sit in an exposure-management workflow if the platform must remain useful, reproducible, and reviewable when every provider is unavailable?

Method

I reconstructed the redesign from three commits on 26 July. The first consolidated execution around one scan brain and described the LLM as an adviser. The second made triage durable during provider failure. The third exposed reasoning and evidence in the operator interface. The exact hashes and timestamps are in a sanitised evidence record.

I checked the flow of authority: which component schedules work, which records observations, which can change state, and which merely proposes a next step.

Evidence

The scanner redesign placed deterministic orchestration, evidence-led findings, resource governance, and enumeration depth under one execution path. Model output could influence prioritisation, but it did not become the sole source of scan state.

Durable triage addressed a predictable outage case. Work was retained rather than silently lost when a model provider failed. This matters more than quickly switching providers because a fallback is only safe when the queued evidence survives the transition.

The dashboard work then made reasoning and evidence visible together. Showing a rationale without the underlying observation would only make uncertainty sound confident. The evidence viewer let the operator inspect what supported the suggestion.

Finding

The useful architecture has two clocks. The deterministic clock advances collection, state, evidence, and retries. The advisory clock can rank, summarise, or suggest, but it may be late or absent without corrupting the first.

I still value model assistance. It helps navigate a wide evidence set and can surface connections worth reviewing. The control is that a suggestion must resolve back to retained observations, and an unavailable provider must not erase or stall the underlying work.

Limitations

Repository evidence demonstrates the intended architecture, not the quality of every recommendation. This retrospective does not benchmark model accuracy or compare providers. RedBlue’s repository is private, which limits independent review to the metadata published here.

Deterministic software can also be wrong. Moving authority out of the model does not remove the need for tests, bounded inputs, operator review, and clear recovery paths.