My RAG agent declined to answer 15 of 34 test questions. In 11 of those 15 it had retrieved chunks that its own grader accepted as relevant, then cited them wrong and threw the answer away.

No unit test caught that. The eval did, after code review had already caught a long list of other problems.

What I built, and why

Outage Matcher takes a description of an incident, for example "checkout latency spiked after a config push to our load balancers". It finds similar past outages in a corpus of public postmortems and returns a likely cause and a suggested fix, with citations to the documents it used.

I built it to close gaps. I had not shipped RAG, a vector database, real tool use, LangGraph, evals, or observability, and I wanted one project that touched all six.

The corpus is 235 public postmortems: 103 from the community post-mortems list, 81 GitHub availability reports, 29 from Cloudflare, and 22 from Google Cloud. They are split by section into 4,482 chunks, embedded locally with two small models, and indexed in Postgres with pgvector. The agent is a LangGraph graph with four nodes. Retrieve calls search tools through the Anthropic API with Claude Haiku, grade scores what came back, rewrite retries with a new query, and answer writes the cited response. Around it sit a FastAPI endpoint, optional Langfuse tracing, a Docker image, and a CI gate. The architecture diagram is in the README.

One rule shaped everything else. No number goes in the README unless it comes from a recorded run committed to the repo. It is easy to write "improves retrieval by 30%" in a portfolio project, and I wanted the repo to make that hard.

When one person writes the spec, the code, and the README, that rule needs something else to enforce it.

The factory

I did not write most of this code by hand. I drove it with factory, a Claude Code plugin I wrote that runs a feature from spec to merge, with separate agents for building and for reviewing.

I scaffolded the repo by hand, then ran the setup command. It read the codebase and proposed a shape for the repo: five reviewers, a merge gate, and a list of changes that must stop for me. I approved it and it wrote one YAML file.

Reviewers. Three run on every change. The rules reviewer checks for secrets, fabricated metrics, and source terms. The logic reviewer checks edge cases in retrieval, grading, and citations. The maintainability reviewer checks conventions and dead code. Two more run only for certain paths. The eval-rigor reviewer looks at anything under the evals, and the api-contract reviewer looks at the API, the Dockerfile, and CI. Each reviewer is a different agent from the builder and has its own checklist that cites the repo's rules.

Milestones. The unit of work was a milestone, not a ticket. I wrote four by hand: M1 ingest and index, M2 agent, M3 evals, M4 ops. Each has a goal, numbered steps, and one command that proves it works.

One builder per milestone. A single builder agent takes the whole milestone in its own git worktree and makes one commit per step. When reviewers ask for changes, the same builder is resumed with its context intact, so it does not relearn the code each round.

Merge gate. Tests, lint, and the format check must pass, and every reviewer in the final round must say PASS. Then the milestone merges without me.

Flagged steps. A step marked to escalate stops the milestone for the owner. So does anything on the escalation list: touching CI workflows, writing a key anywhere, adding a data source without a terms check, changing a reported metric without a matching run, or deleting tests.

The M2b split. M4 originally held all the ops work. After M1 merged, I moved the skeleton into a new milestone, M2b. It covered the FastAPI app, a tracing wrapper, the Dockerfile, and a lint and test workflow, all in files M2 would not touch. Then I ran both builders at once.

Six jobs ran in total. Two passed review on the first round and the other four did not.

What the reviewers caught

M1, round one. All three reviewers asked for changes. The logic reviewer did not just read the chunker, it ran it. A section with a body under 40 characters was dropped along with its heading, so an Impact section reading "Users saw 5xx for 12 minutes" and a Fix section reading "Rolled back the change" both vanished. Those are some of the most useful sections in a postmortem.

It also ran a search against the live index with k=100 and got 40 rows back. The HNSW index stops at its default candidate list of 40 and does not say so. That would have capped every recall@k number I measured later.

The same round found that ingest --skip-collect on a fresh clone emptied the index and exited 0. The rules reviewer found an emoji in a scraped page title that had been committed to the manifest. The rule is no emojis in committed text, and it does not exempt text someone else wrote.

M2. A database failure was reported to the user as "no close match". The Anthropic client also had no explicit timeout, and the default is ten minutes with two retries.

M3, the judge. The faithfulness judge receives the chunks an answer cited, up to a size cap. The code hit the cap and stopped with break, so chunks the answer had used could be cut and the judge would mark true claims as unsupported. The fix keeps the used chunks first and skips only what does not fit. The judge also did not retry on invalid JSON, even though the README said it did.

M4, the leak. This one would have cost the most. The CI gate checked that the gold set's held-out chunk ids still exist in the index. But the ids are serial numbers that restart on every re-ingest. If one early page changes its chunk count, every later id shifts by a few. All of them still exist, and they now point at the wrong chunks.

The faithfulness run would then hide the wrong chunks and leave the real root-cause text searchable. The agent would be graded on questions with the answer key in its index, the score would go up, and nothing would flag it. Two reviewers found this independently. The gate now verifies identity, meaning the document URL and the chunk position, instead of existence.

What the eval caught

The gold set is 34 synthetic queries. For each one, a postmortem's root-cause section is held out of the index, and DeepSeek writes a description of the symptoms only. I asked for about 40, but only 35 documents had a clear root-cause section. The judge is also DeepSeek, on purpose, so that it comes from a different model family than the agent it grades.

The first two faithfulness runs abstained on 19 and then 15 of 34 queries. The builder added per-query diagnostics and found the cause. The answer prompt listed excerpts like this:

[0] doc_id=59 | title=... | section=...

The model cited 0. The citation validator rejected a document that was never retrieved, as it should, and the agent fell back to "no close match". That accounted for 11 of the 15 abstentions in the second run.

The fix was to drop the bracketed position from the prompt. After it, the agent abstained 7 times and none of those came from dropped citations.

Then the builder wrote a before and after table with one "before" row. There had been two runs before the fix, and the one left out had a higher fully-supported rate than the "after" run. The rules reviewer found the second result file in the repo, compared the two, and failed the round. Both runs are in the table now.

The fix did one thing, which was to cut abstentions. Claim support moved from 0.671 and 0.683 to 0.711. That gap is within the noise of single runs of a nondeterministic agent, and the runs judged answer sets of different sizes. Only one of 27 answers had every claim supported. That is not a flattering number, and it is in the README anyway.

The gate I designed wrong

Not every flaw was the builder's. I wrote the spec for the CI eval gate: re-ingest the corpus, compare the index version hash with the baseline, and fail on a mismatch.

The M4 builder ran the CI path by hand before trusting it. It got the same documents, chunks, and ids, and identical metrics, but a different hash. The cause was one chunk of one forum page where a view counter had gone from 3721 to 3722.

My gate would have failed on its first run, and again whenever any scraped page changed a counter.

The redesign has three rules. The gate fails hard only on structural incompatibility, meaning gold ids that are missing or mismatched, or counts that drift by more than 2 percent. A change in the version hash alone produces a warning. A metric drop beyond 0.05 fails. With 34 queries that tolerance is about 1.7 queries, so one flip passes and two fail.

The workflow file was a flagged step, so it stopped for me to read. The paid faithfulness job runs only on manual dispatch. The repo is public, and pull-request code should not run next to API keys.

What went wrong around the process

The agents behaved well. The surrounding setup caused more trouble.

A broken python3. The new repo pinned a Python version that was not installed, so the python3 shim failed. The scripts that updated the job board and one that resolved a merge conflict did nothing and printed nothing. I noticed when the board stopped changing.

The corpus, deleted twice. The fetched HTML is git-ignored and lived inside a job's worktree. Removing a worktree also removes ignored files, and I did that twice. The index survived in the Postgres container and I took a dump outside the repo. A cold rebuild takes about 5.5 minutes of fetching and about 18 minutes of local embedding.

API keys. The first run had no key, then the key returned a 401, then a 400 saying it was not scoped to a workspace. Along the way I moved gold-set generation and the judge to DeepSeek, which cut cost and gave me the independent judge.

Parallel jobs. M2 and M2b were meant to touch different files. They still collided in the CLI entry point, pyproject.toml, the lockfile, and the project docs. I merged main into the branch, kept both sides, and regenerated the lockfile. The tests caught a return 0 that the merge had dropped.

A clean base. The factory refuses to merge into a dirty checkout, but its own board and job records live in the same repo and change at every step. Every merge started with a commit of bookkeeping.

Rework costs time. M1 and M2 took two rounds, and M3 and M4 took three. The whole build took 4 hours 7 minutes of wall-clock time, from the first scaffold commit to the last merge. Review-driven rework came to about 38 minutes of that, roughly 16 percent of the summed milestone time, or about 57 minutes (24%) if I count all of M3's second round, which mixed review fixes with the first real eval runs. M1 had the largest share, with 22 minutes of rework after a 49-minute first build. Two fixes were not review rework at all: the citation bug the eval found took about 9 minutes, and my gate redesign took about 19, most of it an 18-minute cold ingest. These figures come from agent timestamps in the session log and include the time I spent waiting on API keys and approvals.

Results

All numbers are on the same 34 queries, with each query's held-out chunks excluded. One query is 0.029 of recall, so small gaps mean nothing.

Neighbor chunking did nothing. Its MRR was 0.515 against 0.519 for the baseline. The reranker is the only real gain. Recall@1 went from 0.412 to 0.588, which is 14 queries to 20, and MRR rose by 0.127.

The reranker costs something too. Ten queries ranked better, five ranked worse, and 19 stayed the same.

The repo ended at 353 tests passing with clean lint, and one ask costs a few cents in tokens.

Takeaways for next time

For the factory workflow:

• Specs fail too. The version-hash gate was my design, and it took a builder running the real path to show it was wrong. The orchestrator needs review as much as the builder does.
• Reviewers should run things. The best findings, 40 rows for k=100 and the vanishing Impact section, came from running the code instead of reading it. Checklists that cite the repo's own rules worked better than generic ones.
• A rules reviewer enforces honesty. An agent with no stake in the number caught the hidden second run. I wrote the rule and the reviewer applied it.
• Parallel jobs need separate files, and the lockfile is shared. Plan for that conflict before it happens.
• Keep data outside worktrees. Anything git-ignored inside a worktree is one cleanup away from gone.
• Validate keys before long runs. A ten-second check script that never prints the key is cheaper than a failed eval.
• Verify bookkeeping scripts. After the python3 no-ops, every bookkeeping step should check its own effect.

For the project:

• Investigate the five unexplained abstentions. Five queries had relevant chunks and still produced no valid citation or text. I do not know why yet.
• Repeat the faithfulness runs. Three single runs are not a measurement. I want a mean and a spread.
• Watch the first real GitHub run of the eval workflow. It has only been run by hand, and a cold run takes over 20 minutes.
• Grow the gold set. At 34 queries, one query is 0.029. Real incident descriptions would be better than synthetic ones.

The agent has a rule that it cannot cite a document it did not retrieve. The repo has the matching rule that it cannot report a number it did not record. Each rule blocked me once, and each time it was right.

The code, the eval results, and every review file are in the repo.