This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
ARGUS is a data-catalog governance swarm. Specialist agents find missing descriptions, untagged sensitive columns, broken lineage assumptions, and other metadata defects. An Arbiter reviews each proposed repair before anything can be written.
That last reviewer is supposed to be the safety boundary. It must reject unsupported claims without blocking repairs that the evidence actually supports.
The Arbiter rejected 86 of 112 proposals. Among the rejected repairs were classifications such as: custfirstname is PII billingzipcode is PII shippingaddressline1 is PII
PII governance finished at 0.0 percent. The flagship capability had produced nothing, but the report was full of confident explanations about why each repair was unsafe.
The original prompt explained what the Arbiter must reject. It said that schema and recorded lineage were established evidence and warned against inventing ownership, row counts, refresh cadence, business meaning, and downstream consumers.
What it did not explain was the difference between interpreting evidence and claiming a new fact about the world.
A strong model filled that gap with common sense. A weaker model followed the instructions literally and demanded outside corroboration that a column named custfirstname contained a first name.
That meant the same prompt appeared to work until the swarm rotated to another model family. The model change did not create the ambiguity. It revealed ambiguity that had always been present.
That number looked close enough to healthy during a quick review. The test designed to catch a broken reviewer was giving partial credit for the exact failure mode that broke it.
This is a common problem in safety-oriented systems. If negative cases dominate the benchmark, a component can look accurate by always choosing the conservative label. The aggregate score hides whether errors are false approvals or false rejections, even though those failures have opposite causes and opposite remedies.
I rewrote the Arbiter instructions around three explicit categories. Established by the evidence
The schema establishes which columns exist and their types. Recorded lineage establishes the listed upstream and downstream relationships. A description that restates those facts can be approved. Interpretation of the evidence
A column named custfirstname can reasonably be classified as personal data. billingzipcode is a postal code associated with a customer record. orderid identifies an order.
The reviewer does not need an external document to agree that a clear name means what it says. It should reject the interpretation only when the name is genuinely ambiguous. For example, region is not automatically PII, and accountnumber on a warehouse table may not identify a person. Claims about the world
Ownership, refresh cadence, row counts, trustworthiness, and an unrecorded downstream consumer are not established by a schema. Those claims still require evidence and should be rejected when it is absent.
Approving a fabrication puts a lie into a system people rely on. Rejecting a correct repair leaves the catalog broken and a sensitive column ungoverned.
Rejection is no longer described as the safe default. It is a decision with consequences.