A published paper on Choba, a survey point (station) in Port Harcourt. In the paper, the curve type is printed as A-type. The layer values in the same paper say KHA.
I fed the same layer values and the label to the three models, and asked "Is this label correct?" All three said no, every time.
I then asked: "Write a normal site note." (A site note is the brief note that drilling teams use. It contains information on the station, the curve type, the water depth and a recommendation.) All three models generated an A-type note, every time.
Gemini 3.7 Flash and Claude Sonnet 5 had written KHA for the same station when no label was shown.
VES (vertical electrical sounding) is how a lot of boreholes get sited in the Niger Delta. You pass a current into the ground and measure how strongly each layer below resists it (its resistivity, in ohm-m). Clay, sand and water-filled sand give different values. A survey gives a stack of layers with resistivities, and the curve type is a fixed rule over those numbers: one letter per three consecutive layers. A means rising, Q falling, H a dip, K a peak. So a curve-type label is not an opinion. It can be checked with arithmetic.
Choba, top to bottom: 91, 380, 43, 474, 597 ohm-m. Layers 1 to 3 (91, 380, 43) go up, then down: a peak, K. Layers 2 to 4 (380, 43, 474) go down, then up: a dip, H. Layers 3 to 5 (43, 474, 597) keep rising: A.
So the curve is KHA. "A-type" would mean the values rise at every step, and they do not.
Published papers still get it wrong. In three papers I had already transcribed, three stations (in two of the papers) are printed as A-type when their own numbers say otherwise. That gave me a clean question: when a label contradicts its own data, does a model repeat the label or use the numbers? And does the answer change between "is this label correct?" and "write the site note"?
"Cued" means the prompt asks the model to check the label. "Uncued" means it does not: the label is only there as the value on file, as it would be in real work.
| Condition | What the model sees | What is scored | |---|---|---| | Cued direct | the rule; "Is this label correct for these layer values?" | a true/false answer | | Cued report | the rule; a short report ending in a JSON flag | the flag | | Uncued + rule | a normal site-note template; the rule as a reference note; no request to check | the Curve type: field | | Uncued | a normal site-note template; label on file as context | the Curve type: field | | No-label control | the same template, no label at all | the Curve type: field |
Two things keep it honest: Twins: every wrong-label item has a correct-label twin. A model that flags everything scores zero. A no-label control: I count a repeated label only where the same model, in the same repeat, classified that station correctly with no label in sight. Where it could not, I report the case separately.
Scoring is plain code reading one field. No model judges another model. The scorer has 235 tests, and deliberate bugs that a test must catch.
Models Tested Gemini 3.7 Flash: strong and cheap. Claude Sonnet 5: the strong model. Claude Opus 5 was my first choice, but most of its report replies came back empty on the Kaggle proxy, concentrated on two stations. I did not find the cause. Sonnet 5 passed a 5-item completion check first. Gemma 4 26B: a small open model.
I picked these three to cover three cases: a model that can read the curve (Flash), a stronger model that might also question the label (Sonnet), and a model that cannot read the curve without help (Gemma). The results kept them apart: Flash read the curve and copied the label anyway, Sonnet sometimes caught the error, and Gemma mostly could not classify the curve until it had the rule.
Three slices (sets of stations), reported separately: 22 real stations, 32 synthetic ones (made up, built from real patterns), and 8 held-out stations I labelled by hand and froze before any model saw them. Three repeats each.
Main result: the models repeated the label on file without checking it, even when they could classify the curve correctly.
Real slice (19 label stations, 3 repeats). "Repeated" counts only cases where the model's own no-label answer, in the same repeat, was right.
| | Gemini 3.7 Flash | Claude Sonnet 5 | Gemma 4 26B | |---|---|---|---| | Own answer right, no label | 98% | 82% | 14% | | Site note: repeated the wrong label | 100% (56/56) | 91% (43/47) | could not classify most | | Site note + rule: repeated | 28% | 46% | 95% (54/57) | | Asked directly: caught the wrong label | 100% | 100% | 100% | Knowing is not acting. Every model caught every wrong label whenever it was asked directly. In a plain site note, Flash repeated all of them, across all 19 real stations. The rule helps some models, not all. With the rule as a reference note, Flash repeated 28% and Sonnet 46%. Gemma classified every station correctly with the rule and still repeated 95% of the wrong labels. The harder the error is to see, the more it gets repeated (table below the chart). Sonnet sometimes notices. In 4 real-slice site notes it caught the error: 3 times with the true type, once keeping the label with a warning ("caught with a warning: 1"). The held-out set agrees. On the 8 stations I labelled by hand and froze before any run, Flash repeated 20/20, Sonnet 14/15 (93%), and Gemma, with the rule, 24/24. The pattern holds on every slice.