This week my maths practice app, Tuzak Avcısı: LGS Matematik, went public on Google Play (store page checked without signing in on 7 October 2026). It drills multiple-choice questions for the LGS, the national exam Turkish students sit at the end of 8th grade. Every question is generated from parameters, so there is no fixed answer sheet a teacher could read through once. Whether the marked answer is right has to be checked by code.
Answer keys are recomputed by an independent third solver (SymPy) — 1,475 items.
I sat down to write that up properly, opened the gate to quote it, and found the sentence isn't true. This post is about what the gate actually checks, what it can't see, and where real recomputation does exist.
The repository is private, so I can't link it. All numbers below were measured on 7 October 2026 on commit 5fe5cef, whose app file is byte-identical to the one inside the released APK (SHA-256 f14c132c…), unless a line says otherwise.
independent sympy code gate — extract the choices (jsdom), then re-solve the key with sympy (exactly one correct + no distractor value-equal to it)
The first half of that comment became my CV. The part in brackets is what the code does.
It has two steps. A Node script loads the app in jsdom, runs every multiple-choice mode and phase through 40 seeds (the CI setting; my local runner uses 16), and dumps each question's options with an ok flag. A Python script then parses the options with SymPy and asks two questions: Is exactly one option marked correct? Is any distractor equal in value to the correct option? (simplify(a - b) == 0 for expressions, a solveset comparison for inequalities.)
That's all. It never reads the question text, so it never solves the problem. If a generator computes 42 and marks 42 correct when the answer is 41, the gate has nothing to compare 41 to.
That reads: 1,560 checked, 160 not applicable, 56 intentional form traps (counted inside the 1,560), 0 real failures. 1,720 questions came out of 42 modes. One multi-select phase yields no options at all and is skipped without a word; that's a second blind spot I found while writing this. 160 have answers SymPy can't compare by value (words, drawn options, side-length triples, letter orderings), so only the one-correct check applies to them. The 56 "intentional" ones have the right value in the wrong form: 13×10⁻⁴ instead of 1.3×10⁻³ (marked wrong), or 2√28 instead of 4√7 (deliberately not counted wrong). The 1,475 on my CV was an older count.
Duplicate-value options are a real bug class in generated multiple choice: two buttons that are both right, one of which the app calls wrong. This gate covers that class across every mode it can parse, except the three form-trap modes, where any value-equal option is allowed.
It also once went red because of itself. The release's build report (30 September 2026) records an item whose options included "20 ve 9" and "12 ve 15". Ve is Turkish for "and". SymPy's implicit multiplication split ve into two unknowns, v·e: 20·v·e·9 = 180·v·e = 12·v·e·15. Two different answers, declared equal. The fix taught the parser that "a ve b" is an unordered pair of numbers, and that any other option containing ve, veya ("or") or ile ("with") is compared as plain text.
In this app a wrong key doesn't crash anything. A student picks the right answer, the app marks it wrong and may open a card explaining a trap they didn't fall into. Nobody files a bug report. A 14-year-old just learns to distrust a method that works.
To put a number on the blind spot, I simulated a wrong answer key. For each mode and phase, I added 1 to every correct option that was an integer in the extracted file and re-ran the gate. 21 mode-phases had at least one integer key in the sample (from 6 to 40 of 40). 6 stayed green: exponent rules, scientific notation, multiplying and dividing roots, rational and irrational numbers, cylinder volume and cylinder surface area. In the two cylinder modes all 40 of the 40 sampled keys were now wrong. In two of the six, scientific notation and the root mode, a collision would have been waved through anyway, because value-equal options are allowed traps there. 15 went red, for the wrong reason. Many distractors are deliberate off-by-one traps, so "correct + 1" often landed exactly on a distractor, and the gate reported "distractor equals the correct answer". That's detection by coincidence, with a message that points at a different bug.
This experiment edits the gate's input, not the generators, so on its own it doesn't show that any generator is wrong. But the release code records a case where one was. In the exponent-rules mode, a question showed (−b)ᵐ·bⁿ while the key used the parity of m+n. For m = n = 3 the real value is negative and the key said positive. A comment in the generator records that the SymPy gate missed it, because the options were consistent with each other and wrong only against the question. A manual visual review caught it.
The GCD/LCM gate. It carries its own Euclid and never calls the app's own GCD/LCM functions. With a mutant that shows the LCM where the GCD belongs, it fails on the first pair: (30,18): shown GCD=90 ≠ egcd=6. Without the mutant it passes on 3,000 generated pairs.
The GCD/LCM word problems. A separate solver reads each question's raw parameters and text and recomputes the answer by its own route; for the "when do the two bells ring together for the n-th time" type it brute-forces minute by minute. My run: 8,000 questions, 0 mismatches. It isn't in the release runner or CI yet; only its mutants run there.
Both are about GCD and LCM. One more gate is described in its comment as an independent validity check for triangle-inequality questions; I haven't read its assertions yet, so I'm not counting it. The other gates I read check structure, variety and trap behaviour, not the key. For the two cylinder modes I found no file in the test folder that recomputes an answer. I didn't read all of the gates line by line, so treat the rest as "not found", not "absent". There is also a golden-answer harness that lives outside the repository; I didn't re-run it for this post, so I'm not counting it.
I checked the other numbers against the release commit and its build reports the same evening.