~/blog/more-data-made-it-worsePOSTED
by Shady Nathan Tawfik · November 12, 2026 · 9 min

More Data Made It Worse: 353 Rounds of a Plate Matcher That Never Learned

I ran 353 rounds of a training loop for tactical plate recognition on 12 November 2026, adding data the whole time, and the score fell. Then I checked where the test data came from: the 60 evaluation markings were taken from the training set. Re-measured against 300 genuinely unseen markings, the result is 0.00 percent exact across six library sizes and two decision rules. This is the report: why 11.25 percent was re-recognition rather than recognition, how much a per-position classifier squeezes out of the same descriptor, and the five changes that follow from it.

Interactive[blog-2.0]

TL;DR: On 12 November 2026 I ran 353 rounds of a training loop for the plate side of VideoText, 01:38 to 17:11, stopped by hand mid-round 354. The library grew all day and the score fell from 11.25 percent in round one to a 1.56 percent average over the last ten rounds. Then I found the flaw in my own measurement: the 60 evaluation markings were taken from the training set, so the right answer was always already in the library. Re-run against 300 genuinely unseen markings, across six library sizes and two decision rules, the result is 0.00 percent exact. Not low. Zero. A per-position classifier on the same descriptor reaches 1.5 to 2.4 percent for the whole string, so the descriptor carries a little information and far too little. Twice measured, twice the same verdict: the approach does not carry.

The premise

VideoText is a browser tool first: video in, subtitles out, nothing leaves the tab. The OCR is PP-OCR v6 through ONNX and WASM, public at /de/videotext.

The second half is a simulator and dataset generator for tactical plates — 20 by 20 centimetres, white on dark, five positions, German police units. For Berlin the grammar yields 185,328 valid combinations. A luxury a normal OCR pipeline never has: unlimited labelled data, every position known in advance.

The next step looked obvious: build a reference library, add more, watch recognition climb. I ran it for fifteen and a half hours and the curve went the other way.

What was actually measured

One loop, one descriptor, one rule, no tuning.

DimensionValue
Run12 November 2026, 01:38 to 17:11, stopped manually mid-round 354, 353 rounds completed
Training set per round200, then 350, then 500, growing towards a cap of 2000 markings
Embeddings per round6000
Descriptor141-dimensional gradient descriptor
Augmentationlevels 0 to 2
Evaluation set60 markings across 8 levels (3 to 10) = 480 embeddings
Decision rulecosine nearest neighbour, threshold 0.80
Model classno neural network, no backpropagation
Calibration target99 percent precision, reached by 0 of 353 rounds
Generate platesRenderAugment 0 to 2Descriptor 141 dimsReference libraryNearest neighbour cosineThreshold 0.80Answer or abstain

What "training" meant here

Precise about the word: there was no neural network here. No loss function, no backpropagation, no weights that moved. The descriptor is fixed by hand. Training meant one thing: add labelled reference entries, then look up the closest one.

That makes the experiment clean. If the descriptor separated plate strings well, more references would help — more coverage, better odds the right plate is in the library. If it does not, they hurt, because every new entry can win a comparison it should lose. The only variable was library size.

The result

The best round is the first one.

MeasureValue
Best roundround 1, library of 200 — 11.25 percent exact at 18.12 percent precision
Average exact, rounds 1 to 105.10 percent
Average exact, rounds 344 to 3531.56 percent
Round 353, library of 4001.04 percent exact, 1.69 percent precision
Lowest round0.21 percent
Highest round11.25 percent

All of it is closed-book. The 60 evaluation markings came from the training set — eval_markings = markings[:eval] — only at different augmentation levels. The correct plate was already in the library; the matcher never read a plate it had not seen. I did not check that until today, and it changes what the table means.

What the number really means

So I rebuilt the test: 300 markings from a different seed, no label overlap with training, 8 levels from 3 to 10, 2,400 embeddings. Training set 2,000 markings, 6,000 embeddings, threshold 0.80, everything else identical.

Configurationexact, whole string
Library 50, 100, 200, 400, 1000, 1968, k = 10.00 percent
Same sizes, k = 3, majority vote0.00 percent
Old basis, 60 markings, library of 2000.00 percent

Six sizes, two rules, one number. The 1.56 to 11.25 percent was re-recognition of something already stored. On a plate the library has never seen, nothing of it survives.

The closed test had said it: levels 8 to 10 scored 0.00 percent in all 353 rounds. I read that as the end of the scale. It was the descriptor running out.

One cheap question remained: does the descriptor contain any character information? Same 141 values, one small classifier per position — Berlin, five positions, trained on levels 0 to 2, tested on unseen markings at 3 to 10, 2,000 training and 300 test markings.

PositionSoftmaxMLP, 64 hiddenMajority line
120.8 percent20.6 percent11.7 percent
216.1 percent17.3 percent11.7 percent
325.4 percent24.5 percent17.3 percent
446.7 percent45.8 percent34.7 percent
59.6 percent9.9 percent5.0 percent
Whole string1.5 percent2.4 percentnone

Positions 3 and 4 sit well above the majority line, so the descriptor does carry something. Positions 2 and 5 barely move. Whole-string exact lands at 1.5 to 2.4 percent. The bigger model does not help: the MLP is roughly the softmax at every position.

That kills the plan I ended the previous version with — keep the features, replace the decision rule. It is not the decision rule. It is the feature.

The arithmetic behind the silence

The system abstains when the best match falls under the threshold. In the closed test that rate never moved, and it explains the curve: abstention near 38 percent, so about 62 percent answered, and exact match equals the answered share multiplied by precision.

~/code/text text
round 1    11.25 / 18.12 = 0.621  ->  abstained 37.9 %
round 353   1.04 /  1.69 = 0.615  ->  abstained 38.5 %

Both reproduce the reported "around 38 percent" within half a point. Arithmetic on published figures, not a new measurement — and it localised the failure before I understood it. Abstention stayed flat, so the whole decline was a decline in precision. The system did not get more cautious; it got more wrong.

A threshold of 0.92 lifted abstention to 47.5 percent and left the hit rate practically unchanged. A threshold filters answers, it cannot reorder them.

Why more data made it worse

Four things, in order of confidence.

  1. The descriptor describes the plate, not the characters. 141 dimensions averaged over five positions. Two plates differing in one position are almost the same vector, because four of the five contributed identical evidence; the difference is a small residual inside a large sum.

  2. Cosine nearest neighbour returns one candidate — no shortlist, no reasoning per position, no way to say these differ in the fourth character. If the closest entry is a near miss, that is the answer.

  3. Every added entry is a new chance to be the closest. With a metric that cannot separate near-identical strings, a bigger library does not add coverage — it adds confusers: more plates that look almost right.

  4. There is no learning step that could compensate. Nothing here could discover that position four matters more, because nothing looked at positions at all. The representation was fixed before round one and stayed fixed for all 353.

The bottleneck was never the data. It was the discriminative power of the measurement. More data multiplies what the representation can already do — and if it cannot tell two plates apart, the multiplier lands on the confusion.

What I am changing

Five changes, cheapest first. The order is the point.

  1. Grammar as the decision rule. Berlin has 185,328 valid combinations. Whatever the model emits gets checked against that set, so an impossible string cannot leave the system. No training, no new data, and a whole class of wrong answers dies at the last step.
  2. Character crops instead of the whole plate. One 64 by 64 image per position, 5 positions by 37 classes — Stage B. The whole-plate descriptor averages away exactly the difference that decides the answer.
  3. CTC instead of an autoregressive decoder. Our strings are codes, not language. A pretrained autoregressive recogniser brings a language prior that measurably fails on rare combinations; CTC recognisers do better there, because there is no sentence to predict.
  4. Fine-tune on our own synthetic crops. We can generate unlimited images. In the literature this is the single biggest lever — and the one thing this project can do better than anyone without a generator.
  5. Vote across frames. In video the same plate stands across several frames. Temporal redundancy is free accuracy, and the OCR side has already shown it can hold a frame budget.

Not on the list: more reference entries. Two days of measurement say that lever does not exist.

What this does not say

Not that the product is broken: the OCR path is untouched — 15.4 milliseconds per line, about 3.9 times real time, 20 of 20 cues correct. Nor that the data is bad: the plates were correct, the labels exact, the augmentation ran as specified. It did its job; the measurement could not use it.

Nor that the descriptor is worthless: the table shows real signal at 3 and 4. It says the signal is too thin to decide a five-character string.

Takeaways

  • A test set drawn from the training set measures memory, not recognition. 11.25 percent became 0.00 percent once the plates were actually unseen.
  • Six library sizes, two decision rules, one result: zero. When a lever produces nothing at every setting, stop pulling it.
  • A bigger model on the same feature does not fix a feature problem. Softmax and MLP land within a point at every position.
  • The descriptor carries something — positions 3 and 4 sit clearly above the line — but not enough. 1.5 to 2.4 percent is the honest baseline.
  • Grammar is the cheapest accuracy in the system: 185,328 valid combinations, zero training.
  • Order of work: constrain the output, fix the representation, then the decoder, fine-tune, then use time.
  • Twice measured, twice the same verdict. The approach does not carry.
Shady Nathan Tawfik

Founder & Cyber IT Security Engineer in Bielefeld. Offensive security, audits and full-stack builds. More at

For educational purposes only. Every reproducible command targets my own lab environment (localhost), never a live system. © 2026