A few months ago I needed to change one line in a scanned contract. Every "free" PDF site I tried wanted me to upload the file to their server first. One let me edit for ten minutes and then asked me to pay to download the result.
That's a strange deal for a contract, a passport scan or a payslip. So I started building PDFkind, a PDF editor where everything happens in the browser. The hardest part by far was OCR: reading the text out of a scan, then turning it into real text you can edit.
This post is about how that works, what broke along the way, and the numbers I got. Everything runs client-side; the server only hands out static files.
The rule I set myself was simple: the document never leaves the device. No "we delete it after an hour", no API call to a cloud OCR service. That rules out the easy options and leaves two questions: Which OCR engine can run in a browser tab with acceptable quality? How big a download can I ask people to accept?
Tesseract.js is the obvious choice. It's mature, easy to use, and works well on clean scans. On my test set it was fine for flat, well-lit documents.
Then I tried phone photos: slightly tilted pages, shadows, uneven lighting. That's what people actually have when they "scan" something. Tesseract mostly fell apart:
| Document type | Tesseract.js | PaddleOCR (what I ended up with) | |---|---|---| | Poor-quality scans | 3.5 to 6.4% of characters wrong | about 1% | | Phone photos | unusable | 10 to 14% |
(Character error rate on my own small test set of French and English documents. Not a benchmark, but the gap was big enough to decide.)
The difference comes from how the text is found. Tesseract relies on classical layout analysis, which struggles once the page isn't flat and evenly lit. PaddleOCR uses a neural network to find text lines first, and that detector copes much better with real-world photos.
PaddleOCR is normally a Python library, but its models can be exported to ONNX and run with onnxruntime-web in WebAssembly. I use the PP-OCRv5 mobile models: a detection model that finds text lines a recognition model for Latin-script languages (English, French, Spanish, German and so on)
Together with the WASM runtime that's about 20 MB. It's only downloaded the first time someone uses OCR, and it's cached for 30 days:
One setting worth knowing about: I run ONNX Runtime with numThreads = 1. Multi-threaded WASM needs SharedArrayBuffer, which in turn needs the page to be cross-origin isolated (COOP/COEP headers). Those headers break a lot of third-party embeds. One thread is slower, but a page of OCR still takes a few seconds, which is fine.
The detector (a DBNet model) takes the page image and returns a probability map: for every pixel, how likely it is to be text.
Preparing the input has a couple of traps. The image size must be a multiple of 32, and PaddleOCR models expect BGR channel order, not RGB, normalized with the ImageNet mean and standard deviation:
The probability map isn't a list of boxes yet. Turning it into lines is the post-processing that the Python library normally does for you, so I had to rewrite it in TypeScript: Threshold the map (pixels above 0.3 count as text). Flood fill to group connected text pixels into regions. Drop regions that are too small or whose average score is below 0.6. Compute each region's convex hull, then the minimum-area rotated rectangle around it. This is what makes tilted lines work. Expand ("unclip") the rectangle, because the model predicts a slightly shrunk core of each line.
The bug that cost me a day: my first version found the four corners of each box by sorting them by y coordinate. That works on a flat scan. On a tilted photo, the top-right corner of a line can be lower than its bottom-left corner, so lines got read upside down or mirrored. Recognition on photos was garbage and I couldn't figure out why.
The fix was to stop guessing corners from coordinates and derive them from the rectangle's angle instead. I keep the angle within ±45° (swapping width and height when needed), so the width always runs along the text line. Then the corners come out in a known order: top-left, top-right, bottom-right, bottom-left.
The recognizer expects one straight, horizontal strip of text, 48 pixels high. So each detected line, which may be tilted, has to be cut out and straightened.