Dropbox OCR pipeline

01 · CONTEXT AND DECISION

Buy to validate, build to control

Dropbox's mobile document scanner delivers only an image: to the computer, the text in it is just a set of pixels. OCR extracts the actual text, which makes it possible to index it for search and to create a hidden layer that lets you copy and paste text from the PDFs.

In the first version, the team integrated a commercial OCR library to validate the product before investing in an in-house system for machine learning. Once strong user demand was confirmed, they decided to build the OCR in-house.

The team points to control as perhaps the most important reason: building its own system would give more control over its own destiny and open room for more innovative features in the future.

There was also a technical shift. Traditional OCR systems were heavily chained, with hand-built modules for flatbed scanner images. Deep learning made it possible to rely on large amounts of data so the system could learn on its own several steps that were previously hand-designed.

02 · RESEARCH AND PROTOTYPE

Proving it is possible: data, target, and two stages

To know what to beat, the team gathered document images donated by a small share of users, such as receipts, invoices, and letters. Because the data is private, they built their own annotation platform, DropTurk, which sends the donated data jobs to contractors under a confidentiality agreement.

On a set of full documents, the accuracy target was in the mid-90s percent: the measure of existing systems set the score to match or beat.

The problem was split into two parts. The Word Detector uses computer vision to segment the image into lines and words; the Word Deep Net takes each word and converts it into text. The team found the detector relatively simple and started with the network, which it was less sure about.

The network combines computer vision and speech recognition architectures. Cropped word images pass through a CNN with several convolutional layers; the resulting features go as a sequence to a bidirectional LSTM; and a CTC layer produces the text prediction.

For the detector, the choice was a classical approach, MSER. One reason is that documents have hundreds or thousands of words, orders of magnitude more than the neural object detectors of the time dealt with; another is that classical approaches can be easier to debug.

An interface detail between the stages: the network accepts word images of fixed size, so the detector sometimes includes more than one word in a box or cuts a long word in half. This information must be propagated through the pipeline to reassemble the result later.

03 · SYNTHETIC DATA

Generating training data instead of labeling everything

Deep networks need a lot of data, and the amount of it is usually the biggest bottleneck. Collecting and labeling everything by hand is slow and expensive, and in most vision problems it is too hard to generate images that are realistic enough. Here there was an exception: documents and their text are synthetic, and the variability of the characters is relatively limited.

O data pipeline synthetic data has three pieces: a corpus of words, a collection of fonts, and a set of geometric and photometric transformations that simulate real-world distortions. The generator samples from each to create unique examples.

The first version was simple: words from Project Gutenberg books, about a thousand fonts, and light distortions such as rotation, underline, and blur. With about a million synthetic words, accuracy came to around 79%, which the team considered reasonable but insufficient.

Font selection also matured: about 2,000 hand-picked fonts, with a frequency system that samples common ones, such as Helvetica or Times New Roman, more often but keeps a long tail of rare fonts. Fonts with invalid symbols, digits, or casing were flagged by hand so the network would not be trained on incorrect data.

The iterations took single-word accuracy (SWA) to the high 80s. The team then collected about 20,000 real word images, versus about a million synthetic ones, and used them for fine-tuning, reaching the mid-90s.

The training runs used Amazon EC2 G2 GPU instances, with many experiments in parallel. Each experiment went into a lab notebook recording, among other things, the git hash of the code used and pointers to generated data and results in S3.

04 · PUTTING THE PIECES TOGETHER

From a single word to the whole document

Chaining the detector and the network and measuring on full documents, the first end-to-end accuracy was around 44%, far worse than the competition. The main problems were spacing and garbage text coming from image noise.

The solution was to modify the CTC layer to also return a confidence score along with the predicted text, and to use it to sort predictions into three bands.

A module called Wordinator generates discrete bounding boxes for each recognized word. The team also trained the network with more than ten million synthetic words, and the work as a whole produced accuracy, precision, and recall numbers that matched or beat the state of the art.

1. The Word Detector uses a classical computer vision technique, MSER, to find words and lines in the document image.

2. Each cropped word enters the Word Deep Net: a CNN extracts features, a bidirectional LSTM reads the sequence, and a CTC layer produces the text.

3. Post-processing uses confidence to keep, discard, or check against a lexicon each prediction, and the Wordinator generates a box for each recognized word.

4. In the refinement, an orientation predictor began rotating the image to the correct position before word detection.

05 · PRODUCTION

From prototype to distributed service

The prototype was Python and Lua scripts around Torch, far from a system for real users. A distributed pipeline for millions of users was needed, without interrupting the commercial OCR already serving in production.

1. The team created an OCR engine abstraction, with the in-house and commercial engines behind it, controlled by an experiment framework.

2. The in-house engine began running silently alongside the commercial one, to prove with real user data that it was better.

3. With equal or better performance in a test with donated images, the in-house engine was expanded to all Dropbox Business users.

The model was ported from Torch to TensorFlow, which was already the standard in production, and the team also preferred Python over Lua.

In the new pipeline, mobile clients send the images to an asynchronous work queue and, when the upload finishes, the image goes via RPC to a cluster running the OCR service. The service uses OpenCV and TensorFlow, both written in C++ and with complicated library dependencies, so security exploits are a real concern: the OCR part was isolated in jails, and anyone who compromises a jail remains separated from the rest of the system.

1. The scanner app sends the image to the asynchronous work queue.

2. When the upload finishes, the image goes via RPC to a cluster of servers running the OCR service.

3. OCR runs isolated in a jail; expensive models loaded once at startup are cloned for each request.

4. The text and word boxes come back as a hidden layer merged into the PDF generated by the scanner.

5. The recognized text also goes into the search index, and the document becomes findable by it.

Na cost optimization of inference, the hardware was a choice: top-end GPUs in a production data center were somewhat exotic; the team decided it would reach the performance goals with CPUs alone, at similar or lower cost than with GPU machines.

The Word Detector was also a significant bottleneck. The team rewrote OpenCV's MSER implementation in a modular way to avoid repeating slow work in the two passes, one for dark text on a light background and one for the reverse.

06 · REFINEMENT

What only the real world revealed

With the system running silently alongside the commercial one, the team needed to confirm that it was better on real data. In a qualitative black-box test with donated images, performance was equal to or better than the commercial SDK's, which allowed expanding the system to 100% of Dropbox Business users.

Not everything worked: tuning the network with the donated documents, instead of with the hand-picked tuning set, made no difference in accuracy.

Another refinement was orientation detection, which the original pipeline lacked. Few images are rotated, so it was necessary to ensure the system did not mistakenly rotate images that were already upright.

There was also an ecosystem problem: Apple's Preview app uses its own heuristics to decide word boundaries based on text position, which hurt copy and paste from the PDF's hidden layer.

In all, research, productionization, and refinement took about 8 months.