PDF OCR

PDF OCR#

The PDF OCR (optical character recognition) step converts images of text embedded in PDF files into machine-encoded text. Images of text are typically found in PDFs of scanned documents.

Overview#

The PDF OCR step extracts text from files of MIME type application/pdf that don’t contain any machine-readable text.

The Content Augmentation step needs to run before and the Content Extraction (formerly Content Conversion) step after the PDF OCR step.

Configuration#

Enabling redo_ocr or force_ocr will apply the corresponding settings to the tool used for OCR. More information about these settings can be found in the ocrmypdf documentation.

Field	Default	Description
`replace_file`	True	Replace original PDF file with a PDF file containing the extracted text overlay.
`ocr_timeout`	60	Maximum time in seconds spent on OCR per document.
`redo_ocr`	False	Applies OCR on every page, including those that already have text. It replaces any existing text layers with OCR text. Aims to create a single, clean, and hopefully accurate text layer, in the expense of potentially increased file size and processing time.
`force_ocr`	False	Applies OCR on every page, including those that already have text. It layers new OCR text over any existing text, without replacing it. It ensures that all the elements in the PDF are OCR’d, however, it can significantly increase file size, and increase processing time. Force OCR takes precedense over Redo OCR when both options are enabled.
`confidence`	False	Annotate document with informative OCR confidence score. Enabling this may slow down OCR processing and even influence its results. Unless explictly needed, it is recommended to leave this disabled.
`ignore_ocr_errors`	True	When OCR fails for a PDF file, the step logs the error, leaves the file unchanged, and succeeds. Set to `False` to report an error for the item instead, as described in Reporting OCR Errors.
`ocr_language`	(empty/eng)	Tesseract language code for OCR text recognition. Defaults to English (`eng`) when not specified. For non-English documents, this setting is essential for proper text extraction. Supports single languages like `deu` (German) or multiple languages with `+` separator (for example, `eng+chi_sim`). Some languages include `_fast` and `_best` model variants for performance optimization (for example, `chi_sim_fast`, `chi_sim_best`). For the definitive list of languages supported, use `tesseract --list-langs` on your Squirro cloud instance. See Tesseract Data Files for more information.

Reporting OCR Errors#

By default, a PDF file that cannot be OCRed is left unchanged and the step succeeds. If ignore_ocr_errors is False, the step reports an error for the item instead when one of the following happens to a file:

The file is a PDF file and OCR fails, for example because the file is password protected or ocr_timeout is reached.
The file content cannot be read from storage, and the file is a PDF file or of unknown type. This covers read errors that the storage reports as file errors, such as a missing file in the local file storage. Other read errors, such as errors returned by the Amazon S3 service, are not handled by the step: the item is retried, whatever the value of ignore_ocr_errors.
replace_file is True and the OCRed file cannot be stored.

PDF files that already contain text, PDF files in which OCR finds no text, and files that are not PDF files are not failures.

When the step reports an error, it applies no OCR results to any file of the item and keeps the original files. The item is then retried with increasing waits. Every attempt repeats the OCR work done before the failure, so an item that keeps failing adds OCR processing time and cost on each attempt.

With the default max_retries = 10 in the [processor] section of ingester.ini, an item whose PDF file keeps failing, such as a password-protected one, moves to the failed batch after 11 attempts over about a day.

Setting ignore_ocr_errors to False can also raise peak resource use for items with several PDF files. While such an item is processed, peak memory use can be higher than with the default. If replace_file is True, peak use of the file storage can be higher too. The temporary storage described in Temporary Storage is not affected. The extra use grows with the number and size of the PDF files in the item.

Temporary Storage#

The PDF OCR step writes each PDF file and its intermediate files to temporary storage while it processes the file. The space needed grows with the size and page count of the PDF file. By default, the step uses the default temporary directory of the service that runs the pipeline.

To keep all temporary files of the PDF OCR step in a dedicated location, for example a larger volume, set pdf_ocr_temp_directory to the absolute path of that location in the [pipeline] section of /etc/squirro/common.ini. Leave the value empty to use the default temporary directory.

[pipeline]
pdf_ocr_temp_directory = /data/pdf-ocr-tmp

For Docker-based deployments, set the SQ_PIPELINE_PDF_OCR_TEMP_DIRECTORY environment variable instead of editing common.ini directly.

Restart the sqingesterd service for the change to take effect. For instructions, see the Services page.

The setting applies to all projects of the instance. It is not part of the step configuration.

The step removes its temporary files from the directory after it processes each PDF file. The removal is best effort, so files can remain in the directory. Monitor the free space of the directory.

Note

The directory must already exist and be writable by the service that runs the pipeline. The step does not create the directory and does not fall back to the default temporary directory. If the directory is missing or not writable, OCR fails and the PDF files are left unchanged. The error log names the pdf_ocr_temp_directory setting and the path. By default, the step tries every PDF file. If ignore_ocr_errors is False, the step stops at the first PDF file and reports an error for the item, which is then retried as described in Reporting OCR Errors.