Draft

PDF layout parser

A small detector finds the parts of a PDF page: titles, text, tables, pictures and more. Then a short program reads the page text in the correct order and writes markdown.

Model: D-FINE nano, 3.7M parameters, 8 MB · Data: DocLayNet · Runs in your browser

Try it

Open a PDF or use the sample. The model downloads once (8 MB). The PDF stays on your computer.

confidence 0.50
Loading model…

What the project does

A PDF stores characters and their positions. It does not store the structure of the page. The PDF does not tell you which line is a heading, where a table starts, or which column to read first. This project adds that structure.

  1. Draw the page as an image.
  2. Find the layout regions with an object detector. There are 11 region types (classes).
  3. Put each word from the PDF text layer into its region.
  4. Sort the regions into reading order.
  5. Write each region as markdown. The region type sets the format: # for a title, - for a list item, and so on.

Only step 2 uses a neural network. The other steps are short rules.

Architecture

PDF pdf.js renderpage image, 144 DPI pdf.js text layerwords + positions D-FINE nano (3.7M params) HGNetv2backbone Hybridencoder Decoderqueries → boxes Boxesclass + score Rulesdedupe, order Markdown
The neural network only finds boxes. The text comes from the PDF itself, so no OCR is necessary for normal (not scanned) PDFs.

The detector: D-FINE

D-FINE is a DETR-style detector. DETR-style detectors do not use anchor boxes or non-maximum suppression (NMS). The model has a fixed set of learned queries. Each query outputs one box and one class score.

  • Backbone (HGNetv2). A small CNN. It makes feature maps of the page at three scales.
  • Hybrid encoder. Attention inside the smallest feature map, then a CNN that mixes the three scales. This design comes from RT-DETR. It keeps the model fast.
  • Decoder. The queries look at the features with cross-attention. Each decoder layer makes the boxes more accurate.
  • Fine-grained Distribution Refinement. This is the new idea in D-FINE. The model does not predict one number for each box edge. It predicts a probability distribution for the position of each edge, and each layer refines that distribution. This gives accurate edges, which is important for thin regions like a page footer.

We use the nano size: 3.7M parameters. In fp16 the file is 8 MB, small enough to download on a web page.

The rules after the detector

  • Remove duplicates. Sometimes two boxes cover the same region (IoU above 0.6). We keep the box with the higher score.
  • Reading order. A full-width region (for example a title or a wide figure) cuts the page into horizontal bands. In each band, we read the left column from top to bottom, then the right column. A one-column page is one long left column.
  • Assign text. Each piece of text from the text layer goes to the box that contains its center. Words broken with a hyphen at a line end are joined again.
  • Format. The class sets the markdown: Title → #, Section-header → ##, List-item → -, Caption → italic, Picture → *[figure]*. Page headers and footers are removed.

Training

Data: DocLayNet

DocLayNet has about 80,000 human-labelled pages from six types of documents: financial reports, manuals, science papers, laws, patents and tenders. Each page has boxes for 11 classes:

Caption Footnote Formula List-item Page-footer Page-header Picture Section-header Table Text Title

Some classes are rare: Footnote, Formula and Title. We repeat the training pages that have these classes three times.

Recipe

SettingValue
Start weightsD-FINE nano trained on COCO (ustc-community/dfine-nano-coco)
Input size1024 × 1024 (the page is resized, not cropped)
Epochs / batch12 epochs, batch 16, one A100 GPU
Learning rate2e-4 for the new layers, 2e-4 / 10 for the backbone
AugmentationSmall brightness and contrast changes only. Documents do not need flips or mosaics.
Weight averagingEMA (exponential moving average) of the weights. We evaluate the EMA weights.

The code is in pdf_parser/layout/train_dfine.py. Run it as follows:

python prepare_data.py --full                          # DocLayNet -> labels, and the oversampled list
python layout/train_dfine.py --epochs 12 --batch 16 --imgsz 1024
python layout/train_dfine.py --eval runs_dfine/best --split test
python web/export_onnx.py --weights models/layout_dfine2_ep9   # -> web/models/layout_fp16.onnx

Results

COCO mAP on the 2,500 DocLayNet test pages:

MetricValue
mAP50-95 (all classes)0.652
mAP500.828
Best class: Text0.793
Worst class: Footnote0.454

Running it in the browser

  • We export the model to ONNX and convert it to fp16. The fp16 model gives the same boxes as PyTorch on our check pages.
  • The page uses onnxruntime-web. It tries WebGPU first. If WebGPU is not available, it uses WebAssembly on the CPU.
  • On an M2 Max laptop: about 37 ms per page with WebGPU and about 560 ms with WebAssembly.
  • int8 quantization changed the detections, so we do not use it.
Limits. A scanned PDF has no text layer. You get boxes but no text. Tables and formulas come out as plain text, without table structure or LaTeX. That needs a reader model, for example a small vision-language model.

Papers and links