PDF layout parser
A small detector finds the parts of a PDF page: titles, text, tables, pictures and more. Then a short program reads the page text in the correct order and writes markdown.
Model: D-FINE nano, 3.7M parameters, 8 MB · Data: DocLayNet · Runs in your browser
Try it
Open a PDF or use the sample. The model downloads once (8 MB). The PDF stays on your computer.
What the project does
A PDF stores characters and their positions. It does not store the structure of the page. The PDF does not tell you which line is a heading, where a table starts, or which column to read first. This project adds that structure.
- Draw the page as an image.
- Find the layout regions with an object detector. There are 11 region types (classes).
- Put each word from the PDF text layer into its region.
- Sort the regions into reading order.
- Write each region as markdown. The region type sets the format:
#for a title,-for a list item, and so on.
Only step 2 uses a neural network. The other steps are short rules.
Architecture
The detector: D-FINE
D-FINE is a DETR-style detector. DETR-style detectors do not use anchor boxes or non-maximum suppression (NMS). The model has a fixed set of learned queries. Each query outputs one box and one class score.
- Backbone (HGNetv2). A small CNN. It makes feature maps of the page at three scales.
- Hybrid encoder. Attention inside the smallest feature map, then a CNN that mixes the three scales. This design comes from RT-DETR. It keeps the model fast.
- Decoder. The queries look at the features with cross-attention. Each decoder layer makes the boxes more accurate.
- Fine-grained Distribution Refinement. This is the new idea in D-FINE. The model does not predict one number for each box edge. It predicts a probability distribution for the position of each edge, and each layer refines that distribution. This gives accurate edges, which is important for thin regions like a page footer.
We use the nano size: 3.7M parameters. In fp16 the file is 8 MB, small enough to download on a web page.
The rules after the detector
- Remove duplicates. Sometimes two boxes cover the same region (IoU above 0.6). We keep the box with the higher score.
- Reading order. A full-width region (for example a title or a wide figure) cuts the page into horizontal bands. In each band, we read the left column from top to bottom, then the right column. A one-column page is one long left column.
- Assign text. Each piece of text from the text layer goes to the box that contains its center. Words broken with a hyphen at a line end are joined again.
- Format. The class sets the markdown: Title →
#, Section-header →##, List-item →-, Caption → italic, Picture →*[figure]*. Page headers and footers are removed.
Training
Data: DocLayNet
DocLayNet has about 80,000 human-labelled pages from six types of documents: financial reports, manuals, science papers, laws, patents and tenders. Each page has boxes for 11 classes:
Caption Footnote Formula List-item Page-footer Page-header
Picture Section-header Table Text Title
Some classes are rare: Footnote, Formula and Title. We repeat the training pages that have these classes three times.
Recipe
| Setting | Value |
|---|---|
| Start weights | D-FINE nano trained on COCO (ustc-community/dfine-nano-coco) |
| Input size | 1024 × 1024 (the page is resized, not cropped) |
| Epochs / batch | 12 epochs, batch 16, one A100 GPU |
| Learning rate | 2e-4 for the new layers, 2e-4 / 10 for the backbone |
| Augmentation | Small brightness and contrast changes only. Documents do not need flips or mosaics. |
| Weight averaging | EMA (exponential moving average) of the weights. We evaluate the EMA weights. |
The code is in pdf_parser/layout/train_dfine.py. Run it as follows:
python prepare_data.py --full # DocLayNet -> labels, and the oversampled list python layout/train_dfine.py --epochs 12 --batch 16 --imgsz 1024 python layout/train_dfine.py --eval runs_dfine/best --split test python web/export_onnx.py --weights models/layout_dfine2_ep9 # -> web/models/layout_fp16.onnx
Results
COCO mAP on the 2,500 DocLayNet test pages:
| Metric | Value |
|---|---|
| mAP50-95 (all classes) | 0.652 |
| mAP50 | 0.828 |
| Best class: Text | 0.793 |
| Worst class: Footnote | 0.454 |
Running it in the browser
- We export the model to ONNX and convert it to fp16. The fp16 model gives the same boxes as PyTorch on our check pages.
- The page uses onnxruntime-web. It tries WebGPU first. If WebGPU is not available, it uses WebAssembly on the CPU.
- On an M2 Max laptop: about 37 ms per page with WebGPU and about 560 ms with WebAssembly.
- int8 quantization changed the detections, so we do not use it.
Papers and links
- D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinement (Peng et al., 2024). The detector.
- DETRs Beat YOLOs on Real-time Object Detection (RT-DETR) (Zhao et al., 2023). The hybrid encoder and the real-time DETR design.
- End-to-End Object Detection with Transformers (DETR) (Carion et al., 2020). Queries and set prediction, no NMS.
- DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis (Pfitzmann et al., 2022). The training data.
- pdf.js and ONNX Runtime. Page rendering and model inference in the browser.