Building a Vision-Language API to Convert PDFs into Markdown with SmolDocling

The standard way to pull structured content out of a PDF is a pipeline: OCR for the text, layout detection for the structure, table parsers for the tables, equation recognisers for the equations, and a lot of glue between all of them. Every stage has its own failure mode, and the failures compound. Anyone who has tried to extract a clean Markdown version of a slightly-messy PDF knows the shape of the resulting evening.

SmolDocling is interesting because it collapses the pipeline. It’s a 256M-parameter vision-language model that takes a rendered page as input and produces a document markup format directly, with tables, equations, and code blocks recoverable from the output. The pipeline becomes one model call.

This post is a short writeup of the FastAPI wrapper I built around it, what works, and where the rough edges still are.

What SmolDocling actually does

It’s a vision-language model trained to read a page image and emit DocTags, a structured markup format that captures both the text and the layout, headings, lists, tables, code blocks, equations, multi-column flow. DocTags is HTML-adjacent and parseable, and the docling-core library exposes a clean conversion to Markdown.

It handles, in my testing:

Category	Capabilities
Text structure	Lists, headings, code blocks
Visual elements	Tables, equations, charts (text labels)
Document layouts	Multi-column, business forms

It’s trained on a deliberately varied corpus, not just academic papers, which is why business forms and scanned reports come through reasonably. At 256M parameters it runs on a laptop: Apple MPS works, CUDA is faster, CPU is possible but slow.

How the wrapper works

The pipeline inside the API is four steps:

1. Render each PDF page to PNG via pdf2image.
2. Pair the image with the prompt "Convert this page to docling." and run inference.
3. Parse the returned DocTags with docling_core.
4. Export to Markdown with doc.export_to_markdown().

There’s no separate OCR stage, no layout post-processor, no table extractor. The model does layout and text in one pass, which is what makes the failure mode different from a traditional pipeline: when it fails, it fails on the whole page, not on one stage.

Running it

git clone https://github.com/mustafa-zidan/document-converter.git
cd document-converter
./setup.sh

pdf2image needs Poppler:

brew install poppler           # macOS
sudo apt install poppler-utils # Ubuntu

Then:

python run.py
python examples/client_example.py --api-version=v2 your_document.pdf

A few things I learned while using it:

• Keep page width under ~1500px before inference. Beyond that, latency increases faster than accuracy improves.
• Quality is best on documents with deliberate structure: technical reports, forms, scanned books. Free-form magazine layouts are still hard.

What it replaces

If you’ve built a PDF-to-Markdown pipeline before, the shift is significant. One model handles the whole conversion. Tables come through as tables, equations as equations, code blocks as code blocks, and the file you get out is something you can index, diff, or feed back into another LLM without further cleanup. For document ingestion, report automation, or any case where the input is “a PDF nobody wants to deal with,” this is a much shorter path than the OCR-plus-glue stack it replaces.

It is not perfect; small models miss things, and SmolDocling has its preferred document shapes. But the floor is much higher than I expected from 256M parameters, and the ceiling on the architecture (larger models trained the same way) is the thing worth watching.