Benchmark

Check the files. Compare the results.

In EPUBLab's benchmark of November 3, 2026, the EPUBLab engine reached a mean 8-gram recall of 0.767 on 12 PDFs with an independent reference text, against 0.743 for pdftotext and 0.746 for PDFium; 15 of its 15 EPUBs passed EPUBCheck with no errors.

Updated: Nov 3, 2026

The files, by document type

Document	Two columns	Three columns	Footnotes	Repeated headers	Chapters	Arabic (RTL)	Japanese
Source PDF	file (PDF)	file (PDF)	file (PDF)	file (PDF)	file (PDF)	file (PDF)	file (PDF)
Result EPUB	file (EPUB)	file (EPUB)	file (EPUB)	file (EPUB)	file (EPUB)	file (EPUB)	file (EPUB)
EPUBCheck errors / warnings	0 / 0	0 / 0	0 / 0	0 / 0	0 / 0	0 / 0	0 / 0

Methodology

Same input. 15 PDFs, every one public domain or under a free licence, run through 8 text-extraction engines on November 3, 2026.
Automated scoring. 12 of the PDFs have an independent reference text of the same document (the HTML of the Wikipedia or arXiv article, the Project Gutenberg text, or govinfo's plain-text edition), never another tool's output. Text quality is measured against it with n-gram metrics.
Valid EPUB. Every EPUB our engine writes is checked with EPUBCheck: 15 of 15 had no errors.

4-gram recall: Share of the reference text's 4-word sequences found in the output: content recovered.
4-gram precision: Share of the output's 4-word sequences found in the reference: content that belongs.
8-gram recall: Like 4-gram recall with 8 words: drops when reading order or line joins break sentences (continuity).
Merged words / 1,000: Words glued to their neighbour, per 1,000 words.
Split words / 1,000: Words broken in two, per 1,000 words.
Margin leak: Share of repeated running headers, footers and page numbers left in the text.
Unmapped glyphs / 1,000: Characters reported as U+FFFD because the PDF has no Unicode for them. Reported, never hidden: a higher number can mean the engine is honest about a problem others drop silently.
Seconds: Wall time for the document.
Peak MB: Peak memory.

Results: mean per engine

Text metrics are means over the 12 documents with a reference; margin leak and unmapped glyphs over the documents where they apply. Seconds are the total for all 15 documents; peak MB is the highest single value.
Engine	Version	Licence	4-gram recall	4-gram precision	8-gram recall	Merged words / 1,000	Split words / 1,000	Margin leak	Unmapped glyphs / 1,000	Seconds	Peak MB
EPUBLab engine (EPUB output) Text measured inside the EPUB the user downloads; time and memory include writing the EPUB.	epublab-core, commit c953279 (2026-11-03)	Ours	0.866	0.663	0.767	2.460	2.045	0.071	5.254	15.3	286
EPUBLab engine (plain text) The same layout engine before EPUB writing.	epublab-core, commit c953279 (2026-11-03)	Ours	0.865	0.657	0.765	2.419	2.100	0.988	5.185	3.6	101
pdf_oxide · extract_text	pdf_oxide 0.3.78	MIT OR Apache-2.0	0.842	0.636	0.726	3.985	0.286	0.887	0.525	5.7	206
pdf_oxide · to_plain_text (XY-cut)	pdf_oxide 0.3.78	MIT OR Apache-2.0	0.811	0.609	0.658	3.715	1.243	0.794	0.525	5.8	205
pdf_oxide · Markdown	pdf_oxide 0.3.78	MIT OR Apache-2.0	0.795	0.602	0.653	7.118	2.785	0.102	0.271	8.2	202
PDFium · text().all()	pdfium-render 0.9.5, PDFium chromium/8125	BSD-3-Clause / Apache-2.0	0.828	0.617	0.746	1.659	2.997	0.999	5.971	2.6	101
OpenDataLoader PDF (local mode) Reference for heuristics; ruled out as an engine for its memory use.	opendataloader-pdf 2.5.17	Apache-2.0	0.827	0.625	0.749	2.067	0.835	0.010	0.019	32.2	1358
pdftotext (Poppler) Measuring stick only: GPL software is never shipped in EPUBLab.	Poppler 25.09.1	GPL	0.841	0.629	0.743	1.970	1.840	0.046	5.652	3.9	36

Where other engines beat ours

Documents where another engine's 8-gram recall is more than 0.02 above our EPUB output. We publish these because they are what we work on next.

Wikipedia (English): Alice's Adventures in Wonderland: pdf_oxide · extract_text 0.800 · EPUBLab 0.761
Wikipedia (Spanish): Don Quijote de la Mancha: pdf_oxide · extract_text 0.963 · EPUBLab 0.941
Wikipedia (French): Les Misérables: pdf_oxide · extract_text 0.903 · EPUBLab 0.819
Wikipedia (German): Johann Wolfgang von Goethe: pdf_oxide · extract_text 0.877 · EPUBLab 0.845
OLMo paper (arXiv 2402.00838): pdf_oxide · extract_text 0.854 · EPUBLab 0.800
Mistral 7B paper (arXiv 2310.06825): PDFium · text().all() 0.872 · EPUBLab 0.760
QLoRA paper (arXiv 2305.14314): OpenDataLoader PDF (local mode) 0.917 · EPUBLab 0.786
Calculus textbook typeset in LaTeX (Project Gutenberg #33283): OpenDataLoader PDF (local mode) 0.533 · EPUBLab 0.487
Alice's Adventures in Wonderland, 1867 printing (scan, Internet Archive): PDFium · text().all() 0.651 · EPUBLab 0.513

Results per document (8-gram recall)

Best value per row in bold. — means no reference text for that document.
Document	Pages	EPUBLab engine (EPUB output)	EPUBLab engine (plain text)	pdf_oxide · extract_text	pdf_oxide · to_plain_text (XY-cut)	pdf_oxide · Markdown	PDFium · text().all()	OpenDataLoader PDF (local mode)	pdftotext (Poppler)
Mistral 7B paper (arXiv 2310.06825)	9	0.760	0.758	0.767	0.778	0.800	0.872	0.859	0.821
Wikipedia (English): Alice's Adventures in Wonderland	24	0.761	0.760	0.800	0.800	0.672	0.774	0.781	0.769
Wikipedia (Spanish): Don Quijote de la Mancha	50	0.941	0.941	0.963	0.963	0.857	0.945	0.954	0.944
Wikipedia (French): Les Misérables	22	0.819	0.820	0.903	0.903	0.751	0.884	0.891	0.846
Wikipedia (German): Johann Wolfgang von Goethe	86	0.845	0.848	0.877	0.877	0.758	0.854	0.862	0.853
Wikipedia (Arabic): One Thousand and One Nights	17	0.673	0.674	0.315	0.315	0.273	0.001	0.000	0.659
Wikipedia (Japanese): The Tale of Genji	67	0.963	0.963	0.970	0.970	0.797	0.952	0.942	0.965
OLMo paper (arXiv 2402.00838)	21	0.800	0.800	0.854	0.273	0.719	0.761	0.820	0.781
QLoRA paper (arXiv 2305.14314)	26	0.786	0.782	0.817	0.852	0.868	0.908	0.917	0.770
Federal Register document 2026-22443	45	0.859	0.853	0.334	0.035	0.339	0.859	0.801	0.835
The 9/11 Commission Report (GPO)	585	—	—	—	—	—	—	—	—
The Constitution of the United States (House Document 110-50)	85	—	—	—	—	—	—	—	—
Economic Report of the President 2024	487	—	—	—	—	—	—	—	—
Calculus textbook typeset in LaTeX (Project Gutenberg #33283)	292	0.487	0.487	0.479	0.496	0.481	0.494	0.533	0.511
Alice's Adventures in Wonderland, 1867 printing (scan, Internet Archive)	212	0.513	0.494	0.632	0.639	0.524	0.651	0.625	0.156

Documents and licences

Document	Licence	Traits	Reference text
Mistral 7B paper (arXiv 2310.06825)	CC BY 4.0	text PDF, one column, figures, academic paper, short	yes
Wikipedia (English): Alice's Adventures in Wonderland	CC BY-SA 4.0 (text); images under their own licences	text PDF, one column, images, English	yes
Wikipedia (Spanish): Don Quijote de la Mancha	CC BY-SA 4.0 (text); images under their own licences	text PDF, one column, images, Spanish	yes
Wikipedia (French): Les Misérables	CC BY-SA 4.0 (text); images under their own licences	text PDF, one column, French	yes
Wikipedia (German): Johann Wolfgang von Goethe	CC BY-SA 4.0 (text); images under their own licences	text PDF, one column, German	yes
Wikipedia (Arabic): One Thousand and One Nights	CC BY-SA 4.0 (text); images under their own licences	text PDF, one column, Arabic, right-to-left	yes
Wikipedia (Japanese): The Tale of Genji	CC BY-SA 4.0 (text); images under their own licences	text PDF, one column, Japanese, CJK	yes
OLMo paper (arXiv 2402.00838)	CC BY 4.0	text PDF, two columns, tables, footnotes, figures, academic paper	yes
QLoRA paper (arXiv 2305.14314)	CC BY 4.0	text PDF, one column, tables, footnotes, academic paper, long	yes
Federal Register document 2026-22443	Public domain (work of the U.S. Government)	text PDF, three columns, repeated headers, footnotes	yes
The 9/11 Commission Report (GPO)	Public domain (work of the U.S. Government)	text PDF, book, chapters, repeated headers, endnotes, long, load test	no
The Constitution of the United States (House Document 110-50)	Public domain (work of the U.S. Government)	text PDF, book, repeated headers, table of contents	no
Economic Report of the President 2024	Public domain (work of the U.S. Government)	text PDF, book, tables, charts, boxes, footnotes, long	no
Calculus textbook typeset in LaTeX (Project Gutenberg #33283)	Public domain in the U.S. (Project Gutenberg; its trademark only under its licence)	text PDF, book, chapters, repeated headers, footnotes, formulas	yes
Alice's Adventures in Wonderland, 1867 printing (scan, Internet Archive)	Public domain (1867; University of Toronto scan on the Internet Archive)	scanned, OCR layer, book, chapters, illustrations	yes

Limits of these numbers

The margin-leak metric is only reliable for engines that split lines like pdftotext -layout; it is not reliable for OpenDataLoader, pdf_oxide Markdown or pdftotext without -layout.
Wikipedia references leave out tables and infoboxes, so precision is low for every engine alike on those documents.
The 1867 Alice scan is compared with a different edition (Project Gutenberg, 1865): it is good for recall, not for exact precision. It is read through the PDF's own OCR layer; EPUBLab does not do OCR yet.
Text-quality averages cover only the 12 documents with an independent reference text (HTML, Gutenberg text or govinfo plain text of the same document, never another tool's output). The other 3 report time, memory, failed pages, unmapped glyphs and margin leak.
These metrics measure extracted text. They do not measure chapter detection, footnote linking or how the EPUB looks on a reader; EPUBCheck validity is reported separately for our EPUB output.
Calibre is not part of this run.