How to benchmark a resume extraction model, step by step
The harness, schema, prompt and scorer are open source, so you can run the same test on your own parser.
Six steps
- 1
Collect resumes that look like your intake
Mix everyday resumes with the hard kinds: scans, handwriting, designed templates, long careers. That’s where parsers differ most (see the gaps).
- 2
Write ground truth for every resume
Label each resume by hand into one fixed schema, then check it against the page. Ours has 14 sections.
- 3
Give every parser the same input
Same PDF, same schema, same instructions. Document APIs take the schema directly; vision LLMs get the rendered pages and the prompt. All systems receive the identical PDF, the same 14-section JSON schema and the same 144-line extraction prompt.
- 4
Score every field, word for word
Match each entry to the truth, then mark every field and bullet right or wrong. No fuzzy matching, no judge model. F1 per section, then average.
- 5
Record cost and time
Log what each call cost and how long it took. Accuracy alone picks the most expensive system; the useful question is what each extra point costs.
- 6
Look at the slices, not just the average
Rank systems on each kind of resume you care about. A parser ranked fifth overall can be first on scans.
Run ResumeBench on your parser
Grade a pipeline, or a file of predictions you already have.
$ pip install "resume-bench[runners] @ git+https://github.com/careerflow/ResumeBench"
$ hf auth login # the dataset is gated: accept its terms on Hugging Face first
$ resume-bench download
$ resume-bench providers
$ resume-bench run llamaextract_agentic_plus --split test
$ resume-bench grade llamaextract_agentic_plus --split test
$ resume-bench grade-file my_predictions.jsonl --split testThe 20 published resumes with their ground truth: huggingface.co/datasets/Careerflow/ResumeBench.
Code, schema and scorer: github.com/careerflow/ResumeBench.