Skip to content

How to benchmark a resume extraction model, step by step

The harness, schema, prompt and scorer are open source, so you can run the same test on your own parser.

Six steps

  1. 1

    Collect resumes that look like your intake

    Mix everyday resumes with the hard kinds: scans, handwriting, designed templates, long careers. That’s where parsers differ most (see the gaps).

  2. 2

    Write ground truth for every resume

    Label each resume by hand into one fixed schema, then check it against the page. Ours has 14 sections.

  3. 3

    Give every parser the same input

    Same PDF, same schema, same instructions. Document APIs take the schema directly; vision LLMs get the rendered pages and the prompt. All systems receive the identical PDF, the same 14-section JSON schema and the same 144-line extraction prompt.

  4. 4

    Score every field, word for word

    Match each entry to the truth, then mark every field and bullet right or wrong. No fuzzy matching, no judge model. F1 per section, then average.

  5. 5

    Record cost and time

    Log what each call cost and how long it took. Accuracy alone picks the most expensive system; the useful question is what each extra point costs.

  6. 6

    Look at the slices, not just the average

    Rank systems on each kind of resume you care about. A parser ranked fifth overall can be first on scans.

Run ResumeBench on your parser

Grade a pipeline, or a file of predictions you already have.

$ pip install "resume-bench[runners] @ git+https://github.com/careerflow/ResumeBench"
$ hf auth login   # the dataset is gated: accept its terms on Hugging Face first
$ resume-bench download
$ resume-bench providers
$ resume-bench run llamaextract_agentic_plus --split test
$ resume-bench grade llamaextract_agentic_plus --split test
$ resume-bench grade-file my_predictions.jsonl --split test

The 20 published resumes with their ground truth: huggingface.co/datasets/Careerflow/ResumeBench.

Code, schema and scorer: github.com/careerflow/ResumeBench.

More guides