Senior interview question

How would you design the evaluation for a two-step résumé parser: PDF to text, then text to current employer?

Q: How would you design the evaluation for a two-step résumé parser: PDF to text, then text to current employer?

Score the end-to-end answer and each step on its own: step 1 by similarity of the extracted text to the true text, step 2 by accuracy when it's given correct text. Then a failure points at a step, not the whole system. Add slices such as file formats and layouts, include out-of-scope inputs, and log each step's output so failures can be traced.

Model answer

Score the end-to-end answer and each step on its own: step 1 by similarity of the extracted text to the true text, step 2 by accuracy when it's given correct text.

Score each step, not just the endStep 1 / 5

Scroll sideways to see the whole diagram

A two-step system: pull the text out of a résumé PDF, then pull the current employer out of the text.

Text version

A two-step system: pull the text out of a résumé PDF, then pull the current employer out of the text.
When the final answer is wrong, the end-to-end score alone can't say whether step 1 or step 2 broke.
Score step 1 on its own: how close is the extracted text to the real text?
Score step 2 on correct text: how often does it name the right employer?
Now a failure points at a step. The same applies to turns and whole tasks in a conversation.

Then a failure points at a step, not the whole system. Add slices such as file formats and layouts, include out-of-scope inputs, and log each step's output so failures can be traced.

Practise the rest

10 Success criteria questions, 492 in all, each with a model answer. Free.

Read the lesson Practise Success criteria

Senior interview question

How would you design the evaluation for a two-step résumé parser: PDF to text, then text to current employer?

Model answer

Score the end-to-end answer and each step on its own: step 1 by similarity of the extracted text to the true text, step 2 by accuracy when it's given correct text.

Score each step, not just the endStep 1 / 5

Scroll sideways to see the whole diagram

A two-step system: pull the text out of a résumé PDF, then pull the current employer out of the text.

Text version

A two-step system: pull the text out of a résumé PDF, then pull the current employer out of the text.
When the final answer is wrong, the end-to-end score alone can't say whether step 1 or step 2 broke.
Score step 1 on its own: how close is the extracted text to the real text?
Score step 2 on correct text: how often does it name the right employer?
Now a failure points at a step. The same applies to turns and whole tasks in a conversation.

Then a failure points at a step, not the whole system. Add slices such as file formats and layouts, include out-of-scope inputs, and log each step's output so failures can be traced.

Practise the rest

10 Success criteria questions, 492 in all, each with a model answer. Free.

Read the lesson Practise Success criteria

Model answer

More Success criteria interview questions

Model answer

More Success criteria interview questions