Topic 4

Check the plan in code

Here are five plans a model might propose for the refund task from lesson 1. The harness checks each one in code before anything runs: is every tool in the inventory, are the arguments the ones each tool takes, is the plan within 5 steps, and does any refund come after a policy check?

Predict first

One plan is: list charges, search the web for the refund policy, refund, reply. What does the harness do?

Runs it: searching the web is reasonableRuns the steps it canRejects it: web_search isn't in the tool inventory, and the plan never calls get_policy before refundingAsks the model to run web_search anyway

Check the plan, then run it

Illustrative plans · checks run in code · limit 5 steps

Task: refund the duplicate charge on order 4417. Pick a plan the model proposed.

1. list_charges(order_id)
2. get_policy(topic)
3. refund(charge_id, amount)
4. reply(text)

Valid: execute it.

Tool inventory: list_charges(order_id) · get_policy(topic) · refund(charge_id, amount) · reply(text). The last check is a business rule: refunds come after a policy check.

Each rejected plan matches a failure or check the book names:

Invalid tool: web_search doesn't exist.
Valid tool, invalid parameters: refund(order_id) when refund takes a charge ID and an amount.
Too many steps: checking everything twice makes 6 steps against a limit of 5.

The last check is different. “Refund only after a policy check” isn't a rule about planning in general: it's a rule of this business, written in code. That's the point. Rules that must always hold go in the harness, where the model can't skip them, as it did in the plan that refunds straight away.

What no plan check can catch is the book's third tool-use failure: a valid tool with valid parameters but the wrong values, like refunding $94 instead of $49. Those values only exist once earlier steps have run, so they have to be checked at the moment of the call.

Quick check

Your plan check passes, and the agent then calls refund(charge_id: “ch_1”, amount: 49) on the original charge instead of the duplicate. Which check would have caught it?

A check at call time: compare the arguments with what the earlier tool results said (ch_2 is the later charge)The plan checkThe step limitNone is possible