Tool calls and constrained sampling interview questions

For tools you define, your application. The model only emits a name and arguments, such as lbs_to_kg with {"lbs":40}.

Your code runs it and sends the output back, a tool_result in Anthropic's API, in the next request. Every result becomes context: billed tokens the model reads like any other text.

The two tool descriptions. The model never sees your function: the API builds a system prompt from the definitions, billed as input tokens, and the model chooses from that text. Anthropic calls detailed descriptions "by far the most important factor in tool performance" and suggests at least 3 to 4 sentences: what the tool does, when to use it and when not, what each parameter means, and its limits.

Score every token, mask the ones the rule forbids, renormalise the survivors, then sample as usual. It guarantees form, not truth.

On "The largest planet in our solar system is", GPT-2's recorded favourite is " about" (9.4%). Allowing only planet names, " Earth" (2.8% before masking) edges out " Jupiter" (2.7%); renormalised, 24.8% to 23.7%, upper bounds since only the top 50 tokens are named. Renormalising keeps the survivors' order.

Treat the arguments as a stranger's input: an allowlist of tools, a schema check, then app-side policy (who may delete what, how many, after which confirmation). Code catches an invalid tool or invalid parameters, not a wrong value: lbs = 100 when the user said 120 passes every check. Confirmation is app state, the user's click, never a "confirmed": true the model wrote. Strict tool use guarantees shape, not values: a schema-valid delete is still a delete.

Not blindly. It needs a grammar per format, can add latency, and output cut off at the length limit is still unparsable. It can also override the model, so watch the mass on allowed tokens. On "I opened the box and found", the digits in GPT-2's top 50 held 1.5%, and the top one, " 2", was its 20th choice; " two" had 7.7 times the probability of " 2". When that mass is small, ask for the format the model writes naturally, or convert afterwards.

Try providers in preference order, and bench one for its retry-after time when it rate-limits or errors.

import random, time

class RateLimited(Exception):
    retry_after = None

class Pool:
    def __init__(self, providers, cooldown=30.0):
        self.ps = providers  # in preference order
        self.until = {p.name: 0.0 for p in providers}
        self.cooldown = cooldown

    def call(self, req, rounds=3):
        for r in range(rounds):
            for p in self.ps:
                now = time.monotonic()
                if self.until[p.name] > now:
                    continue
                try:
                    return p.complete(req)
                except RateLimited as e:
                    wait = e.retry_after or self.cooldown
                    self.until[p.name] = now + wait
                except (TimeoutError, ConnectionError):
                    self.until[p.name] = now + self.cooldown
            time.sleep(2 ** r + random.random())
        raise RuntimeError("all providers failed")

Interviewers look for jittered backoff, never retrying a bad request, timeouts on every call, and no silent failover mid-stream once tokens reach the user.

Give the model a few typed tools, like sum a range or delete rows, plus a compact view of the sheet: headers, size, a few rows.

It emits a tool call; your code validates it against the schema and the live sheet, checking column K exists and row 5 is in range, then executes. Destructive actions show a preview and need confirmation, and everything is undoable. Evaluate on commands with expected calls.

The model proposes; deterministic code decides. Every tool call passes a policy layer the model can't change: a tool allowlist, argument schemas, the user's own permissions, and rate and spend limits. Irreversible or high-value actions need human approval, bound to the exact arguments shown.

Tool outputs are untrusted: they can't grant permissions, and sensitive data can't reach outbound tools. Loops end at step, token and time budgets, or on repeated actions. Scale with stateless workers over a durable task queue, logging every step for audit and replay.