Topic 6

Capstone: the planner

The planner opens on the naive configuration. Your job is to find one that meets all four targets, then decide which of the passing ones you'd actually deploy. It is a simulation: the step and prefillPrefillProcessing the whole prompt in one parallel pass before the first output token can be produced.Where this is taught times are lesson 2's measurements, and the queueing, memory budget and price are modelled. Its assumptions are listed under it.

The capstone tasks

Keep a notes file, as in Module 01's capstone. For each task, write the configuration and copy the numbers from the screen.

1. Read the failure. With the naive configuration, which checks fail, and which one causes the others?
2. Meet all four. Find any configuration that passes. Which batching policy did it take, and why do the other two policies fail even with a large batch?
3. Find both edges. Keeping that policy, find the smallest and the largest batch limit that pass. Which target sets each edge?
4. Try the CPU. Is there any configuration that fits in 2 GB and passes? Which check never passes, and what is the best it reaches?
5. Move the price. On a passing configuration, raise the price per hour until cost fails. What did it change, and what did it leave alone?
6. Choose. Pick the one configuration you would deploy, and say what you are trading for what.