Topic 6

The model gateway, and wrap-up

Once you have several models, from several providers, used by several applications, something has to hold them together. AI Engineering (Ch. 10, "Gateway") describes the model gateway: an intermediate layer giving your organisation one unified, secure interface to different models, both self-hosted and commercial.

Predict first

Ten teams each call a provider's API directly with a shared key. The provider changes its API. How many places need updating?

Ten, one per team, and the shared key is a leak waiting to happen; with a gateway, oneOneNoneOnly the provider's

A model gatewayStep 1 / 3

Scroll sideways to see the whole diagram

AI Engineering: a model gateway gives your organisation a unified interface to different models, commercial and self-hosted.

Text version

AI Engineering: a model gateway gives your organisation a unified interface to different models, commercial and self-hosted.
If a provider's API changes, only the gateway changes.
It centralises access: instead of handing out provider keys, you grant access to the gateway, with per-user limits, usage tracking, and fallback when a provider fails (the next lesson).

The book lists what a gateway centralises:

Access control: grant access to the gateway, not to provider keys, with fine-grained rules on who can use which model.
Cost management: monitor and limit API usage to prevent abuse and manage costs (lesson 3).
Fallbacks: when an API is down or rate-limited, route to an alternative, retry after a wait, or fail gracefully (lesson 2).
Logging and analytics, load balancing, and in some gateways caching and guardrails.

The book names off-the-shelf examples such as Portkey's AI Gateway, MLflow AI Gateway, Kong and Cloudflare, and notes gateways are relatively easy to build.

Wrap-up. From this lesson, you can now:

choose models by filtering on hard attributes, narrowing with public data, evaluating on your own, and monitoring;
account for licences and for API models that change underneath you, and pin versions;
route by intent, including cheap refusals for out-of-scope queries;
route by difficulty, and read the cost–accuracy curve a router traces;
use a cascade when a reliable, cheap check exists;
put a gateway between applications and models for access, cost control, fallbacks and logging.

Quick check

One sentence: what's the production view of “which model?”

Pick the top of the leaderboardNot one model but a policy: which model for which query, chosen by evaluation on your data, behind a gateway that controls access, cost and failureAlways self-hostAlways use the cheapest

The model gateway, and wrap-up

Predict first

Ten teams each call a provider's API directly with a shared key. The provider changes its API. How many places need updating?

Ten, one per team, and the shared key is a leak waiting to happen; with a gateway, oneOneNoneOnly the provider's

A model gatewayStep 1 / 3

Scroll sideways to see the whole diagram

AI Engineering: a model gateway gives your organisation a unified interface to different models, commercial and self-hosted.

Text version

AI Engineering: a model gateway gives your organisation a unified interface to different models, commercial and self-hosted.
If a provider's API changes, only the gateway changes.
It centralises access: instead of handing out provider keys, you grant access to the gateway, with per-user limits, usage tracking, and fallback when a provider fails (the next lesson).

The book lists what a gateway centralises:

Access control: grant access to the gateway, not to provider keys, with fine-grained rules on who can use which model.

Cost management: monitor and limit API usage to prevent abuse and manage costs (lesson 3).

Fallbacks: when an API is down or rate-limited, route to an alternative, retry after a wait, or fail gracefully (lesson 2).

Logging and analytics, load balancing, and in some gateways caching and guardrails.

The book names off-the-shelf examples such as Portkey's AI Gateway, MLflow AI Gateway, Kong and Cloudflare, and notes gateways are relatively easy to build.

Wrap-up. From this lesson, you can now:

choose models by filtering on hard attributes, narrowing with public data, evaluating on your own, and monitoring;

account for licences and for API models that change underneath you, and pin versions;

route by intent, including cheap refusals for out-of-scope queries;

route by difficulty, and read the cost–accuracy curve a router traces;

use a cascade when a reliable, cheap check exists;

put a gateway between applications and models for access, cost control, fallbacks and logging.

Quick check

One sentence: what's the production view of “which model?”