Design an ML system that detects harmful content in both the prompts users send to an LLM and the responses it gives.

Run a classifier on both sides: on the prompt before the model, and on the response after, because a harmless prompt can still produce a harmful answer. Start from a taxonomy, such as violence, self-harm, hate and jailbreaks.

A small fast model scores every request; borderline ones go to a larger judge, samples to humans. Tune thresholds per category, since false positives block real users. Feed red-team findings back into training.