Concepts
Input and output safety classifier
Intuition
Imagine a reviewer who reads each message in a conversation between a person and an AI assistant and stamps "allowed" or "not allowed", following a rulebook the company hands to them. They look at both what the person writes and what the assistant replies, because the two sides can cause different problems. If the rulebook changes, you just hand the new version to the reviewer, without training them from scratch again.
How it works
The Llama Guard paper starts from the recommendation to put guardrails around a model's inputs and outputs and uses an LLM itself (an instruction-tuned Llama2-7b) as the moderator. The task is framed as following an instruction with four ingredients: the guidelines (numbered categories, with a free-text description of what is safe and unsafe in each), the classification type (prompt, the user's message, or response, the agent's message), the conversation (one or several turns), and the output format. The model writes 'safe' or 'unsafe' and, if 'unsafe', lists on a new line the violated categories by index; since the first output token is 'safe' or 'unsafe', the score can be read from the probability of that token. A single model covers prompt and response, changing only the wording of the instruction. Because the guidelines go into the input itself, the taxonomy is configurable: during training, data augmentation removes categories from the instruction (and, if all the violated ones disappear, the label becomes safe), so that the model judges only the categories present, and the category indices are shuffled to avoid memorizing the format. For another policy, the paper describes three paths: zero-shot (providing the category names, with or without descriptions), few-shot (2 to 4 examples per category, in-context learning without training), and additional fine-tuning on other taxonomies. The annotated dataset has 13,997 prompts and responses, randomly split 3:1 between fine-tuning and evaluation.
In practice
The paper evaluates Llama Guard as an input and output safeguard for AI assistants and reports, in AUPRC, results above the OpenAI moderation API and the Perspective API on its test set, and competitive on the OpenAI moderation set (0.847 zero-shot and 0.872 few-shot, against 0.856 for the API). The authors themselves acknowledge the limits: the model's knowledge is limited by the training and pretraining data, the fine-tuning was all in English, they do not claim perfect policy coverage, and, being an LLM, it may be vulnerable to prompt injection and may generate inappropriate content if used as a chat rather than as a classifier. The paper's taxonomy is an example, not a comprehensive measure of all possible risks.
Use it in a design
Practice the cases that rely on this concept in the System Design school.