Papers
Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, Madian Khabsa · arXiv preprint (GenAI at Meta), 2023
The problem: guardrails for the inputs and outputs of an AI assistant
The authors start from a recommendation in the Llama 2 responsible use guide: products based on generative AI should put guardrails around the model's inputs and outputs, both to avoid high-risk or policy-violating content and to protect against adversarial inputs and jailbreak attempts. The paper's question is how to build those guardrails. The natural starting point would be to reuse online content moderation tools, such as the Perspective API, the OpenAI moderation API and Azure Content Safety. The paper argues that they work poorly in this role for four reasons. First, none distinguishes the risk created by the user from the risk created by the agent, which are possibly different tasks: users in general ask for information and help, and the agent in general provides them. Second, each tool applies a fixed policy, so it can't be adapted to new policies. Third, they offer only API access, with no possibility of fine-tuning for a use case. Fourth, they are all built on conventional, small transformer models, which limits capability compared to more capable LLMs. The paper's answer is to use an LLM itself as the moderator's base and to publicly release the model weights.
Taxonomy of risks and the task as an instruction
The paper defines an example taxonomy, which it itself says is not intended to comprehensively measure how well the model mitigates all possible risks, but to be general enough for broad use: Violence and Hate, Sexual Content, Guns and Illegal Weapons, Regulated or Controlled Substances, Suicide and Self-Harm, and Criminal Planning. The last one is a residual category: it only applies to items that fit none of the others. Each category comes with guidelines that say where the line falls between what is permitted and what is discouraged. The safeguard task is then formulated as instruction following, and the paper identifies four ingredients. The first is the set of guidelines: numbered categories, with a free-text description of what is safe and unsafe in each, and the model must consider only those categories when judging. The second is the classification type, that is, whether the target is the user's messages (called prompts) or the agent's (called responses). The third is the conversation, which may have a single turn or several. The fourth is the output format: the model writes 'safe' or 'unsafe' and, if 'unsafe', lists on a new line the violated categories, using a letter followed by the category index, counted from 1.
Classifying the input and the output with the same model
A contribution highlighted by the authors is separating two moderation tasks: classifying the prompt (user message, LLM input) and classifying the response (agent message, LLM output). To their knowledge, the work is the first to treat the two as separate tasks. The paper makes this distinction with a single model, only changing the wording of the instruction. This makes it possible to capture the semantic difference between the user and agent roles without the effort of maintaining two classifiers. One example is in the taxonomy itself: when someone expresses or suggests intent to hurt themselves, any response that is not a referral to appropriate health resources is considered inappropriate, even if it is entirely supportive. Because the first output token is 'safe' or 'unsafe', the classifier's score can be read from the probability of that token, which allows binary and multi-label classification. The same format allows 'one-vs-all' classification: just include a single category of interest in the instruction and generate one token. As for its position in the system, Llama Guard is a separate safeguard model that judges messages in a conversation; the paper does not describe it as part of the model that generates the responses.
Instruction-configurable taxonomy: prompting and fine-tuning
Because the guidelines are part of the model's input, the policy can be swapped without training from scratch. The model is an instruction-fine-tuned Llama2-7b. The annotated set has 13,997 prompts and responses, randomly split 3:1 between fine-tuning and evaluation, and was annotated by the internal red team with category labels and safe or unsafe labels for the prompt and for the response. The authors chose the smallest of the three available Llama 2 sizes because it is friendlier to use, with potentially lower inference and deployment costs. So that the model respects only the categories present in the instruction, they used two data augmentation techniques: randomly removing non-violated categories from the prompt, and removing all violated categories, then switching the label to safe. The category indices are also shuffled, to avoid memorizing the format. When adapting to other policies there are three paths. In zero-shot, you state the target domain's category names in the instruction, with or without a description, with no training at all. In few-shot, you add 2 to 4 examples per category, including hard safe examples; learning happens in context, without training. And it is possible to do additional fine-tuning on other taxonomies, and even to decide which one to use at inference time.
Reported results and declared limitations
The evaluation uses AUPRC (area under the precision-recall curve). In the reported results, zero-shot Llama Guard with the target taxonomy obtains 0.945 on prompt classification and 0.953 on response classification on the authors' test set, versus 0.764 and 0.769 for the OpenAI API and 0.728 and 0.699 for the Perspective API. On ToxicChat, it reaches 0.626, above the compared baselines (0.588 and 0.532). On the OpenAI moderation evaluation set, it obtains 0.847 zero-shot versus 0.856 for the OpenAI API; with 2 to 4 examples in the prompt (few-shot), 0.872, beating that API on its own dataset. The authors emphasize that the internal test set follows the same taxonomy as the training, and that the others test adaptability to different taxonomies. In the limitations section, the paper acknowledges that the model's knowledge is limited by the training and pretraining data, that the fine-tuning was entirely in English, and that it does not claim perfect coverage of the policy. It also warns that, because it is an LLM, the model may be vulnerable to prompt injection that can alter or bypass its intended use, and that, if used as a chat model instead of a classifier, it may generate inappropriate content.
See the ideas at work
Practice the designs that use this paper in the System Design school.