Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training
Today's paper introduces a novel approach called Decoupled Refusal Training (DeRTa) to improve the safety of large language models (LLMs). The method enables LLMs to refuse generating unsafe content at any point in their responses, not just at the beginning, thus enhancing the models' ability to recognize and halt potential risks throughout the generation process.
Method Overview
The DeRTa method introduces two key components to enhance LLM safety:
First, it uses Maximum Likelihood Estimation (MLE) with Harmful Response Prefix. This involves appending a segment of a harmful response to the beginning of a safe response during training. By doing so, the model learns to transition from potentially unsafe content to safe responses at any point. This provides additional context for the model to identify and avoid unsafe content.
Second, it employs Reinforced Transition Optimization (RTO). This technique trains the model to transition from potential harm to safety refusal at every position within the harmful response sequence. It reinforces the model's ability to consistently recognize and prevent potential threats throughout the generation process.
The combination of these two components allows LLMs to develop a more robust understanding of unsafe content and the ability to refuse or redirect at any point during response generation. This approach differs from standard safety tuning methods that typically only train models to refuse at the beginning of responses.
By embedding constructed harmful responses within the training process, DeRTa explicitly teaches LLMs to refuse compliance at any response position. This mitigates the refusal position bias present in conventional safety tuning data and equips models with more flexible and reliable safety mechanisms.
Results
The paper demonstrates significant improvements in model safety across multiple LLM architectures, including LLaMA3 and Mistral models. Key findings include:
Substantial reduction in Attack Success Rate (ASR) across various attack scenarios without compromising helpfulness.
Effectiveness in defending against advanced attack methods like CodeAttack and CompletingAttack, which have proven challenging for even highly secure systems.
Ability to halt the generation of unsafe content even after initially beginning to produce it, showcasing improved safety throughout the response generation process.
Conclusion
The DeRTa method enables models to refuse answering and generating unsafe content at any point in their responses. For more information please consult the full paper.
Congrats to the authors for their work!
Yuan, Youliang, et al. "Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training." arXiv preprint arXiv:2407.09121 (2024).