This paper introduces a new language model called RHO-1 that employs Selective Language Modeling (SLM). Unlike traditional language models that learn to predict every next token in a corpus, RHO-1 selectively trains on useful tokens that align with the desired distribution. The main idea is that not all tokens in a corpus are equally important for language model training.
Rho-1: Not All Tokens Are What You Need
Rho-1: Not All Tokens Are What You Need
Rho-1: Not All Tokens Are What You Need
This paper introduces a new language model called RHO-1 that employs Selective Language Modeling (SLM). Unlike traditional language models that learn to predict every next token in a corpus, RHO-1 selectively trains on useful tokens that align with the desired distribution. The main idea is that not all tokens in a corpus are equally important for language model training.