Photo credit: venturebeat.com
DeepSeek AI, a Chinese research institution known for its innovative open-source language models like DeepSeek-R1, has unveiled a groundbreaking approach to reward modeling for large language models (LLMs).
Their latest method, termed Self-Principled Critique Tuning (SPCT), seeks to develop versatile and scalable reward models (RMs). This advancement holds promise for enhancing AI applications in complex and open-ended tasks, where existing models often struggle to grasp the intricacies of user interactions and contextual variations.
Understanding Reward Models and Their Constraints
Reinforcement learning (RL) has emerged as a fundamental aspect of creating cutting-edge LLMs. The RL process involves the fine-tuning of models based on feedback signals that assess the quality of their outputs.
Reward models play a pivotal role in this context, functioning as evaluators that score the outputs of LLMs and guide them in generating more valuable responses. Despite this importance, current RMs have notable shortcomings. They typically perform well only in restricted areas characterized by specific rules or easily identifiable answers. For instance, leading models like DeepSeek-R1 have been trained using RL techniques on tasks with distinctly defined correct outcomes, such as mathematical problems and coding challenges.
Nonetheless, crafting a reward model appropriate for more nuanced, subjective, or broad queries remains a significant challenge. Researchers at DeepSeek AI express that “generalist RM requires the generation of high-quality rewards beyond specific domains, where the criteria for rewards are more diverse and complex, often lacking clear reference points.”
They identify four main challenges in developing generalist RMs that can engage with a wider variety of tasks:
Input flexibility: The RM should accommodate different types of inputs and assess multiple responses at once.
Accuracy: It must ensure precise reward signals across diverse fields, particularly in situations where criteria can be complex and ground truth may be absent.
Inference-time scalability: The RM needs to improve the quality of rewards as more computational resources are applied during inference.
Learning scalable behaviors: To scale efficiently during inference, RMs should acquire behaviors that enhance performance with increased computational resources.
Different types of reward models Credit: arXiv
Broadly, reward models can be categorized based on their “reward generation paradigm” (e.g., scalar RMs that produce a single score, and generative RMs that create textual critiques) and their “scoring pattern” (e.g., pointwise scoring assigns scores to each response while pairwise scoring selects the superior response between two). These configurations significantly impact a model’s versatility for generalist applications, particularly concerning its input flexibility and ability for inference-time scaling.
For example, scalar RMs often struggle with inference-time scaling due to their tendency to provide repetitive scores, whereas pairwise RMs face challenges in evaluating single responses effectively.
Researchers suggest that “pointwise generative reward modeling” (GRM), which involves the model producing textual critiques alongside scores, can deliver the adaptability and scalability necessary for addressing generalist demands.
The DeepSeek team has undertaken initial experiments with models like GPT-4o and Gemma-2-27B. They discovered that certain guiding principles enhance reward generation within the established criteria for GRMs, leading to improved reward quality and indicating that inference-time scalability could be realized by scaling generation techniques for high-quality principles and critiques.
Developing RMs Capable of Self-Generated Principles
They advocate for incorporating principles directly into the reward generation process, rather than treating them as a preliminary step. This integration allows GRMs to dynamically create principles tailored to specific tasks and subsequently generate critiques based on those principles.
“This transformation permits principles to be generated in response to the input query and responses, which adaptively aligns the reward generation process, further enhancing the quality and detail of both the principles and the critiques through post-training,” the researchers explain.
Self-Principled Critique Tuning (SPCT) Credit: arXiv
SPCT consists of two core phases:
Rejective fine-tuning: In this phase, the GRM is trained to produce principles and critiques tailored to various input types. The model generates principles, critiques, and rewards for specific queries and responses, with generation attempts being accepted only when the predicted reward matches the ground truth. This iterative process enhances the model’s ability to generate effective principles and critiques.
Rule-based RL: This phase further refines the model using outcome-based reinforcement learning. The GRM generates principles and critiques for each query, with reward signals computed based on straightforward accuracy rules. The model is updated based on these evaluations, fostering the ability to dynamically create effective principles and critiques.
“Utilizing rule-based online RL, SPCT allows GRMs to learn adaptively to posit principles and critiques according to the input queries and responses, leading to enhanced outcome rewards in more general domains,” the researchers note.
To address the challenge of inference-time scaling, researchers have the GRM perform multiple evaluations on the same input, generating distinct sets of principles and critiques. The final reward is determined by a voting mechanism that aggregates scores, enabling the model to consider a wider array of perspectives and potentially leading to more nuanced evaluations as computational resources are increased.
Recognizing that some generated critiques may be inconsistent or biased due to limited model capabilities or randomness, the team introduced a lightweight “meta RM.” This model is specifically trained to predict whether a principle or critique produced by the primary GRM is likely to yield a correct final reward. During inference, the meta RM assesses generated samples, filtering out low-quality judgments before the final aggregation, thereby enhancing overall performance.
Implementing SPCT in DeepSeek-GRM
The researchers applied SPCT to create DeepSeek-GRM-27B from Gemma-2-27B, Google’s open-weight model. They conducted evaluations against several robust baseline RMs (including models like LLM-as-a-Judge and various scalar RMs) and public models such as GPT-4o and Nemotron-4-340B-Reward across diverse benchmarks.
The findings indicate that DeepSeek-GRM-27B outperforms baseline methods that were trained on the same data, demonstrating significant improvements in both reward quality and scalability when compared to traditional fine-tuning approaches.
The performance of DeepSeek-GRM (trained with SPCT) continues to improve with inference-time scaling Credit: arXiv
As the model’s performance was enhanced by additional sampling during inference, DeepSeek-GRM-27B exceeded even larger models like Nemotron-4-340B-Reward and GPT-4o. The introduction of the meta RM proved beneficial by refining outcome judgments, producing the best results through effective filtering.
“With extensive sampling, DeepSeek-GRM is capable of providing more accurate judgments based on diverse principles, thereby yielding rewards with greater precision,” the researchers assert.
Notably, SPCT exhibited less bias across various domains compared to scalar RMs, which often excelled in verifiable tasks but performed poorly in more subjective areas.
Enterprise Implications of Reward Model Advancements
The evolution toward more generalist and scalable reward models presents promising opportunities for enterprise AI applications. Potential areas for the implementation of generalist RMs include creative tasks and scenarios requiring adaptation to shifting environments, such as fluctuating consumer preferences.
Despite the promising results, DeepSeek-GRM still cannot compete with specialized scalar RMs in purely objective tasks, where explicit reasoning processes may outperform direct scoring methods. Efficiency also remains a challenge compared to non-generative RMs.
The team at DeepSeek intends to pursue further developments focused on enhancing efficiency and fostering deeper integrations. They conclude by outlining that “future endeavors could involve embedding GRMs within online RL frameworks, optimizing co-scaling with policy models, or establishing robust offline evaluators for foundational models.”
Source
venturebeat.com