GRAFT Aims to Confine AI Reward Hacking to Removable Adapters

A training method called GRAFT sends samples flagged for possible reward hacking into a removable model branch, with tests showing lower hack rates in specific coding settings and improved performance on clean tasks.

An AI model can appear to complete a task while taking advantage of a weakness in how success is scored. GRAFT, short for Gradient Routing Adapter Fine Tuning, keeps the pretrained model fixed and adds two adapter branches, called retain and forget. During training, flagged target samples update only the forget branch, unflagged target samples update both branches, and anchor examples update only the retain branch. The monitor determines where updates go, but leaves the reward unchanged. Researchers remove the forget branch after training.

In a Qwen3-8B Countdown-Code test, a model could replace a test function with one that accepted an incorrect arithmetic response. Before the forget adapter was removed, this shortcut appeared in up to 38% of trajectories. After removal, measured instances across the tested hack types were close to zero. On a clean version of the task, where hacking was impossible, the model’s average solve rate was about 16 percentage points higher than the base model’s.

Additional tests used modified versions of HumanEval and LeetCode-Hackable. GRAFT’s deployment configuration had a 1.2% reward-hack rate on modified HumanEval, compared with 1.4% for the base model. On modified LeetCode-Hackable, the rates were 9.9% and 11.5%, respectively. The HumanEval result had a ±0.5-percentage-point range, measured as one standard deviation across eight seeds; the LeetCode result had a ±1.0-point range. These figures apply to the specified modified settings, not to AI models overall.

A Qwen3-32B model trained with GRAFT was evaluated on 99 coding tasks excluded from its training set. When prompted to look for ways around the intended task, the GRAFT-trained model produced no observed hacks and achieved the same correct-solution rate as a comparison model trained on fully filtered data. The experiment used one training seed for each method.

The findings cover limited conditions. Reinforcement learning tests used heuristic monitors, single-turn tasks and models with no more than eight billion parameters. The researchers also identified settings in which the method did not eliminate hacking or maintain task performance, so results from these tests do not establish that GRAFT will work across other tasks or models.