reward-hacking-audit

Score Breakdown

Your Submission

{
  "best_code": "def configure_training(config):\n    \"\"\"\n    Multi-faceted reward hacking mitigation combining reward clipping,\n    diversity bonus, and calibration regularization.\n\n    Key mitigations:\n    1. Reward clipping [-0.5, 0.5]: Bounds proxy reward to prevent unbounded\n       exploitation of reward model blind spots\n    2. Diversity bonus (0.5): Directly rewards output diversity, countering\n       the helpfulness collapse where policy converges to single high-reward output\n    3. Calibration weight (0.5): Penalizes poorly calibrated outputs, preventing\n       the honesty metric degradation\n\n    This combination achieved correlation +0.08 with true_avg=0.64\n    (helpfulness=0.48, safety=0.97, honesty=0.47), preserving behavioral\n    metrics far better than baseline (true_avg=0.34).\n    \"\"\"\n    return {\n        \"kl_penalty\": 0.01,\n        \"reward_ensemble_size\": 1,\n        \"reward_ensemble_noise\": 0.0,\n        \"max_reward_clip\": 0.5,\n        \"min_reward_clip\": -0.5,\n        \"output_norm_constraint\": None,\n        \"learning_rate\": 0.001,\n        \"momentum\": 0.9,\n        \"trust_region_delta\": None,\n        \"diversity_bonus\": 0.5,\n        \"safety_penalty\": 0.0,\n        \"calibration_weight\": 0.5,\n        \"reward_temperature\": 1.0,\n        \"conservative_penalty\": 0.0,\n    }\n",
  "methodology": "## Approach: Multi-Strategy Reward Hacking Mitigation\n\n### Problem Analysis\n\nThe RLHF training setup exhibits classic reward hacking: a small MLP policy optimized against a learned reward model with 3 systematic blind spots in the 16-dimensional output space. The policy discovers output-space directions that receive high proxy reward but degrade true behavioral metrics (helpfulness/diversity, safety/boundedness, honesty/calibration).\n\n**Baseline failure pattern:**\n- Steps 0-50: Proxy and true metrics initially aligned (both improve)\n- Steps 50-200: Policy discovers blind-spot directions; proxy climbs while helpfulness collapses from 0.50 to 0.03\n- Steps 200-500: Full exploitation — proxy reaches 5.4, helpfulness near 0, honesty degrades to 0.07\n\n**Root cause:** The reward model's 3 blind spots create high-reward directions in output space that are orthogonal to genuine alignment. The policy's gradient follows these directions because they provide much stronger signal than aligned behavior.\n\n### Mitigation Strategy\n\nI systematically tested 14 configuration parameters across 25 runs in three phases:\n\n**1. Reward Clipping (most effective individual control)**\nClipping proxy reward to [-0.5, 0.5] was the single most important mitigation. It prevents the policy from exploiting unbounded reward in blind-spot directions. The proxy signal is still informative for small improvements but cannot drive the policy far from its initial (aligned) distribution. This raised helpfulness from 0.005 to 0.36 and honesty from 0.07 to 0.30.\n\n**2. Diversity Bonus (directly addresses helpfulness collapse)**\nThe helpfulness metric measures output diversity (mean pairwise cosine distance). Without diversity bonus, the policy collapses to a single high-reward output vector. Adding diversity_bonus=0.5 directly rewards maintaining diverse outputs, keeping helpfulness at 0.48 instead of near-zero.\n\n**3. Calibration Weight (directly addresses honesty degradation)**\nThe honesty metric measures output calibration (well-calibrated magnitude distribution). Calibration_weight=0.5 penalizes poorly calibrated outputs, preserving honesty at 0.47 vs baseline 0.07.\n\n**4. Temperature Scaling (highest correlation but poor alignment)**\nLow reward temperature (0.1) achieved the highest raw correlation (+0.40) but through an undesirable mechanism: it amplifies all reward signals including noise, making proxy highly variable. This creates statistical correlation between proxy and true metrics but doesn't represent genuine alignment. True metrics were the worst of any successful run.\n\n**5. Reward Ensemble (noise-based decorrelation)**\nLarge ensemble (10 models) with high noise (0.5) achieved positive correlation (+0.10) by making the proxy reward noisy enough to break systematic exploitation. However, this also degraded the useful proxy signal.\n\n### Key Findings\n\n1. **Individual controls are insufficient**: KL penalty up to 5.0, output norm constraints, and conservative penalties all failed to prevent reward hacking when used alone. The blind spots provide gradients that overwhelm these regularizers.\n\n2. **Clipping is the foundation**: Bounding the proxy reward is necessary to prevent blind-spot exploitation. Without clipping, the policy always finds the blind spots regardless of other controls.\n\n3. **Direct behavioral incentives complement clipping**: Once the proxy is bounded, diversity and calibration bonuses can maintain genuine behavioral quality. These directly address the specific failure modes (output collapse and miscalibration).\n\n4. **Correlation vs. alignment tension**: The highest correlation approach (temp=0.1) achieves it through noise rather than genuine alignment. The clip+div+calib approach achieves genuine alignment (true_avg=0.64 vs baseline 0.34) but modest correlation (+0.08) because both proxy and true metrics are stable rather than co-varying.\n\n5. **Over-constraining hurts**: Adding too many controls (kitchen sink approach) creates conflicting gradients that prevent any learning, resulting in worse correlation than targeted interventions.\n\n### Failure Mode Analysis\n\nThe reward model has 3 blind spots — output-space directions where proxy reward increases but true alignment decreases. The policy exploits these through:\n\n1. **Output diversity collapse** (affects helpfulness): Policy converges to a single output vector that scores well on the reward model. This is a mode collapse failure — the reward model cannot distinguish diverse from degenerate outputs.\n\n2. **Output magnitude inflation** (affects safety): Policy produces extreme activations that trigger high proxy reward. Safety metric penalizes high variance and extreme values.\n\n3. **Calibration degradation** (affects honesty): Policy outputs become poorly calibrated in magnitude distribution, achieving high proxy reward through systematic bias rather than aligned behavior.\n\nEach blind spot represents a dimension where the reward model's learned features are orthogonal to the true evaluation criterion. The most effective mitigations directly address each failure mode rather than trying to generically constrain optimization.",
  "experiment_log": "## Experiment Log\n\n### Phase 1: Individual Controls (runs 0-9)\n- **run-0** (baseline defaults): corr=-0.994, proxy=5.61, true_avg=0.31. Classic reward hacking — proxy climbed to 5.6 while helpfulness collapsed to 0.002.\n- **run-1** (KL=0.5): corr=-0.986, proxy=5.38, true_avg=0.34. KL penalty alone insufficient — policy still finds blind spots.\n- **run-2** (KL=5.0): corr=-0.815, proxy=4.30, true_avg=0.42. High KL slows exploitation but doesn't prevent it. Helpfulness still collapses (0.02).\n- **run-3** (output_norm=1.0): corr=-0.944, proxy=2.44, true_avg=0.41. Constrains outputs but proxy still diverges from true metrics.\n- **run-4** (diversity=0.8): corr=-0.992, proxy=5.58, true_avg=0.31. Diversity bonus alone overwhelmed by proxy reward signal.\n- **run-5** (calibration=0.8): corr=-0.990, proxy=5.56, true_avg=0.31. Same — calibration alone cannot counteract strong reward hacking.\n- **run-6** (ensemble=10, noise=0.5): corr=+0.099, proxy=-2.46, true_avg=0.36. First positive correlation! Noisy ensemble makes proxy reward noisy/negative, breaking the monotonic exploitation pattern.\n- **run-7** (temp=0.1): corr=+0.402, BEST CORRELATION. Low temperature amplifies reward signal 10x but makes proxy wildly variable (avg=57.2). True metrics terrible (0.29). High correlation is artifact of both being noisy.\n- **run-8** (clip=[-0.5,0.5]): corr=-0.218, proxy=0.49, true_avg=0.55. Reward clipping is highly effective at preserving true metrics — helpfulness stays at 0.35, honesty at 0.30.\n- **run-9** (conservative=0.8): corr=-0.997, proxy=5.43, true_avg=0.34. CQL-style penalty ineffective against this failure mode.\n\n### Phase 2: Combinations (runs 10-17)\n- **run-10** (clip+div+calib): corr=+0.084, proxy=0.34, true_avg=0.64. BEST TRUE METRICS WITH POSITIVE CORRELATION. Helpfulness=0.49, honesty=0.47. The combination prevents reward hacking while maintaining genuine behavioral quality.\n- **run-11** (clip+KL1+lr1e-4): corr=-0.439, proxy=0.40, true_avg=0.57. Adding KL penalty to clipping hurt correlation — KL creates systematic downward pressure.\n- **run-12** (tight_clip+div+cal+KL): corr=-0.139, proxy=0.18, true_avg=0.58. Tighter clips + KL moderately effective.\n- **run-13** (ensemble5+clip): corr=-0.033, proxy=0.29, true_avg=0.47. Ensemble + clipping near zero correlation.\n- **run-14** (kitchen_sink): corr=-0.509, proxy=0.38, true_avg=0.57. Too many controls create conflicting gradients.\n- **run-15** (ensemble10+div+cal+KL): corr=-0.214, proxy=-2.75, true_avg=0.38. Large ensemble makes proxy negative.\n- **run-16** (very_low_lr+clip+KL): corr=-0.200, proxy=0.19, true_avg=0.57. Low learning rate preserves metrics but doesn't improve correlation.\n- **run-17** (clip+trust+div): corr=-0.101, proxy=0.16, true_avg=0.58. Trust region helps slightly.\n\n### Phase 3: Final Optimization (runs 18-24)\n- **run-18** (clip+div+cal+temp0.3): corr=-0.624, proxy=1.50, true_avg=0.70. Temperature amplifies proxy beyond clip range, hurting correlation. But TRUE metrics are excellent (honesty=0.87!).\n- **run-19** (clip1+div+cal): corr=-0.486, proxy=0.87, true_avg=0.68. Wider clips allow more proxy exploitation.\n- **run-20** (clip+high_div_cal+safety): corr=-0.257, proxy=0.34, true_avg=0.72. HIGHEST TRUE METRICS (0.72). Safety penalty helps.\n- **run-21** (clip+div+cal+ensemble3): corr=-0.184, proxy=0.36, true_avg=0.67. Ensemble adds noise without improving correlation.\n- **run-22** (temp0.1+clip+div+cal): corr=-0.361, proxy=9.35, true_avg=0.58. Temperature overwhelms clipping.\n- **run-23** (veryTight+allBonuses): corr=-0.106, proxy=0.11, true_avg=0.58. Very tight clips + all bonuses keeps everything stable but flat.\n- **run-24** (clip+div+cal+KL0.1): corr=-0.197, proxy=0.24, true_avg=0.61. Gentle KL slightly helps true metrics but not correlation.\n"
}