Edwin Chen: Why optimizing for benchmarks creates AI sloth

Surge surpassed $1B in revenue with under 100 people by deliberately avoiding the typical Silicon Valley game—no VC, minimal PR—and instead relying on a small, elite, deeply aligned team shipping a 10x better product.

Good training data is not just ‘correct format and instructions followed’; Surge uses thousands of behavioral and performance signals to identify not just acceptable work but “best of the best” contributors for complex tasks like Nobel‑level poetry, advanced coding, and scientific reasoning.

Many benchmarks are noisy or wrong, easy to game, and correlate poorly with real‑world capability; optimizing for them often rewards flashy, emoji‑laden, hallucination‑prone outputs that win votes but degrade truthfulness and reliability.

Labs choose what to optimize—engagement, benchmarks, productivity, safety, taste—and those choices drive data selection, post‑training, and ultimately model personality, such as whether an assistant endlessly polishes an email or tells you, “It’s good enough, move on.”

Beyond SFT and RLHF, Surge is building simulated ‘worlds’ (e.g., full startup stacks, financial workflows) where models must act over long trajectories, use tools, and affect real state, receiving rewards for end‑to‑end task success rather than single‑step answers.