YC Root AccessDiamond Maps: Efficient Reward Alignment for Generative Models
CHAPTERS
- 0:07 – 0:38
Diamond Maps overview: steering a base generative model toward a target task/style
Douglas introduces Diamond Maps as a method for reward alignment—taking a strong base generative model and steering it toward a user-specific objective. The motivating example is image generation where you want consistent control over a particular style or concept.
- •Defines reward alignment as steering a base model to a specific task
- •Motivating use case: better controllability for image generation
- •Example framing: aligning outputs to a desired style or prompt intent
- 0:38 – 1:49
How reward guidance depends on a value function
The conversation explains that reward alignment is typically implemented via reward guidance, which requires estimating a value function. This value function scores how well a sample matches the desired objective (e.g., prompt-image match).
- •Reward guidance requires a value function estimate
- •Value function represents “how aligned/high-reward” a sample is
- •Prompt alignment example: value tied to similarity between generated image and prompt
- 1:49 – 2:00
Why value estimation is hard: intermediate diffusion/flow states don’t look like real samples
A key obstacle is that the generative process transitions from noise to a final sample, but intermediate states are not meaningful images. Because rewards are naturally defined on final samples, intermediate-step value estimates become challenging.
- •Generative process starts from noise and ends at a data sample
- •Intermediate states (noisy images) aren’t directly scorable by typical rewards
- •You often need to reach the final sample before computing reward reliably
- 2:00 – 2:11
Limitations of existing approaches: slow or overly deterministic value estimates
Douglas contrasts existing methods that either take too long or produce weak value estimates because they’re deterministic. Determinism is problematic because one intermediate state can lead to multiple plausible final outcomes.
- •Current value-estimation methods trade off speed vs quality
- •Deterministic approaches under-sample possible outcomes from an intermediate state
- •A single roll-out can’t capture the distribution of possible final samples
- 2:11 – 2:56
Core idea of Diamond Maps: stochastic sampling of multiple finals from the same intermediate state
Diamond Maps enables sampling multiple final images conditioned on the same intermediate point in the generative process. This produces a better value function estimate by averaging reward across diverse plausible completions.
- •From one intermediate state, generate multiple possible final samples
- •Compute reward across multiple completions to improve value estimation
- •Better value estimates improve reward guidance and alignment quality
- 2:56 – 3:36
Empirical results: improved alignment metrics over vanilla flow matching
Douglas points to results showing Diamond Maps improves alignment performance compared to deterministic/vanilla baselines. The metric highlighted is LPIPS, used here as an alignment/quality proxy.
- •Diamond Maps shows better alignment performance in reported graphs
- •LPIPS is used as the key metric referenced
- •Baseline comparison: deterministic/vanilla flow matching
- 3:36 – 3:55
Qualitative demos: prompt-aligning an unconditioned ImageNet model (out-of-distribution prompts)
The talk shows sample outputs where an unconditioned ImageNet model becomes prompt-aligned. Examples include unusual compositions (e.g., “a lighthouse in the middle of a desert”) that are unlikely to occur naturally in the training distribution.
- •Demonstrates steering a previously unsteered/unconditioned model
- •Uses deliberately unlikely prompts to test controllability
- •Qualitative outputs illustrate successful prompt alignment
- 3:55 – 4:14
Two usage modes: fine-tuning vs training-free inference-time alignment
Diamond Maps is presented in two variants: a training/fine-tuning approach built on a pre-trained Flow Map, and a training-free method applied at inference time. This sets up a practical trade-off between training compute and inference compute.
- •Approach 1: fine-tune starting from a pre-trained Flow Map
- •Approach 2: training-free method used only at inference time
- •Trade-off: training-time compute vs inference-time compute
- 4:14 – 4:33
Plug-and-play integration: attach Diamond Maps to an existing Flow Map model
The inference-time method is described as something that can “plug onto” an existing trained Flow Map. The key benefit is getting stochastic sampling and improved alignment without additional training.
- •Designed to work with existing Flow Map models
- •No additional training required for the inference-time variant
- •Stochastic sampling is introduced at inference to improve guidance
- 4:33 – 4:40
Can you combine training + inference? Practical compute considerations
Ankit asks whether using Diamond Maps during both training and inference helps. Douglas suggests it likely adds compute without meaningful gains, implying the best choice depends on whether you can afford training or prefer inference-only adaptation.
- •Question: use Diamond Maps in both training and inference?
- •Answer: likely redundant—extra compute with limited benefit
- •Recommendation implied: pick the mode that fits your constraints
- 4:40 – 5:30
Scaling motivation: inference-time method enables experiments on larger models/datasets
Douglas notes fine-tuning can be prohibitively expensive at scale. The training-free inference approach enabled them to run larger text-to-image experiments (e.g., on higher-scale datasets/models) despite limited compute.
- •Fine-tuning large models is compute-intensive and not always feasible
- •Inference-time method makes scaling to larger settings easier
- •Example mentioned: larger-scale text-to-image experiments vs smaller ImageNet setup
- 5:30 – 5:35
Wrap-up and acknowledgments
Ankit closes by praising the work and thanking Douglas for presenting. The segment ends with brief thanks from both speakers.
- •Host summary/praise of the work
- •Presenter thanks
- •Conversation concludes
