How rollout-1 guidance is calculated

How many updates happen?

Cap-50 means up to 50 guidance updates at each denoising step where guidance is enabled. All 50 happen at the same noise level. Each update repeats steps 1–5 below. Then step 6 lowers the noise level.

The tuned Case 2 video uses cap-50 during the last 25 of 50 denoising steps: up to 1,250 guidance updates. Its guidance strength decreases from 0.7 to 0.2, except that the last denoising step uses 0.10.

The ten-video comparison uses cap-75 and the following settings:

Video	5 balls; 49 frames at 128 × 128 pixels
Given frames	First 5 video frames; their first 2 latent frames stay fixed
Denoising steps	\(K\in\{10,20,50,100,200\}\); random seed 0
When guidance runs	Step indices 25 through \(K-1\), counting from zero
Update cap	Up to 75 guidance updates at each of those steps
Frame to update	The frame with the lowest error above the stopping threshold
Stopping threshold	Half the lowest error at the start of that denoising step; at least \(10^{-6}\)
Guidance strength	Decreases linearly from 0.7 to 0.2 while guidance is enabled
Loss and update	Mean squared error at one latent time slice; the gradient passes through the DiT and can change all future latents
Sampler	Flow matching, shift 5, denoising strength 1; empty prompts and classifier-free guidance scale 1

The 10- and 20-step runs finish before guidance starts. All ten videos use the same settings.

1. Predict the clean video

Before denoising starts, encode the input video with the VAE. Keep its first two latent frames fixed and fill the remaining latents with Gaussian noise. For the ten-video comparison, reproduce each unguided run’s starting noise by drawing the same batch of noise and taking the same item from it.

At every guidance update, run the DiT on the current noisy latents \(z\). At denoising step \(j\), the model output \(v\) gives the clean-latent prediction \(\hat z_0\):

Here, \(\sigma_j\) is the noise level and \(t_j\) is the DiT’s timestep input. Decode \(\hat z_0\) into video with the VAE, then restore the five given frames. Use this video to find the balls and build the physics target. Gradients do not pass through this decoding or target-building process.

2. Choose the frame to update: lowest error above the threshold

Choose the frame to update with the lowest Rollout-1 error above the stopping threshold. For example, use the balls’ estimated positions and velocities at frame 12 to simulate where they should be at frame 13. Compare those positions with the model’s frame 13 to calculate its error.

Find the balls in each frame and match them across frames by color. Record their centers, including any extra balls, and note any missing balls or wrong colors. Convert pixel positions to the simulator’s coordinates.

To estimate velocities, find stretches of at least three consecutive frames containing exactly the five expected balls with the correct colors. Estimate velocity from how far each ball moves between frames. At the first frame of each stretch, use the movement from the first to the third frame divided by the elapsed time. Then use the simulator to adjust the velocity estimates for collisions.

Consider frames \(q\in\{5,\ldots,48\}\), counting from zero. Frame \(q\) can be chosen if the preceding frame, \(q-1\), has position and velocity estimates. Simulate one frame forward from those estimates. For each ball, measure the distance between its simulated and predicted centers at frame \(q\). Their average is the error \(e_q\).

A missing or wrong-color ball gets an error of \(\sqrt{200}\approx14.1421\), the diagonal length of the simulation area. An extra ball gives all five balls that error. The simulator uses a 10 × 10 area, ball radius 0.7, radius adjustment −0.05, and \(1/15\) second between frames.

At the start of each denoising step, set the stopping threshold to half the lowest error, but no lower than \(10^{-6}\). Keep that threshold fixed throughout the guidance updates at this noise level:

\(S_{j,0}\) lists the frames that can be chosen at the start of the denoising step; \(S\) lists them at the current update. After every update, find the balls again, recalculate the errors, and choose a frame again. The same frame can be chosen repeatedly. Break exact ties randomly, using seed 0.

3. Build the target video and encode it

Redraw the balls found in every frame on the simulator’s background, including any extras. In the chosen frame \(q^*\), replace that drawing with the five balls at their simulated positions. Leave the other redrawn frames as they are. This is the reference video.

Scale its RGB values to \([-1,1]\), then encode all 49 frames with the VAE. Tiling is disabled, so the VAE processes the video without splitting it into tiles:

Keep these reference latents \(r\) fixed when calculating the gradient. Rebuild the reference from the new prediction at the next guidance update.

4. Compare the predicted and target latents

The VAE compresses the 49 video frames into 13 latent time slices. Each slice contains 16 channels on a 16 × 16 grid. The code assigns the chosen video frame \(q^*\) to slice \(\ell\):

At that slice, subtract the reference latents from the predicted clean latents, square the differences, and average all 4,096 values. This is the mean squared error (MSE) loss \(L\), calculated in float32:

The ball-position error chooses the frame; the latent MSE supplies the gradient. The Rollout-1 and Rollout-5 metrics shown under the videos are measured separately on the finished clips.

5. Use the gradient to update the noisy latents

Calculate the gradient of \(L\) with respect to the noisy latents \(z\). This passes through the DiT’s clean prediction \(\hat z_0=z-\sigma_jv_\theta(z,t_j)\), with the reference held fixed:

The loss compares one time slice, but its gradient can change all future latent slices because the DiT connects them. Set the gradient to zero for the first two slices, which hold the given frames.

Divide the remaining gradient by its root mean square (RMS), with a minimum divisor of \(10^{-6}\). Multiply it by the guidance strength, the denoising step size, and \(\sqrt{L}\). Subtract this correction from the noisy latents:

Here, \(P_F\) keeps the gradient at future slices 2–12 and zeros the first two slices. \(|F|=45{,}056\) is the number of future latent values. \(\operatorname{restore}_C\) puts the two given slices back after the update. The strength \(\gamma_j\) follows the ten-video settings above; the tuned Case 2 run uses 0.10 at the last step. The RMS is calculated in float32, then converted to the gradient’s data type before division.

Apply each update directly, without checking whether to accept or undo it. Start a fresh gradient calculation at the next update. Repeat at the same noise level until reaching the cap (50 or 75), no frame can be chosen, or all remaining errors are at or below \(\tau_j\).

6. Take one denoising step

Once the guidance updates finish, run the DiT again if the latents have changed since its last call. Use its output to take the ordinary denoising step to the next noise level:

When guidance is disabled, run the DiT once and take this denoising step directly. The final noise level is \(\sigma_K=0\). After the last step, decode the latents with the VAE, put the five given video frames back, and save the 49-frame video.

Where the ten-video results come from

The ten-video viewer uses saved clips from out/guidance/best-rollout1-matched-10samples-5steps-v1/case-{case}/steps-{K}/guided-final.mp4. Each of the ten cases was run at five denoising step counts. All 50 runs used the same guidance settings; the viewer does not pick a best result from several variants.

These are the recorded counts per video:

Denoising steps	Guidance updates	DiT calls
10	0	10
20	0	20
50	1,875	1,925
100	5,580–5,625	5,680–5,725
200	13,125	13,325

A range means the count differs between videos because some guidance loops stop early. Each guidance update also VAE-encodes one reference video and calculates a gradient through the DiT.

Code and saved run files

This description is based on code revision 06c91ab and the saved settings and logs for the 50 rollout-1 runs in the ten-video viewer.

• configs/guidance/smallest_error_rollout_h1_best_u75.yaml: explicit guidance settings; defaults are in src/sshv2/guidance/latent_rollout_guidance.py.
• scripts/sshv2/run_best_rollout1_matched_matrix.py: ten source videos, five step counts, and noise replay arguments.
• scripts/sshv2/run_rollout1_single_source_case.py: released bidirectional checkpoint, conditioning, task constants, generation, and video saving.
• src/sshv2/diffsynth/pipelines/wan_video_guided.py: _smallest_error_direct_call, _smallest_error_endpoint_gradient, _single_reference_latent_slice_loss, _five_frame_window_latent_interval, _normalised_correction, and _prepare_inputs.
• src/sshv2/guidance/rollout_guidance.py: build_smallest_error_endpoint_reference and RolloutGuideContext.from_bounce_task.
• src/sshv2/simulation/reliable_rollout.py: tracking, local penalized scores, simulation, and build_single_rollout_endpoint_reference_videos; velocity repair is in src/sshv2/simulation/fix.py.
• src/sshv2/diffsynth/models/wan_video_vae.py and lib/diffsynth/diffsynth/schedulers/flow_match.py: frame mapping, VAE operations, and scheduler update.
• out/guidance/best-rollout1-matched-10samples-5steps-v1/: each case/step directory contains run.json, diagnostics.json, and guided-final.mp4.

Return to the video comparison →