Skip to content

fix: match reference first-chunk semantics in Wan VAE temporal upsample - #21

Merged
olyasir merged 2 commits into
2026-07-03from
qvac/wan-vae-first-chunk-fix
Jul 17, 2026
Merged

fix: match reference first-chunk semantics in Wan VAE temporal upsample#21
olyasir merged 2 commits into
2026-07-03from
qvac/wan-vae-first-chunk-fix

Conversation

@olyasir

@olyasir olyasir commented Jul 17, 2026

Copy link
Copy Markdown

Problem

In Resample::forward (upsample3d, src/wan.hpp), the first latent-frame chunk skips time_conv entirely and never seeds the temporal feat cache, so the second chunk runs time_conv against zero padding instead of chunk-0 context.

The reference implementation (Wan vae2_2.py, "Rep" cache semantics) instead:

  • chunk 0: runs time_conv with causal zero padding, doubles the frames, trims the first one
  • every chunk (including chunk 0): stores the input tail in feat_cache for the next chunk

Measured effect on Wan2.2 VAE decode vs PyTorch (5-frame 480×832 video, CPU backend, f16 weights): cosine 0.9959 / 27.2 dB PSNR. The output is visually near-identical, so this never shows up in eyeball testing — only in numeric parity. The deviation is concentrated in the first frames of a decode, which matters for any autoregressive/streaming use where early frames feed later context.

Note: upstream leejet/stable-diffusion.cpp has the same logic (src/model/vae/wan_vae.hpp), so this is worth an upstream PR too.

Fix

Mirror the reference semantics exactly (see inline comment referencing vae2_2.py Resample.forward). Encode and TAEHV paths are untouched.

Verification

Measured with an out-of-tree parity harness: it loads the VAE/TAEHV file, runs the real engine graph (CPU) on a raw tensor dumped from the Wan reference implementation, and compares outputs directly — no diffusion model involved. Test input: 5-frame 480×832 video → 2 latent frames.

Results with Wan2.2 VAE + taew2_2 f16 GGUFs (CPU, vs fp32 PyTorch):

Path cosine PSNR
VAE decode (before fix) 0.9959 27.2 dB FAIL
VAE decode (after fix) 1.000000 79.0 dB PASS
VAE encode 0.999998 89.7 dB PASS (unchanged)
taew2_2 decode 0.99955 37.3 dB PASS (unchanged)

In Resample::forward (upsample3d), the first latent-frame chunk skipped
time_conv entirely and never seeded the temporal feat cache, so the second
chunk ran time_conv against zero padding instead of chunk-0 context.

The reference implementation (Wan vae2_2.py, "Rep" cache semantics) runs
time_conv with causal zero padding on chunk 0, doubles the frames, trims
the first one, and stores the input tail in the cache for the next chunk.

Effect on Wan2.2 VAE decode vs the PyTorch reference (5-frame 480x832,
CPU backend, f16 weights): cosine 0.9959 / 27.2 dB PSNR before, cosine
1.000000 / 79.0 dB after. Output was visually near-identical, so this
only shows up in numeric parity testing. Encode and TAEHV paths are
unaffected.
@olyasir
olyasir force-pushed the qvac/wan-vae-first-chunk-fix branch from 84dd6fc to cc76eb1 Compare July 17, 2026 07:54
Comment thread src/wan.hpp Outdated
feat_idx += 1;
if (chunk_idx == 0) {
// feat_cache[idx] == nullptr, pass
auto time_conv = std::dynamic_pointer_cast<CausalConv3d>(blocks["time_conv"]);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you check if this thing even exists before trying to access time_conv.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e17495a — lookup via find() with GGML_ASSERT on both the map entry and the cast result. Re-ran the parity check after the change: decode still cosine 1.00000 vs the PyTorch reference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants