fix: match reference first-chunk semantics in Wan VAE temporal upsample - #21
Merged
Conversation
In Resample::forward (upsample3d), the first latent-frame chunk skipped time_conv entirely and never seeded the temporal feat cache, so the second chunk ran time_conv against zero padding instead of chunk-0 context. The reference implementation (Wan vae2_2.py, "Rep" cache semantics) runs time_conv with causal zero padding on chunk 0, doubles the frames, trims the first one, and stores the input tail in the cache for the next chunk. Effect on Wan2.2 VAE decode vs the PyTorch reference (5-frame 480x832, CPU backend, f16 weights): cosine 0.9959 / 27.2 dB PSNR before, cosine 1.000000 / 79.0 dB after. Output was visually near-identical, so this only shows up in numeric parity testing. Encode and TAEHV paths are unaffected.
olyasir
force-pushed
the
qvac/wan-vae-first-chunk-fix
branch
from
July 17, 2026 07:54
84dd6fc to
cc76eb1
Compare
aegioscy
reviewed
Jul 17, 2026
| feat_idx += 1; | ||
| if (chunk_idx == 0) { | ||
| // feat_cache[idx] == nullptr, pass | ||
| auto time_conv = std::dynamic_pointer_cast<CausalConv3d>(blocks["time_conv"]); |
There was a problem hiding this comment.
can you check if this thing even exists before trying to access time_conv.
Author
There was a problem hiding this comment.
Done in e17495a — lookup via find() with GGML_ASSERT on both the map entry and the cast result. Re-ran the parity check after the change: decode still cosine 1.00000 vs the PyTorch reference.
aegioscy
requested changes
Jul 17, 2026
aegioscy
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In
Resample::forward(upsample3d,src/wan.hpp), the first latent-frame chunk skipstime_conventirely and never seeds the temporal feat cache, so the second chunk runstime_convagainst zero padding instead of chunk-0 context.The reference implementation (Wan
vae2_2.py, "Rep" cache semantics) instead:time_convwith causal zero padding, doubles the frames, trims the first onefeat_cachefor the next chunkMeasured effect on Wan2.2 VAE decode vs PyTorch (5-frame 480×832 video, CPU backend, f16 weights): cosine 0.9959 / 27.2 dB PSNR. The output is visually near-identical, so this never shows up in eyeball testing — only in numeric parity. The deviation is concentrated in the first frames of a decode, which matters for any autoregressive/streaming use where early frames feed later context.
Note: upstream
leejet/stable-diffusion.cpphas the same logic (src/model/vae/wan_vae.hpp), so this is worth an upstream PR too.Fix
Mirror the reference semantics exactly (see inline comment referencing
vae2_2.py Resample.forward). Encode and TAEHV paths are untouched.Verification
Measured with an out-of-tree parity harness: it loads the VAE/TAEHV file, runs the real engine graph (CPU) on a raw tensor dumped from the Wan reference implementation, and compares outputs directly — no diffusion model involved. Test input: 5-frame 480×832 video → 2 latent frames.
Results with Wan2.2 VAE + taew2_2 f16 GGUFs (CPU, vs fp32 PyTorch):