feat(grpo): in-kernel force-on-policy TIS weight - #8
Open
dongseokmotif wants to merge 2 commits into
Open
Conversation
Support force_on_policy_ratio with the fused chunked GRPO loss. When old_per_token_logps is None the kernel already sets old := curr.detach() (=> PPO ratio == 1, truly on-policy). Add force_on_policy_tis: when set, the vllm_is_ratio slot CARRIES the generation logprobs and the kernel forms the TIS weight exp(old - gen) == exp(curr - gen) internally, clamped to truncated_importance_sampling_ratio. Needed because curr only exists inside the fused chunk, so the caller cannot precompute it. Threaded as fixed scalar kwargs (mirroring entropy_coef) through LigerFusedLinearGRPOLoss.forward -> Function.forward/backward -> base.forward -> _compute_chunk_loss -> ppo_loss_fn; gen rides the existing vllm_is_ratio chunked slot. Both params default off => backward-compatible. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…eight Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dongseokmotif
force-pushed
the
feat/grpo-entropy-bonus
branch
from
June 14, 2026 05:25
9982957 to
3a8fd70
Compare
dongseokmotif
force-pushed
the
feat/grpo-force-on-policy
branch
from
June 14, 2026 05:25
7a47e5b to
de93042
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
main기준으로 force-on-policy TIS만 담도록 재작성한 PR입니다 (base를 entropy 브랜치 →main으로 변경, clamp/entropy 커밋 제거).Supports an in-kernel force-on-policy truncated-importance-sampling (TIS) weight for the fused chunked GRPO loss. When
old_per_token_logps is Nonethe kernel setsold := curr.detach()(PPO ratio == 1, truly on-policy). Withforce_on_policy_tis=True, thevllm_is_ratioslot carries the generation logprobs and the kernel formsexp(old - gen) == exp(curr - gen)internally — needed becausecurronly exists inside the fused chunk, so the caller cannot precompute it.Details
entropy_coefthreading):force_on_policy_tis,truncated_importance_sampling_ratio(upper clamp),truncated_importance_sampling_ratio_min(lower clamp / floor).genrides the existingvllm_is_ratiochunked slot;nan_to_numguards the exp.Commits
feat(grpo): in-kernel force-on-policy TIS weightfeat(grpo): add lower clamp (min) for in-kernel force-on-policy TIS weight