Skip to content

feat: harden Qwen 3.5/3.6 LoRA distributed training #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
NolanHo wants to merge 157 commits into
base: main
Choose a base branch
from
Draft

feat: harden Qwen 3.5/3.6 LoRA distributed training #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
157 commits
Select commit Hold shift + click to select a range
1d023d0
feat: harden qwen hybrid lora distributed training
NolanHo Jul 16, 2026
e2a03a9
feat: persist dynamic lora and add replicated dp
NolanHo Jul 16, 2026
ae75152
fix: make native accumulation and multi-lora updates logical-step aware
NolanHo Jul 16, 2026
0794e6a
test: cover native micro-step optimizer boundary
NolanHo Jul 16, 2026
0530e79
fix: invalidate native lora graph between microbatches
NolanHo Jul 16, 2026
f65af15
docs: clarify native loss and parallel reduction semantics
NolanHo Jul 16, 2026
538cb5e
docs: describe multi-lora logical update
NolanHo Jul 16, 2026
4e242ee
feat: add Megatron-style parallel topology
NolanHo Jul 16, 2026
afcf091
fix: restore native optimizer step on checkpoint load
NolanHo Jul 16, 2026
5a58d6f
feat: isolate dynamic lora optimizer state
NolanHo Jul 16, 2026
e75b3c5
feat: add latent-rank tensor parallel lora
NolanHo Jul 16, 2026
5aa4ea4
feat: accumulate native lora gradients in fp32
NolanHo Jul 16, 2026
ed25daa
feat: add rank-aware tp lora checkpoints
NolanHo Jul 16, 2026
ff382a0
fix: reject unsafe native parallel topologies
NolanHo Jul 16, 2026
2702bf0
docs: record abi11 tp checkpoint evidence
NolanHo Jul 16, 2026
76eace6
fix: reject native pp and cp topologies
NolanHo Jul 16, 2026
56a6ea1
docs: record native pp cp rejection
NolanHo Jul 16, 2026
b7050ea
fix: make dynamic lora dp and ep numerically correct
NolanHo Jul 16, 2026
d6f897c
docs: record dp ep and optimizer verification
NolanHo Jul 16, 2026
bfa3d8e
feat: prototype variable split ep token a2a
NolanHo Jul 16, 2026
1a72183
feat: add sharded fixed lora ep a2a
NolanHo Jul 16, 2026
b51e9c1
docs: add native ep benchmark evidence
NolanHo Jul 16, 2026
0f39f07
fix: normalize dynamic sharded EP LoRA gradients
NolanHo Jul 17, 2026
54bcce5
feat: add frozen dense MLP tensor parallelism
NolanHo Jul 17, 2026
e5bd9c4
feat: add qwen attention tensor parallelism
NolanHo Jul 17, 2026
69c4153
feat: add qwen gdn tensor parallelism
NolanHo Jul 17, 2026
99aecef
feat: add qwen tensor data parallelism
NolanHo Jul 17, 2026
b13bbbd
feat: add vocabulary tensor parallelism
NolanHo Jul 17, 2026
f6da76a
feat: compose Qwen LoRA tensor and expert parallelism
NolanHo Jul 17, 2026
f19a1d1
feat: pack Qwen expert-parallel dispatch
NolanHo Jul 17, 2026
e21dd13
feat: shard Qwen experts across tensor parallel ranks
NolanHo Jul 17, 2026
b1c51a5
feat: add distributed LoRA checkpoint v5
NolanHo Jul 17, 2026
ed3a510
feat: add distributed Qwen adapter export and resume
NolanHo Jul 17, 2026
e439f5c
feat: validate Qwen tri-axis LoRA training
NolanHo Jul 17, 2026
de5887c
feat: add resilient source-sharded server dispatch
NolanHo Jul 18, 2026
fbf0e70
feat: add transactional distributed Qwen checkpoints
NolanHo Jul 18, 2026
c51561f
feat: add binary tensor slab transport
NolanHo Jul 18, 2026
a3edf09
fix: restore heterogeneous LoRA checkpoints
NolanHo Jul 18, 2026
bbe0aa6
feat: make dynamic Adam transactional
NolanHo Jul 18, 2026
9532174
feat: train heterogeneous dynamic adapters
NolanHo Jul 18, 2026
6111ae5
perf: unify heterogeneous LoRA finalization
NolanHo Jul 18, 2026
dbb313b
fix: enforce distributed LoRA optimizer invariants
NolanHo Jul 18, 2026
9fb792b
perf: group LoRA gradient collectives
NolanHo Jul 18, 2026
887fb72
perf: slab LoRA gradient accumulators
NolanHo Jul 18, 2026
ac3700b
fix: guard LoRA registry mutations
NolanHo Jul 18, 2026
9b627c0
fix: make LoRA registration transactional
NolanHo Jul 18, 2026
a3be24e
perf: scale EP dispatch metadata packing
NolanHo Jul 18, 2026
474a8df
perf: fuse GDN alpha beta projection
NolanHo Jul 18, 2026
97ce513
fix: align GDN QK normalization epsilon semantics
NolanHo Jul 18, 2026
546afc7
fix: reject unsafe GDN padding layouts
NolanHo Jul 18, 2026
684eadc
perf: scope multi-LoRA checkpoint projection caches
NolanHo Jul 18, 2026
7f96893
perf: skip strict right padding in GDN kernels
NolanHo Jul 18, 2026
cdebe91
perf: gate multi-LoRA training trace logs
NolanHo Jul 18, 2026
6c1602f
fix: enumerate dynamic LoRA registries without truncation
NolanHo Jul 18, 2026
1de20b5
fix: exclude fixed adapter from train-all selection
NolanHo Jul 18, 2026
e45e11a
perf: pack LoRA gradient collectives and guard batch registration
NolanHo Jul 18, 2026
d1ee7e0
perf: add optional GDN state checkpoints
NolanHo Jul 18, 2026
d254836
perf: bound EP dispatch and preserve async runtime
NolanHo Jul 18, 2026
2561489
perf: batch heterogeneous LoRA tenants by padded rank
NolanHo Jul 18, 2026
4f7069e
perf: fuse GDN recurrent backward sweep
NolanHo Jul 18, 2026
d7181bd
perf: coalesce opt-in multi-LoRA server batches
NolanHo Jul 18, 2026
b6efba1
perf: add chunkwise GDN and tenant loss reports
NolanHo Jul 18, 2026
778359c
feat: add five-axis Qwen communicator foundation
NolanHo Jul 18, 2026
1a6a2fb
feat: add pipeline stage ownership ABI
NolanHo Jul 18, 2026
5d12729
feat: wire pipeline-local qwen training ownership
NolanHo Jul 18, 2026
381062d
fix: preserve global dynamic lora layer identity
NolanHo Jul 18, 2026
d37005d
feat: wire pipeline-local qwen server contexts
NolanHo Jul 18, 2026
ba2645d
feat: enable pipeline stage training and stage-union checkpoints
NolanHo Jul 18, 2026
623adfd
fix: validate multi-lora source geometry before dispatch
NolanHo Jul 18, 2026
ac196fe
feat: isolate dynamic tenant learning rates
NolanHo Jul 18, 2026
45d9528
fix: accumulate pipeline microsteps across stages
NolanHo Jul 18, 2026
153890c
fix: include pipeline microstep weight in stage consensus
NolanHo Jul 18, 2026
2e53bdb
feat: stage PP2 pipeline and dynamic LoRA optimizer LR
NolanHo Jul 18, 2026
c8434f5
test: verify PP2 LoRA optimizer parity
NolanHo Jul 18, 2026
87a797a
fix: attach router aux loss to fixed LoRA training
NolanHo Jul 18, 2026
b5aa2fa
feat: add isolated dynamic LoRA evaluation
NolanHo Jul 18, 2026
8672f54
feat: expose selected multi-lora evaluation routes
NolanHo Jul 18, 2026
cf16db0
feat: guard dynamic LoRA optimizer steps
NolanHo Jul 18, 2026
36a0eaa
docs: record H20 adapter step validation
NolanHo Jul 18, 2026
99aa1db
feat: add TP2 sequence-parallel collectives
NolanHo Jul 18, 2026
1a93a75
feat: enable guarded TP2 sequence parallel training
NolanHo Jul 18, 2026
6d5e1ac
test: add TP2 sequence parallel native smoke
NolanHo Jul 18, 2026
ccf99f1
docs: record guarded sequence parallel scope
NolanHo Jul 18, 2026
ccc4e6f
test: assert sequence parallel LoRA update
NolanHo Jul 19, 2026
c538540
fix: validate every pipeline microbatch contract
NolanHo Jul 19, 2026
711ea45
fix: quarantine failed dynamic lora recovery
NolanHo Jul 19, 2026
110fcf9
test: verify distributed lora recovery quarantine
NolanHo Jul 19, 2026
f66e2aa
fix: fail stop poisoned lora worker groups
NolanHo Jul 19, 2026
19b8b4a
perf: pack dynamic lora preflight collectives
NolanHo Jul 19, 2026
db65b44
feat: generalize fixed lora pipeline parallelism
NolanHo Jul 19, 2026
48263fb
fix: synchronize first pipeline metadata errors
NolanHo Jul 19, 2026
2515a85
feat: isolate dynamic tenant Adam hyperparameters
NolanHo Jul 19, 2026
b29ac1d
fix: reject undersized linear attention chunks
NolanHo Jul 19, 2026
7ab8ece
feat: add CP2 headwise GDN training
NolanHo Jul 19, 2026
6dbf430
perf: reuse fixed LoRA cache across pipeline window
NolanHo Jul 19, 2026
ded23a0
bench: add CP2 GDN performance coverage
NolanHo Jul 19, 2026
391a0c3
fix: inherit optimizer defaults in EP benchmark
NolanHo Jul 19, 2026
45ecd50
fix: harden qwen LoRA distributed training
NolanHo Jul 19, 2026
e1460cd
feat: add distributed LoRA gradient clipping
NolanHo Jul 19, 2026
998d863
fix: harden dynamic LoRA pipeline lifecycle
NolanHo Jul 19, 2026
c67232f
docs: correct H20 toolchain status
NolanHo Jul 19, 2026
58f49fa
feat: add guarded dynamic LoRA pipeline window
NolanHo Jul 19, 2026
7c3199a
docs: record guarded dynamic pipeline scope
NolanHo Jul 19, 2026
ef32dff
merge origin/main into qwen audit branch
NolanHo Jul 19, 2026
0133f6e
test: cover CUDA allocator fraction validation
NolanHo Jul 19, 2026
598af90
fix: guard dynamic pipeline adapter composition and commit
NolanHo Jul 19, 2026
387fbe6
Add selected dynamic PP loss reporting
NolanHo Jul 19, 2026
c618cd1
Harden selected PP requests and add opt-in fused QKV
NolanHo Jul 19, 2026
73948d8
Enable configurable dynamic PP microbatches
NolanHo Jul 19, 2026
63da777
Validate dynamic PP microbatch tensor shapes
NolanHo Jul 19, 2026
811b906
Include fused QKV mode in distributed identity
NolanHo Jul 19, 2026
2ddd902
Format dynamic pipeline server wiring
NolanHo Jul 19, 2026
3761f12
Test PP multi-LoRA row admission
NolanHo Jul 19, 2026
e1c174a
Fix native fused QKV compilation
NolanHo Jul 19, 2026
8069506
Test dynamic PP isolation and fused QKV oracle
NolanHo Jul 19, 2026
d0c24c9
Add opt-in fused MLP FC1 path
NolanHo Jul 19, 2026
418faad
Enable dynamic GDN context parallel LoRA
NolanHo Jul 19, 2026
b56b12f
Enable dynamic LoRA sequence parallel mask path
NolanHo Jul 19, 2026
00ea92a
Harden heterogeneous dynamic PP windows
NolanHo Jul 19, 2026
0bfb82a
Add opt-in fused GDN CP exchange
NolanHo Jul 19, 2026
adf9613
Enable guarded dynamic MTP normalization
NolanHo Jul 19, 2026
5892152
Add DP2 dynamic MTP oracle
NolanHo Jul 19, 2026
cb7ab3b
Enable guarded TP2 dense MTP LoRA path
NolanHo Jul 19, 2026
d4ea960
Extend Qwen3.6 MTP to guarded MoE and EP plumbing
NolanHo Jul 20, 2026
b67495b
Validate Qwen3.6 MTP across EP and vocabulary TP
NolanHo Jul 20, 2026
165a6a2
Remove non-beneficial MoE overlap prototype
NolanHo Jul 20, 2026
cb5d81c
Reduce dynamic LoRA host synchronization fences
NolanHo Jul 20, 2026
ff5613a
Optimize sparse heterogeneous LoRA registry selection
NolanHo Jul 20, 2026
29f97f7
Negotiate dynamic multi-LoRA HTTP capabilities
NolanHo Jul 20, 2026
2dfe8fe
Reduce large-world MoE metadata transfers
NolanHo Jul 20, 2026
feacc81
Add binary multi-LoRA training ingress
NolanHo Jul 20, 2026
7a0bedd
Bound dynamic LoRA rank inflation
NolanHo Jul 20, 2026
734dff7
Record H20 PP and CP boundary validation
NolanHo Jul 20, 2026
25d0373
Keep bounded multi-LoRA layout buckets open
NolanHo Jul 20, 2026
717c902
Record A2A async overlap benchmark result
NolanHo Jul 20, 2026
f553d54
Add opt-in pinned host staging
NolanHo Jul 20, 2026
add3eb1
Lock PP scheduler fail-closed contract
NolanHo Jul 20, 2026
033a093
Document PP scheduler guard
NolanHo Jul 20, 2026
40e52ca
Overlap multi-LoRA batching deadline with dispatch
NolanHo Jul 20, 2026
9baecde
Add guarded full-attention CP2 bridge
NolanHo Jul 20, 2026
6552570
Document CP2 full-attention verification
NolanHo Jul 20, 2026
3431c8b
perf: fuse shared QKV LoRA input projection
NolanHo Jul 20, 2026
4da25a0
perf: add guarded MoE shared expert overlap
NolanHo Jul 20, 2026
7692e3e
perf: persist dynamic adapter ID index
NolanHo Jul 20, 2026
282fd4a
perf: pack MoE A2A metadata
NolanHo Jul 20, 2026
140cdc8
perf: lazy dynamic optimizer state and trim GDN barriers
NolanHo Jul 20, 2026
baedae0
perf: reuse EP A2A prefix plans
NolanHo Jul 20, 2026
330b470
perf: fuse packed MoE weighted unpermute
NolanHo Jul 20, 2026
11343a1
perf: pool dynamic Adam transaction shadows
NolanHo Jul 20, 2026
04befa4
perf: fuse MoE local expert permutation
NolanHo Jul 20, 2026
43e46e6
perf: overlap EP count metadata with packing
NolanHo Jul 20, 2026
4a501a8
feat: snapshot dynamic Adam state on host
NolanHo Jul 20, 2026
fac291a
perf: keep restored Adam state on host
NolanHo Jul 20, 2026
6c160bb
perf: page cold dynamic Adam state to host
NolanHo Jul 20, 2026
73ba2a4
test: compare paged Adam snapshots on CPU
NolanHo Jul 20, 2026
a8f1909
docs: record dynamic Adam host paging evidence
NolanHo Jul 20, 2026
0305f94
feat: add arbitrary context parallel ring attention
NolanHo Jul 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

322 changes: 286 additions & 36 deletions crates/rustrain-core/src/runtime.rs
View file
Open in desktop

Large diffs are not rendered by default.

Loading