Skip to content

From NVIDIA Megatron-LM for visibility - #18

Open
RaymondLi0 wants to merge 7892 commits into
bigcode-project:multi-query-attentionfrom
NVIDIA:main
Open

From NVIDIA Megatron-LM for visibility#18
RaymondLi0 wants to merge 7892 commits into
bigcode-project:multi-query-attentionfrom
NVIDIA:main

Conversation

@RaymondLi0

@RaymondLi0 RaymondLi0 commented Jan 24, 2023

Copy link
Copy Markdown
Collaborator

No description provided.

@RaymondLi0
RaymondLi0 changed the base branch from multi-query-attention to before-merge June 20, 2023 20:12
@RaymondLi0
RaymondLi0 changed the base branch from before-merge to multi-query-attention June 20, 2023 20:12
yhgalaxy and others added 27 commits August 18, 2026 02:56
yhgalaxy Guihong Li
Xin Yao Maanu Grover
Enhance optimizer state loading with runtime overrides (#3720)
59ff171 
Signed-off-by: yhgalaxy <yhgalaxy@outlook.com>
Co-authored-by: Guihong Li <guihongl@nvidia.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Maanu Grover <maanug@nvidia.com>
Jingyue Wu
test(mfsdp): cover v2 training memory (#6567)
35ae46c 
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Ajay
fix(ci): avoid JET expansion of HOME in setup scripts (#6599)
5d50b16 
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
Teodor-Dumitru Ene Jorge Albericio
RL: isolate placeholder rollouts from GRPO rewards (#6475)
32756e3 
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com>
Ajay
tests: Update cerno.yml with linear issue status (#6304)
ae5ec14 
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
Yashaswi Karnati
Enable GTP for heterogeneous MIMO training (#6570)
cfb8e46 
Signed-off-by: ykarnati <ykarnati@nvidia.com>
Jingyue Wu
test(mfsdp): separate overlap test (#6591)
dcb2bfd 
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Xiaowei Ren Claude Opus 5 (1M context) hongbinl
GDP perf optimizations (#6553)
bd72caf 
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: hongbinl <hongbinl@nvidia.com>
Teodor-Dumitru Ene
RL: compress per-token staleness metrics (#6376)
c9676f4 
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Teodor-Dumitru Ene
Guard against divide/multiply by 0 in RL task distribution (#4362)
3126ff2 
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Teodor-Dumitru Ene
Support PackedSeqParams CP CG as in #2282 (#6571)
d5a019a 
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Robert Kirby
Use named module logger in dynamic inference engine (#6001)
aaa4dfd 
Signed-off-by: Robert Kirby <rkirby@nvidia.com>
oliver könig
fix(ci): route scoped tokens through community bot (#6632)
74a4894 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Tom Long Claude Opus 5 (1M context)
Correct the split-PR skill's stacking rules for copy-pr-bot CI (#6593)
aeae5d7 
Signed-off-by: ilml <tolong@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
oliver könig
fix(tests): restrict result extraction to rank zero (#6620)
0980313 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Jingyue Wu
test(mfsdp): separate memory tests (#6568)
54c7918 
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Shiqing Fan
fix(gtp+grad-reduce-in-fp32): emit the wgrad in main_grad's dtype so …
e5542e8 
…the reduce-scatter honours FP32 (#6624)

Signed-off-by: Shiqing Fan <shiqingf@nvidia.com>
Yashaswi Karnati
Honor high-priority streams in MIMO topology (#6604)
bf51cf4 
Signed-off-by: ykarnati <ykarnati@nvidia.com>
Teodor-Dumitru Ene Vitaly Kurin Jorge Albericio
Share prefill in sequence packing (#5887)
3ad8c2a 
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Co-authored-by: Vitaly Kurin <yobibyte@users.noreply.github.com>
Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com>
GitHub Actions
Update copy-pr-bot.yaml [skip ci]
5f9b2bb
Cory Ye
[MFSDPv1] Fix gradient sharding / allocation schedule with ZeRO-2, an…
342faa4 
…d fix wgrad accumulate fusion for ZeRO-1. (#6357)

Signed-off-by: Cory Ye <cye@nvidia.com>
Jingyue Wu
Add opt-in MFSDP unified communication stream (#6563)
540f420 
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Maanu Grover
tests: allow trusted TE pickle extra state in affected tests (#6602)
1d69364 
Signed-off-by: Maanu Grover <maanug@nvidia.com>
Teodor-Dumitru Ene
Reflect #4440 onto GPT models, not just hybrid (#6476)
afa707e 
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Charlie Truong
ci: Remove scheduled CI workflow (#6640)
0961f12 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
oliver könig
fix(ci): request pull request permission for queue comments (#6641)
e7d58db 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Philip Petrakian
Remove core teams from FSDP owners (#6623)
29361d6 
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
wdykas and others added 30 commits September 4, 2026 15:15
wdykas shanmugamr1992
test(refit): cover DSA models and enable M2N CI (#6998)
6572312 
Signed-off-by: William Dykas <wdykas@nvidia.com>
Signed-off-by: shanmugamr1992 <shanmugamr1992@gmail.com>
Co-authored-by: shanmugamr1992 <shanmugamr1992@gmail.com>
Jingyue Wu
Golden deepseek_proxy_mfsdp_v2_ep2 against its own output; enable det…
c171f66 
…erminism (#7069)

Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Ajay
ci: Enhance CI env to check functional tests label (#7072)
1bf32f7 
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
Rui Zhu Charlie Truong svcnvidia-nemo-ci
Validate num_attention_heads is divisible by num_query_groups in Tran…
9d790cf 
…sformerConfig (#5756)

Signed-off-by: Rui Zhu <rui.zhu.rz399@yale.edu>
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Tai An svcnvidia-nemo-ci
chore(checkpointing): remove dead load_biencoder_checkpoint and _add_…
fce0676 
…biencoder_args (#6975)

Signed-off-by: Anai-Guo <antai12232931@outlook.com>
Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Co-authored-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
svcnemo-autobot
fix(test): AUT-2220 preserve golden value precision (#7087)
f8d6302 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Tom Long Claude Fable 5
Fix runtime CP group handling in attention for hybrid CP (#6821)
df4996f 
Signed-off-by: Tom Long <tolong@nvidia.com>
Signed-off-by: ilml <tolong@nvidia.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Zhiyu Li
fix: restore each rank's own RNG state on checkpoint load (#6858)
def07af 
Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>
Haoran Zhang
Add CuTeDSL GDP chunkwise context parallelism (#7038)
3b5556e 
Signed-off-by: Haoran Zhang <haoranz@nvidia.com>
GitHub Actions
Update copy-pr-bot.yaml [skip ci]
b1fe759
Jianbin Chang
Expose MFSDP v2 custom-schedule lifecycle (#6484)
b3393bb 
Signed-off-by: Jack Chang <jianbinc@nvidia.com>
jeffnvidia svcnvidia-nemo-ci
Fix FSDP mixed-precision reset for fixed buffers (#6924)
c9b53d0 
Signed-off-by: jeffnvidia <jmahou@nvidia.com>
Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Co-authored-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Dmytro Pykhtar
add raise_on_error argument (#7110)
c768946 
Signed-off-by: dimapihtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Deyu Fu Philip Petrakian
Add native DeepSeek-V4 hybrid attention orchestration (#6402)
907609b 
Signed-off-by: Deyu Fu <deyuf@nvidia.com>
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
svcnemo-autobot
fix: AUT-2228 route partial reruns to the current attempt (#7142)
62fa723 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
going-song Yoonsik Kim
Yechan Kim svcnvidia-nemo-ci
Use metadata dp_cp_group for the tied-embeddings replica_id (#6888)
154f665 
Signed-off-by: Kayeon Song <kayeon.song@navercorp.com>
Signed-off-by: Yoonsik Kim <yoonsik.kim90@navercorp.com>
Signed-off-by: Yechan Kim <yechan.78@navercorp.com>
Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Co-authored-by: Yoonsik Kim <yoonsik.kim90@navercorp.com>
Co-authored-by: Yechan Kim <yechan.78@navercorp.com>
Co-authored-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Tom Long
Keep RerunDataIterator wrapping on the hybrid CP data iterator (#6822)
7fa83b5 
Signed-off-by: Tom Long <tolong@nvidia.com>
svcnemo-autobot
ci: AUT-2245 set CUDA major for lockfile updates (#7151)
be85fc5 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Jiangfei Duan
fix(MoE): Preserve exact expert token counts for expert-bias updates (#…
8dc7ed3 
…7081)

Signed-off-by: Jiangfei Duan <jiangfeid@nvidia.com>
github-actions[bot]
chore: rotate oncall schedule
c6be919
svcnemo-autobot
ci: AUT-2250 retry transient coverage artifact uploads (#7159)
8dcf188 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Rui Zhu Claude Fable 5 svcnvidia-nemo-ci
Enter both autocast and fp8 contexts in combined 1F1B execution (#5761)
0d2dc9c 
Signed-off-by: Rui Zhu <rui.zhu.rz399@yale.edu>
Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Ahmad Kiswani
Derive MFSDP v2 checkpoint chunk metadata from the parameter layout (#…
b9c22eb 
…6214)

Signed-off-by: Ahmad Kiswani <akiswani@nvidia.com>
Philip Petrakian
Add Slack reminders for re-requested reviews (#7071)
b92c3ed 
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Yixuan Xu Philip Petrakian
Propagate rope_scaling_factor from args to GPT model construction (#6306
9ca681c 
)

Signed-off-by: Alvorecer721 <yixuan.xu@ai.ethz.ch>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
oliver könig
ci(pr-review): extract review prompts into a non-activating skill (#7170
56ef6ed 
)

Signed-off-by: oliver könig <okoenig@nvidia.com>
Shanmugam Ramasamy
First pass at comprehensive user guide (#5325)
5590c4b 
Signed-off-by: shanmugamr1992 <shanmugamr1992@gmail.com>
Dingqing Yang
Support FP4 in 1F1B A2A overlap (#6135)
74dc1b5 
Signed-off-by: Dingqing Yang <dingqingy@nvidia.com>
Teodor-Dumitru Ene
RL: prevent unnecessary entry into inference mode (#4125)
b2cea22 
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Jingyue Wu
Document signing rewritten commits (#7037)
ba638f0 
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.