support file-based ASR for nemotron streaming model by rui-ren · Pull Request #885 · microsoft/foundry-local

rui-ren · 2026-07-15T17:13:43Z

Add file-based Nemotron ASR transcription support in C++ SDK v2

Summary

This PR completes file-based audio transcription support for Nemotron streaming ASR models in sdk_v2/cpp by finishing the C++ AudioSession implementation and aligning behavior with the existing OpenAI-JSON transcription flow.

For more details, please refer to github/copilot-cli#4024 (comment)

What changed

• Implemented/finished Nemotron file transcription path in:

 sdk_v2/cpp/src/inferencing/generative/audio/audio_session.cc 
 sdk_v2/cpp/src/inferencing/generative/audio/audio_session.h

• Added WAV file parsing for file-based transcription input:
• RIFF/WAVE validation
• fmt / data chunk handling with bounds checks
• 16kHz enforcement
• PCM16 and float32 decoding
• Added Nemotron-specific language runtime option mapping ( lang_id ) and temperature handling.
• Fixed decode flow to keep a single generator across incremental audio processing (100ms chunks + flush), which resolves truncated transcription output for file-based runs.

Behavior

• OpenAI-JSON file transcription for Nemotron models now returns full transcript text from WAV input in SDK v2.
• Existing streaming and non-Nemotron paths remain unchanged.

End-to-end scenario covered

• Model alias: nemotron-3.5-asr-streaming-0.6b
• Input: sample-speech-1m-16k.wav
• Path exercised: JS SDK v2 AudioClient.transcribe(...) → C++ SDK v2 audio transcription pipeline

vercel · 2026-07-15T17:13:50Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview, Comment	Jul 28, 2026 9:22pm

Copilot

Pull request overview

Adds file-based WAV transcription for Nemotron streaming ASR models in the C++ SDK.

Changes:

Routes Nemotron file requests through the streaming processor.
Parses 16 kHz PCM16/float32 WAV files with channel downmixing.
Adds language mapping, temperature handling, and OpenAI JSON output.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
`audio_session.h`	Declares Nemotron transcription and WAV parsing helpers.
`audio_session.cc`	Implements Nemotron inference, decoding, language mapping, and WAV parsing.

+  int completion_tokens = 0;
+
+  auto decode_all_tokens = [&]() {
+    while (!generator->IsDone() && !original_request.canceled) {


+    size_t count = std::min(kNemotronSamplesPerChunk, samples.size() - offset);
+    run_one_pass(processor->Process(samples.data() + offset, count));
+  }
+  run_one_pass(processor->Flush());


kunal-vaishnavi · 2026-07-27T08:07:12Z

+    }
+  }
+
+  auto samples = LoadPcmWavAsFloatSamples(req.filename);


This is a valid issue. We need unit tests to validate these changes such as:

Happy path (nemotron-3.5-asr-streaming-0.6b with a valid 16kHz WAV)

Any regression fixes in this change

Parser validation cases like malformed headers and unsupported sample rates

PCM16 and float32 input handling

nenad1002 · 2026-07-15T17:42:28Z

Why? Parakeet can execute faster and more accurate file-based transcription. I wouldn't want customers to rely on streaming models for non-streaming scenarios, and we're encouraging them with this.

rui-ren · 2026-07-15T17:48:21Z

Why? Parakeet can execute faster and more accurate file-based transcription. I wouldn't want customers to rely on streaming models for non-streaming scenarios, and we're encouraging them with this.

This is not a way to encourage them to do file-base transcription with RNNT streaming model, as you can see here,

github/copilot-cli#4024 (comment)

they are using streaming model, however they do file-based transcription and get error raised. IMO, we better to cover this edge case for them, instead of switch models?

What do you think? @sylvanc

nenad1002 · 2026-07-15T18:09:28Z

Why? Parakeet can execute faster and more accurate file-based transcription. I wouldn't want customers to rely on streaming models for non-streaming scenarios, and we're encouraging them with this.

This is not a way to encourage them to do file-base transcription with RNNT streaming model, as you can see here,

github/copilot-cli#4024 (comment)

they are using streaming model, however they do file-based transcription and get error raised. IMO, we better to cover this edge case for them, instead of switch models?

What do you think? @sylvanc

Hmm, I haven't realized they're using file-based transcription. Ok, but there are better models for this going forward.

rui-ren · 2026-07-20T16:45:45Z

@skottmckay can you please review this PR ? Thanks.

kunal-vaishnavi · 2026-07-22T18:46:20Z


+  // If the model is a Nemotron speech model, process the transcription using the Nemotron-specific method.
+  // e.g. RNNT models require Nemotron-specific transcription handling.
+  if (IsNemotronSpeechModel()) {


I presume that the v2 SDK already supported Nemotron when it was merged into main. Why do we now need an early return to support it?

kunal-vaishnavi · 2026-07-22T18:51:55Z

+                                                      FOUNDRY_LOCAL_TEXT_ITEM_TYPE_OPENAI_JSON));
+}
+
+std::vector<float> AudioSession::LoadPcmWavAsFloatSamples(const std::string& audio_file_path) {


From Copilot:

Good addition overall — the Nemotron transcription path is clean, the WAV parser is thorough, and keeping a single generator across chunks is the right fix for truncated output. A few things worth addressing before merging:

🔴 Blocking issues

WAV chunk bounds check is subtly wrong (LoadPcmWavAsFloatSamples, chunk-walking loop)
The bounds guard chunk_start + chunk_size > file_size - (chunk_size % 2 == 1 ? 1 : 0) mixes the padding-byte adjustment into the main bounds check, which can undercount available space and let a crafted file slip through. The padding byte lives after the chunk data, so it shouldn't reduce the window used to validate the chunk itself. Suggested fix:

// Before: chunk_start + chunk_size > file_size - (chunk_size % 2 == 1 ? 1 : 0) // After: if (chunk_start < 0 || chunk_start + static_cast<std::streamoff>(chunk_size) > file_size) { FL_THROW(FOUNDRY_LOCAL_ERROR_INVALID_USAGE, "Corrupted WAV chunk."); }

The odd-byte padding seek that follows handles alignment correctly as-is.

Format validation happens after full data load (LoadPcmWavAsFloatSamples, data chunk branch)
data.resize(chunk_size) allocates the full audio buffer before sample rate and format have been validated. For a large or malformed file, this could allocate hundreds of MB only to throw immediately afterward. Consider moving the sample_rate, audio_format, and bits_per_sample checks to right after the fmt chunk is parsed, before the data chunk is even encountered.

🟡 Suggestions

Gap at language ID 5 in NemotronLanguageIdMap — IDs jump from 4 (zh-cn) to 6 (hi). If this is intentional (reserved/unsupported language), a brief comment would prevent future contributors from assuming it's a typo.

prompt_tokens hardcoded to 0 in ProcessNemotronFileTranscription — the existing path computes this from the processor. If it's not meaningful for Nemotron, a // TODO explaining why would help maintain parity expectations with the OpenAI usage spec.

decode_all_tokens lambda and thread-safety — the lambda captures generator by reference and is safe as used today (same-thread, same scope). A short comment noting it must not be moved off the creation thread would protect against future refactors.

rui-ren · 2026-07-24T23:09:06Z

@kunal-vaishnavi can you please review again? We are hoping to add this to next release. Thank you so much .

kunal-vaishnavi · 2026-07-27T08:12:02Z

@kunal-vaishnavi can you please review again? We are hoping to add this to next release. Thank you so much .

I reviewed the PR again. Can you add unit tests to validate your changes and ensure there's no breaking change?

rui-ren · 2026-07-28T23:23:52Z

@kunal-vaishnavi can you please review it again? Thanks

kunal-vaishnavi · 2026-07-28T23:50:02Z

  EXPECT_THROW(session.ProcessRequest(request, response), nlohmann::json::parse_error);
 }

+TEST_F(AudioSessionTest, NemotronOpenAIJsonWithInvalidRiffThrows) {


From Copilot:

Could we add a regression test that uses a multi-chunk WAV and verifies we get the full transcript back? The main bug this PR is fixing is truncation during file-based Nemotron transcription, so having an end-to-end case that proves the full output survives chunked decoding would give us much stronger confidence here.

kunal-vaishnavi · 2026-07-28T23:50:23Z

@@ -431,6 +515,145 @@ TEST_F(AudioSessionTest, OpenAIJsonWithInvalidJsonThrows) {
  EXPECT_THROW(session.ProcessRequest(request, response), nlohmann::json::parse_error);
 }



From Copilot:

Could we add an explicit unit test for the float32 WAV path in LoadPcmWavAsFloatSamples? The parser has separate decoding logic for PCM16 and float32, so having coverage for float32 would help make sure that branch keeps working as expected.

kunal-vaishnavi · 2026-07-28T23:50:47Z

+  DecodeNemotronTokens(generator, tokenizer_stream, text, streaming_callback, response_id, original_request,
+                       completion_tokens);
+}
+


From Copilot:

Could we consider moving decode_all_tokens into a named private helper instead of keeping it as a local lambda? This logic is already central to the new Nemotron path, and giving it a dedicated method would make the control flow a bit easier to follow, test, and extend if we need to change the decode behavior later.

kunal-vaishnavi · 2026-07-28T23:51:09Z


+const std::unordered_map<std::string, std::string>& NemotronLanguageIdMap() {
+  // Language id 5 is intentionally missing because upstream Nemotron lang_id assignments skip it.
+  static const std::unordered_map<std::string, std::string> kMap = {


What is the language id for 5? I see it goes from 4 to 6 directly.

@kunal-vaishnavi That is removed by Nvidia, you can check here.
https://microsoft.visualstudio.com/windows.ai.toolkit/_git/neutron-server/pullrequest/15742356

Rui Ren and others added 2 commits July 14, 2026 10:45

add file based support for streaming model

77170bd

update the cpp code for streaming model

d5eafd6

Copilot AI review requested due to automatic review settings July 15, 2026 17:13

rui-ren changed the title ~~Ruiren/add file based support nemotron streaming~~ support file-based ASR for nemotron streaming model Jul 15, 2026

rui-ren requested a review from skottmckay July 15, 2026 17:14

Copilot started reviewing on behalf of rui-ren July 15, 2026 17:14 View session

rui-ren requested review from baijumeswani, kunal-vaishnavi and nenad1002 July 15, 2026 17:14

Copilot AI reviewed Jul 15, 2026

View reviewed changes

kunal-vaishnavi reviewed Jul 22, 2026

View reviewed changes

Comment thread sdk_v2/cpp/src/inferencing/generative/audio/audio_session.cc Outdated

kunal-vaishnavi reviewed Jul 22, 2026

View reviewed changes

Comment thread sdk_v2/cpp/src/inferencing/generative/audio/audio_session.cc Outdated

kunal-vaishnavi reviewed Jul 22, 2026

View reviewed changes

Comment thread sdk_v2/cpp/src/inferencing/generative/audio/audio_session.cc Outdated

kunal-vaishnavi reviewed Jul 22, 2026

View reviewed changes

fix feedback

f42c16f

vercel Bot deployed to Preview July 24, 2026 22:09 View deployment

add unitest

8e38e8f

vercel Bot deployed to Preview July 28, 2026 21:22 View deployment

kunal-vaishnavi reviewed Jul 28, 2026

View reviewed changes

		@@ -431,6 +515,145 @@ TEST_F(AudioSessionTest, OpenAIJsonWithInvalidJsonThrows) {
		EXPECT_THROW(session.ProcessRequest(request, response), nlohmann::json::parse_error);
		}

Uh oh!

Conversation

rui-ren commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add file-based Nemotron ASR transcription support in C++ SDK v2

Summary

What changed

Behavior

End-to-end scenario covered

Uh oh!

vercel Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

kunal-vaishnavi Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

nenad1002 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rui-ren commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nenad1002 commented Jul 15, 2026

Uh oh!

rui-ren commented Jul 20, 2026

Uh oh!

Uh oh!

Uh oh!

kunal-vaishnavi Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kunal-vaishnavi Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

rui-ren commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kunal-vaishnavi commented Jul 27, 2026

Uh oh!

rui-ren commented Jul 28, 2026

Uh oh!

kunal-vaishnavi Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

rui-ren Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rui-ren commented Jul 15, 2026 •

edited

Loading

vercel Bot commented Jul 15, 2026 •

edited

Loading

nenad1002 commented Jul 15, 2026 •

edited

Loading

rui-ren commented Jul 15, 2026 •

edited

Loading

rui-ren commented Jul 24, 2026 •

edited

Loading

rui-ren Jul 29, 2026 •

edited

Loading