Keep decode memory independent of audio duration - #50
Open
sjoenk wants to merge 7 commits into
Open
Conversation
convertToM4a collected the whole decoded track before the encoder started, so peak memory scaled with the audio duration: a three hour recording needs ~1.9 GB of PCM and hit the per-app heap limit (#49). A pull-based PcmSource now feeds each chunk to the AAC encoder as it leaves the decoder, and trimAudio streams to the encoder or straight to disk instead of buffering the trimmed range twice. getWaveform folds RMS energy into a bounded set of buckets while decoding rather than collecting every sample as a boxed Short, which ran out of memory well before an hour of audio. Also derive M4A timestamps from the running frame count instead of accumulated per-buffer deltas, fill encoder buffers on frame boundaries, keep codec-specific data out of the muxer samples, and release the codecs plus remove a partial output file on failure.
performGetWaveform appended every decoded sample to an array before computing the RMS windows, so a long recording held the entire track in memory. Energy now goes into a bounded bucket array as the asset reader delivers buffers (#49).
GetWaveform was the last caller that buffered a complete track through DecodeToPcm; it now folds RMS energy into buckets while GStreamer pushes samples (#49). DecodeToPcm and PcmResult have no callers left, so remove them.
TrimAudio decoded the whole range into memory before writing M4A, and for WAV output it decoded the input a second time through StreamPcmToWav. A new M4aStreamWriter now takes each PCM buffer as it arrives, initialized from an onFormat callback because the sink writer needs the sample rate up front (#49). GetWaveform accumulates RMS energy the same way, which leaves DecodeToPcm without callers.
Directory.systemTemp resolves to the app's code_cache on Android, which the platform clears after a fresh install and whenever storage runs low. Files written by one test — and sometimes the directory itself — vanished mid-run, failing eight unrelated tests. Use the app's files directory, which the platform leaves alone.
writePcmWavFile streams a synthetic WAV to disk in blocks so the test itself does not allocate the payload. Three minutes is a compromise: emulator software codecs run at roughly real time, so the hundreds of megabytes needed to actually hit the old heap ceiling would make the test run for half an hour.
There was a problem hiding this comment.
Pull request overview
Streams PCM processing to keep memory bounded for long audio files.
Changes:
- Interleaves decoding, encoding, trimming, and waveform accumulation.
- Adds bounded waveform accumulators and regression coverage.
- Updates release metadata and Android test storage.
Reviewed changes
Copilot reviewed 10 out of 11 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
android/.../AudioDecoderPlugin.kt |
Adds streaming decode and encoding. |
android/.../AudioDecoderPluginTest.kt |
Tests waveform accumulation. |
darwin/.../AudioDecoderPlugin.swift |
Streams waveform calculation. |
linux/audio_decoder_plugin.cc |
Streams waveform calculation. |
windows/audio_decoder_plugin.cpp |
Streams trimming and waveforms. |
windows/audio_decoder_plugin.h |
Extends streaming decode callback. |
example/integration_test/test_helpers.dart |
Adds streaming WAV fixture writer. |
example/integration_test/plugin_integration_test.dart |
Adds long-file integration coverage. |
pubspec.yaml |
Bumps package version. |
example/pubspec.lock |
Updates resolved dependencies. |
CHANGELOG.md |
Documents version 0.8.2. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+618
to
624
| init { | ||
| if (timeRangeUs != null && timeRangeUs.first > 0) { | ||
| track.extractor.seekTo(timeRangeUs.first, MediaExtractor.SEEK_TO_CLOSEST_SYNC) | ||
| } | ||
| decoder.configure(track.format, null, null, 0) | ||
| decoder.start() | ||
| } |
| // hundreds of megabytes it would take to actually hit the old heap | ||
| // ceiling would make this test run for half an hour. What it does cover | ||
| // is that a long chunk sequence still streams through end to end. | ||
| const frameCount = 44100 * 60 * 3; |
| * **Fix `OutOfMemoryError` on Android for long recordings** (#49) — `convertToM4a` collected the entire decoded track before encoding started, so a three hour file needed ~1.9 GB of PCM in memory and crashed on the Android heap limit. Decoding and AAC encoding now run interleaved, keeping peak memory independent of the audio duration. | ||
| * `trimAudio` no longer buffers the trimmed range either (both the M4A and the WAV output path stream to disk), on Android and Windows. | ||
| * `getWaveform` / `getWaveformBytes` fold RMS energy into a bounded set of buckets while decoding instead of collecting every sample — the old Android path stored boxed `Short` values (~16 bytes per sample) and ran out of memory well before an hour of audio. Implemented on Android, iOS, macOS, Linux and Windows. | ||
| * For files long enough to exceed the bucket resolution the waveform is now approximated: values stay within ~0.004 of the previous result for regular material, while an extremely short and loud transient can spread into one neighbouring window. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
convertToM4aon Android collected the entire decoded track before the encoder started, so peak memory scaled with the audio duration — a three hour recording needs ~1.9 GB of PCM and hit the per-app heap limit. Decoding and AAC encoding now run interleaved.trimAudio(Android, Windows) and ingetWaveformon every platform that decoded PCM itself.getWaveformfolds RMS energy into a bounded set of buckets while decoding; the Android path previously stored each sample as a boxedShort(~16 bytes per sample) and ran out of memory well before an hour of audio.Closes #49
Changes
android/.../AudioDecoderPlugin.kt— new pull-basedPcmSource/PcmDecoderthat yields one chunk per call, pluswithPcmSourceandstreamPcmToWav.performM4aConversion,performTrimAudioandperformGetWaveformall build on it, which also removes three duplicated codec loops.encodePcmToM4afills encoder buffers on frame boundaries, derives timestamps from the running frame count, keeps codec-specific data out of the muxer samples, and releases codecs plus removes a partial output file on failure.darwin/.../AudioDecoderPlugin.swift—performGetWaveformaccumulates whileAVAssetReaderdelivers buffers instead of appending every sample. The M4A paths already streamed throughAVAssetExportSession.linux/audio_decoder_plugin.cc—GetWaveformstreams;DecodeToPcm/PcmResulthad no callers left and are removed. M4A conversion and trimming already streamed via GStreamer.windows/audio_decoder_plugin.cpp/.h— newM4aStreamWriterfeeds each PCM buffer toIMFSinkWriteras it arrives, initialized from a newonFormatcallback onDecodeToPcmStreambecause the writer needs the sample rate up front.GetWaveformstreams too, leavingDecodeToPcmunused.writePcmWavFilehelper that streams the input to disk.Directory.systemTemp: that is the app'scode_cache, which Android clears after a fresh install and under storage pressure, making files disappear mid-run.Test plan
cd example/android && ./gradlew :audio_decoder:testDebugUnitTest— 16 tests pass, including the new accumulator coveragecd example && flutter test integration_test/plugin_integration_test.dart -d <android device>— 27 tests pass; theregression #49case converts three minutes of audio end to endflutter testandflutter analyzein the package and inexample/flutter test integration_test/plugin_integration_test.dart --dart-define=AUDIO_DECODER_LONG_TEST_MINUTES=45on a physical Android device — the default three minutes keeps the suite fast but stays under the old heap ceilinggetWaveformon Linux and Windows — the C++ changes could not be compiled on macOS, only the accumulator class was verified standalone with clangTime spent
⏱️ Estimated time spent: 9 hours