fix: optimize forum migration to eliminate N+1 queries and improve scalability#58
Open
Alam-2U wants to merge 4 commits into
Open
fix: optimize forum migration to eliminate N+1 queries and improve scalability#58Alam-2U wants to merge 4 commits into
Alam-2U wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR refactors the MongoDB → MySQL forum migration to reduce N+1 query patterns by prefetching users and MongoContent mappings, and by switching many per-row operations to bulk reads/writes with optional parallel per-course execution.
Changes:
- Introduces course-wide prefetching/caching and bulk_create/bulk_update flows for threads, comments, votes, edit history, abuse flaggers, subscriptions, and read states.
- Adds optional multi-course parallelism (
--workers) and configurable bulk operation sizing (--batch-size) in the management command. - Bumps package version to 0.6.8.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| forum/migration_helpers.py | Replaces per-record ORM work with batched caching + bulk operations for migration steps. |
| forum/management/commands/forum_migrate_course_from_mongodb_to_mysql.py | Adds worker-based parallel course migration, batch size wiring, and improved progress reporting. |
| forum/init.py | Version bump. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+378
to
+390
| mc_rows = [ | ||
| MongoContent( | ||
| mongo_id=mid, | ||
| content_type=thread_ct, | ||
| content_object_id=thread.pk, | ||
| ) | ||
| for mid, thread in zip(mongo_ids, created) | ||
| if thread.pk | ||
| ] | ||
| if mc_rows: | ||
| MongoContent.objects.bulk_create( | ||
| mc_rows, batch_size=BATCH_SIZE, ignore_conflicts=True | ||
| ) |
Comment on lines
+40
to
+48
| # Ensure this thread does not reuse a connection inherited from the | ||
| # main thread (Django creates a new one on first access per thread). | ||
| connections.close_all() | ||
|
|
||
| # Django stores DB connections in thread-local storage, so each worker | ||
| # thread automatically gets its own connection on first access. | ||
| # Explicitly close any connection inherited from the spawning thread | ||
| # so the worker starts with a clean slate. | ||
| connections.close_all() |
Comment on lines
+110
to
+115
| create_waffle_flags = not options["no_toggle"] | ||
| workers: int = int(options["workers"]) # type: ignore[arg-type] | ||
|
|
||
| # Override module-level BATCH_SIZE when caller passes --batch-size. | ||
| import forum.migration_helpers as _mh | ||
| _mh.BATCH_SIZE = int(options["batch_size"]) # type: ignore[arg-type] |
| ) | ||
| for cid, err in failed: | ||
| self.stderr.write(self.style.ERROR(f" {cid}: {err}")) | ||
| raise SystemExit(1) |
Comment on lines
+561
to
+564
| if mc_rows: | ||
| MongoContent.objects.bulk_create( | ||
| mc_rows, batch_size=BATCH_SIZE, ignore_conflicts=True | ||
| ) |
Alam-2U
marked this pull request as ready for review
July 17, 2026 09:32
srathod-apphelix
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR optimizes the MongoDB → MySQL forum migration process by eliminating several N+1 query patterns, reducing per-record database operations, and introducing batched/parallel processing capabilities.
The existing migration performed a large number of repeated database lookups and individual inserts, causing migration time to scale poorly with dataset size. Analysis showed that for large courses, the migration generated hundreds of thousands of database queries and MongoDB round trips, resulting in impractical runtimes.
Changes
User Pre-fetching
Userlookups with a single bulk fetch per course.MongoContent Pre-fetching
MongoContentrecords for a course.Bulk Creation of Forum Content
Replaced per-record inserts with
bulk_create()for:Added configurable batch sizes to control memory usage and database load.
Comment Migration Improvements
sort_keyupdates instead of individual saves.Subscription Migration Optimization
$in.Read State Optimization
ReadStateandLastReadTime.Parallel Course Processing
--workersargument.1to preserve compatibility and avoid issues in environments such as SQLite-based tests.Operational Improvements
--batch-sizeparameter.Validation
--workers=1by default).Expected Impact
These changes significantly reduce:
The migration now relies primarily on batched reads and writes, making it substantially more scalable for large datasets and enabling effective performance testing against production-scale data.
Follow-up
The next step is validating performance improvements against a larger dataset by either:
Ticket : LP-832