Target branch: release-v0.8.x
Observed behavior
Morango's handling of self-referential models assumes that all of the reference models have the same morango_model_name. When an implementation uses model hierarchies like a base model (Collection) which is then inherited by other models (Facility, Classroom, LearnerGroup), Morango may not properly find dirty children because it filters by morango_model_name. In the case of deletions in the hierarchy (a Classroom and its child LearnerGroup are deleted), the parent deletion will occur first because it does not find the dirty children (LearnerGroup) nor does it actually prioritize those models (using morango_model_dependencies). The result of the behavior is an uncaught FK constraint, which fails deserialization.
Additionally, when Django processes deletions and the cascade to other related models, it ends up processing the parents before the children. When this is the case, and the field has a non-deferrable FK constraint, this will fail the sync immediately.
Morango's existing test models use a flat structure which does not immediately replicate this issue.
(model examples taken from Kolibri: https://github.com/learningequality/kolibri/blob/v0.19.4/kolibri/core/auth/models.py)
Errors and logs
Traceback (most recent call last):
File "/home/bjester/Projects/learningequality/kolibri/sync-sync-sync/.venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bjester/Projects/learningequality/kolibri/sync-sync-sync/.venv/lib/python3.12/site-packages/django/db/backends/sqlite3/base.py", line 423, in execute
return Database.Cursor.execute(self, query, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.IntegrityError: FOREIGN KEY constraint failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/bjester/Projects/learningequality/morango/s2s-fk-crash/morango/sync/controller.py", line 254, in _invoke_middleware
result = middleware(prepared_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bjester/Projects/learningequality/morango/s2s-fk-crash/morango/registry.py", line 204, in __call__
result = operation(context)
^^^^^^^^^^^^^^^^^^
File "/home/bjester/Projects/learningequality/morango/s2s-fk-crash/morango/sync/operations.py", line 989, in __call__
result = self.handle(context)
^^^^^^^^^^^^^^^^^^^^
File "/home/bjester/Projects/learningequality/morango/s2s-fk-crash/morango/sync/operations.py", line 1326, in handle
_deserialize_from_store(context.sync_session.profile, filter=context.filter)
File "/home/bjester/Projects/learningequality/morango/s2s-fk-crash/morango/sync/operations.py", line 552, in _deserialize_from_store
app_model, _ = store_model._deserialize_store_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bjester/Projects/learningequality/morango/s2s-fk-crash/morango/models/core.py", line 494, in _deserialize_store_model
klass_model.syncing_objects.filter(id=self.id).delete()
File "/home/bjester/Projects/learningequality/kolibri/sync-sync-sync/.venv/lib/python3.12/site-packages/django/db/models/query.py", line 746, in delete
deleted, _rows_count = collector.delete()
^^^^^^^^^^^^^^^^^^
File "/home/bjester/Projects/learningequality/kolibri/sync-sync-sync/.venv/lib/python3.12/site-packages/django/db/models/deletion.py", line 429, in delete
count = query.delete_batch(pk_list, self.using)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bjester/Projects/learningequality/kolibri/sync-sync-sync/.venv/lib/python3.12/site-packages/django/db/models/sql/subqueries.py", line 43, in delete_batch
num_deleted += self.do_query(self.get_meta().db_table, self.where, using=using)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bjester/Projects/learningequality/kolibri/sync-sync-sync/.venv/lib/python3.12/site-packages/django/db/models/sql/subqueries.py", line 23, in do_query
cursor = self.get_compiler(using).execute_sql(CURSOR)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bjester/Projects/learningequality/kolibri/sync-sync-sync/.venv/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
cursor.execute(sql, params)
File "/home/bjester/Projects/learningequality/kolibri/sync-sync-sync/.venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bjester/Projects/learningequality/kolibri/sync-sync-sync/.venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bjester/Projects/learningequality/kolibri/sync-sync-sync/.venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 79, in _execute
with self.db.wrap_database_errors:
File "/home/bjester/Projects/learningequality/kolibri/sync-sync-sync/.venv/lib/python3.12/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/bjester/Projects/learningequality/kolibri/sync-sync-sync/.venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bjester/Projects/learningequality/kolibri/sync-sync-sync/.venv/lib/python3.12/site-packages/django/db/backends/sqlite3/base.py", line 423, in execute
return Database.Cursor.execute(self, query, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.IntegrityError: FOREIGN KEY constraint failed
Expected behavior
For model creations within the hierarchy, the existing behavior will inadvertently order the creations so parents are created before children (because it doesn't matching children when processing a parent). Although, for deletions, the processing should occur in reverse so that deepest deletions within the hierarchy are processed first. The existing morango_model_dependencies can be used to develop order for deletions within the hierarchy, as long as the hierarchy across models can be inferred-- this can be done inspecting the subclasses of models, but it may be more helpful to define another model attribute, like morango_model_collection.
A new morango_model_collection attribute could be handled as such:
- For flat hierarchies (like existing test models), the lack of an explicit
morango_model_collection means it uses the morango_model_name by default.
- For complex hierarchies (like Kolibri), the
morango_model_collection should be defined by syncable models. The existing morango_model_dependencies attribute can be used to order model operations within the collection.
Once a model with a collection is encountered, the deserialization processes all deletions for the collection in reverse order (child -> parent), and adds state tracking for marking the collection as processed, so that the collection's deletions aren't re-processed if it's encountered again for complex hierarchies.
For creates and updates to models, the existing logic which tries to process dirty children will likely be unnecessary, as long as parents are created before their children.
User-facing consequences
In Kolibri, this causes all out failure of deserialization during a sync, because of its recent hardening of FK constraints. A complete breakage of the syncing functionality.
Steps to reproduce
- In Kolibri, create a hierarchy with a Facility containing a Classroom and the Classroom containing a LearnerGroup.
- Sync Kolibri to another device.
- On the original device, delete the Classroom and LearnerGroup.
- Sync Kolibri to the other device again.
- The sync should fail to deserialize, like the error above.
Context
Observed:
Morango 0.8.10
Kolibri 0.19.1+
Target branch: release-v0.8.x
Observed behavior
Morango's handling of self-referential models assumes that all of the reference models have the same
morango_model_name. When an implementation uses model hierarchies like a base model (Collection) which is then inherited by other models (Facility,Classroom,LearnerGroup), Morango may not properly find dirty children because it filters bymorango_model_name. In the case of deletions in the hierarchy (aClassroomand its childLearnerGroupare deleted), the parent deletion will occur first because it does not find the dirty children (LearnerGroup) nor does it actually prioritize those models (usingmorango_model_dependencies). The result of the behavior is an uncaught FK constraint, which fails deserialization.Additionally, when Django processes deletions and the cascade to other related models, it ends up processing the parents before the children. When this is the case, and the field has a non-deferrable FK constraint, this will fail the sync immediately.
Morango's existing test models use a flat structure which does not immediately replicate this issue.
(model examples taken from Kolibri: https://github.com/learningequality/kolibri/blob/v0.19.4/kolibri/core/auth/models.py)
Errors and logs
Expected behavior
For model creations within the hierarchy, the existing behavior will inadvertently order the creations so parents are created before children (because it doesn't matching children when processing a parent). Although, for deletions, the processing should occur in reverse so that deepest deletions within the hierarchy are processed first. The existing
morango_model_dependenciescan be used to develop order for deletions within the hierarchy, as long as the hierarchy across models can be inferred-- this can be done inspecting the subclasses of models, but it may be more helpful to define another model attribute, likemorango_model_collection.A new
morango_model_collectionattribute could be handled as such:morango_model_collectionmeans it uses themorango_model_nameby default.morango_model_collectionshould be defined by syncable models. The existingmorango_model_dependenciesattribute can be used to order model operations within the collection.Once a model with a collection is encountered, the deserialization processes all deletions for the collection in reverse order (child -> parent), and adds state tracking for marking the collection as processed, so that the collection's deletions aren't re-processed if it's encountered again for complex hierarchies.
For creates and updates to models, the existing logic which tries to process dirty children will likely be unnecessary, as long as parents are created before their children.
User-facing consequences
In Kolibri, this causes all out failure of deserialization during a sync, because of its recent hardening of FK constraints. A complete breakage of the syncing functionality.
Steps to reproduce
Context
Observed:
Morango 0.8.10
Kolibri 0.19.1+