Skip to content

refactor(core): use explicit sessions in repositories#1002

Open
phernandez wants to merge 6 commits into
mainfrom
codex/repository-explicit-sessions
Open

refactor(core): use explicit sessions in repositories#1002
phernandez wants to merge 6 commits into
mainfrom
codex/repository-explicit-sessions

Conversation

@phernandez

Copy link
Copy Markdown
Member

Summary

  • make the core Repository base explicit-session based, with no stored session_maker or repository-owned transaction lifecycle
  • migrate project/entity/note-content/observation/relation/project-info repositories and their service/API/CLI callers to caller-owned AsyncSession flows
  • add regression coverage for multi-repository operations in one transaction, same-session uncommitted visibility, and rollback behavior
  • keep search repository changes minimal and scoped to call-site fallout

Closes #750

Verification

  • uv run pytest tests/repository/test_repository.py tests/repository/test_explicit_session_transactions.py tests/repository/test_project_repository.py tests/repository/test_entity_repository.py tests/repository/test_note_content_repository.py tests/repository/test_observation_repository.py tests/repository/test_relation_repository.py tests/repository/test_project_info_repository.py -q
  • uv run pytest tests/services/test_project_service_operations.py tests/services/test_project_service_embedding_status.py tests/services/test_entity_service_disable_permalinks.py tests/services/test_semantic_search.py -q
  • uv run pytest tests/mcp/test_tool_write_note.py tests/mcp/test_tool_read_note.py tests/mcp/test_permalink_collision_file_overwrite.py -q
  • uv run pytest tests/sync/test_sync_service.py tests/sync/test_sync_service_incremental.py tests/sync/test_character_conflicts.py tests/sync/test_watch_service.py tests/sync/test_watch_service_atomic_adds.py tests/sync/test_watch_service_reload.py tests/sync/test_sync_wikilink_issue.py -q
  • uv run pytest tests/api/v2/test_memory_router.py tests/api/v2/test_search_router.py tests/api/v2/test_knowledge_router.py tests/api/v2/test_knowledge_router_telemetry.py tests/api/v2/test_resource_router.py tests/api/v2/test_schema_router.py tests/test_deps.py -q
  • uv run pytest tests/api/v2/test_project_router.py::test_add_project_response_reflects_promoted_default tests/api/v2/test_project_router.py::test_add_project_idempotent_response_reflects_promoted_default tests/api/v2/test_project_router.py::test_list_projects_handles_missing_config_default tests/api/v2/test_project_router.py::test_set_default_project_without_existing_default tests/api/v2/test_project_router.py::test_resolve_project_success_by_external_id tests/api/v2/test_project_router.py::test_resolve_project_success_by_name tests/api/v2/test_project_router.py::test_resolve_project_success_by_permalink tests/api/v2/test_project_router.py::test_resolve_project_accepts_workspace_qualified_identifier tests/api/v2/test_project_router.py::test_resolve_project_not_found_fresh_install_names_setup_command -q --tb=short
  • just fix
  • just format
  • uv run ty check src tests test-int
  • just doctor
  • just package-check

Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bb65be817b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

rel.target,
strict=True,
load_relations=False,
session=active_session,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Don't share one AsyncSession across relation tasks

When a note has multiple relations, the surrounding asyncio.gather launches one resolve_link task per relation but now passes the same active_session into every task. AsyncSession is not safe for concurrent operations (asyncpg/Postgres rejects overlapping commands), and because this gather uses return_exceptions=True those failures are then treated as unresolved links, so MCP/API writes can persist forward references even when targets exist. Resolve these lookups sequentially on the transaction session or give each concurrent task its own session.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a6ad20a: relation target resolution now runs sequentially inside the active transaction session, so one AsyncSession is no longer shared across concurrent tasks. I also narrowed exception handling so cancellation still propagates. Verified with the focused upsert tests, typecheck, and the broad testmon run aside from the local OpenAI quota-only semantic case.

Comment thread src/basic_memory/sync/sync_service.py Outdated
scan_coro = self._scan_directory_modified_since(
directory, project.last_scan_timestamp
)
async with self._session_scope() as session:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Narrow the scan database session scope

This session now wraps the project lookup, _quick_count_files, the full/incremental filesystem scan, and per-file stat/checksum work before it exits. On large projects those non-DB operations can take seconds or minutes; after the first repository query the connection and transaction stay open for the whole scan, so concurrent watch/API syncs can exhaust the pool or leave Postgres idle-in-transaction. Scope sessions only around the project/entity repository calls and do filesystem work outside them.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a6ad20a: scan now snapshots project/entity/checksum/file-path state through short repository sessions, then releases the DB session before file counting, filesystem scans, stat checks, and checksum work. Verified with sync incremental/scan/watch tests in the broad testmon run.

Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor repositories to support caller-owned AsyncSession flows

1 participant