Sql-150: Temporary objects to the catalog

[SangJunBak]

This PR is meant to be merged with everything else in the stack and is split for review purposes

Motivation

sql-150

Description

1. design: update durable temporary objects design doc (de137ed708c8)

Initially the design doc stated that we wanted to persist sessions in the Catalog. Through benchmarking, it was found to create majoir regressions on CPS, even with optimizations. Thus we keep mz_sessions as a builtin table but will continue to persist durable objects.

We also change the requirement of temporary schemas being durable to being a followup given we don't need them to move builtin tables.

2. catalog: bump catalog version to 91 (88dd791836fd)

Generic catalog migration version bump. Copies everything and is intended to make the review easier.

3. catalog: add ephemeral_owner_session to durable items (fa94bb99cb93)

• We add the ephemeral_owner_session property in durable Items to indicate if an object is temporary
• We remove all ephemeral items in savepoint/write open of the catalog. We can't do this for readonly mode since readonly followers would remove it in memory but then eventually panic when it sees a retraction for it

4. catalog: make temporary items durable in the catalog shard (fd2f72fddd15)

• Temporary items now write real durable Item rows, marked with ephemeral_owner_session = the creating session's UUID
• The duplicated TemporaryItem is consolidated
  • Replaces catalog entries' From implementation with durable_item since we now need access to the connection <-> session mapping.
• We introduce two map state variables: ephemeral_owner_conns_by_uuid and ephemeral_owner_uuids_by_conn. Both serve to create a session UUID <-> Conneciton ID mapping. These are used for two purposes:
  • When resolving a temporary object, routing it to its temporary schema via the connection ID <-> Temporary schema mapping usingtemporary_schemas
  • When editing an object, as opposed to creating an object, to store the new object durably, we don't have access to the session UUID but have access to the connection ID. Thus we use these maps to grab it.
  • In a future commit, we'll be merging temporary_schemas with these map variables since all three have the same lifecycle and are all used together

5. adapter: unify temporary schemas with ephemeral owner registration (7b83fefba86e)

• The lifecycle and purpose of our ephemeral_* map state variables are very similar to temporary_schemas. That is, all lazily initialize in-memory temporary item metadata to resolve temporary items. Thus we unify all three in a struct TemporaryNamespaces
• Gets rid of extraneous lazy initialization of the mz_temp schema inside apply_item_update from before given the lazy initialization in the inner insert_entry is all we need
• Gets rid of eager mz_system temporary schema initialization.

Verification

• Future PR in the stack has a more in-depth test suite for testing cleanup, multiple generations, as well as our current tests for temporary objects
• I tested nightlies on a branch with all commits combined https://buildkite.com/materialize/nightly/builds/17923. I've since added the ci-nightly label to the latest PR in this stack

[linear-code]

SQL-150

[SangJunBak]

The body here is copied from the deleted create_temporary_schema below

[ggevay]

The fencing rationale is sound, and #38159/#38160 now cover the single-writer crash paths well (kill -9 item reclamation, read-only non-reclamation, and the pre-existing shard leak). In today's architecture there is no leak path at all: every crash is followed by some process's writable open, which purges.

The open question is the multi-process endgame this series builds toward: with several serving envds, one of them crashing leaks its sessions' temporary items until the next deploy, since a peer crash triggers no fence and no writable open. The design doc's mz_sessions follow-up sketches a durable envd-heartbeat to identify rows left by dead processes. Is ephemeral-item reclamation intended to ride on that same mechanism? A sentence in the design doc's follow-up section would settle it.