Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions benchmarks/string-bench/src/serialized.rs
Original file line number Diff line number Diff line change
Expand Up @@ -348,6 +348,8 @@ mod tests {
use vortex::VortexSessionDefault;
use vortex::array::Canonical;
use vortex::array::VortexSessionExecute;
use vortex::editions::CORE_2026_08_3;
use vortex::editions::EditionSessionExt;
use vortex::io::runtime::BlockingRuntime;
use vortex::io::runtime::current::CurrentThreadRuntime;
use vortex::io::session::RuntimeSessionExt;
Expand Down Expand Up @@ -404,6 +406,7 @@ mod tests {
let (column, expected_uncompressed_bytes) = crate::repeated_fixture();
let runtime = CurrentThreadRuntime::new();
let session = VortexSession::default().with_handle(runtime.handle());
session.enable_edition(CORE_2026_08_3)?;

for (encoder, expected_label) in [
(StringEncoder::OnPair, "onpair-12"),
Expand Down
45 changes: 25 additions & 20 deletions docs/specs/editions.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,9 @@ guarantee applies to that serialized contract, not to the in-memory implementati

Editions belong to independently versioned families and are cumulative within a family. Each edition includes all
components from the preceding edition in that family, plus any newly added components. A writer selects at most one
edition from each family and may use the union of their components. For example, selecting `core2026.07.0` and
`unstable2026.06.0` allows stable components released through July 2026 and unstable components released through June

2026.
edition from each family and may use the union of their components. For example, selecting `core2026.08.1` and
`preview2026.06.0` allows stable components released through August 2026 and preview components released through
June 2026.

The first frozen edition, `core2025.05.0`, contains the components that Vortex `0.36.0` could write. This marks the
start of the Vortex file format's stability guarantee. Every Vortex release from `0.36.0` onward can read
Expand Down Expand Up @@ -81,7 +80,7 @@ the previous selection.
You can change the default configuration to:

- **Target an older `core` edition** when the file must remain readable by an older Vortex deployment.
- **Enable another family** to use components outside `core`. Vortex currently defines `unstable`, `spatial`, and
- **Enable another family** to use components outside `core`. Vortex currently defines `preview`, `spatial`, and
`json` in addition to `core`.

Sessions created without the Vortex facade must register and enable their editions before writing files. The lower-level
Expand Down Expand Up @@ -210,43 +209,49 @@ Minimum Vortex release: `0.54.0`.

- `array`: `fastlanes.rle`, `vortex.fixed_size_list`, `vortex.listview`, `vortex.masked`

#### `core2026.07.0`

Minimum Vortex release: `0.65.0`.

- `array`: `vortex.variant`
- `dtype`: `vortex.uuid`

#### `core2026.08.0`

Minimum Vortex release: `0.84.0`.

- `array`: `vortex.map`
- `layout`: `vortex.zoned`
- `aggregate`: `vortex.bounded_max`, `vortex.bounded_min`, `vortex.max`, `vortex.min`,
`vortex.nan_count`, `vortex.null_count`

#### `core2026.08.1`

Minimum Vortex release: `0.84.0`.

- `array`: `vortex.map`

### Draft editions

Draft component lists may change and have no minimum reader or permanent compatibility guarantee.

#### `unstable2025.05.0`
#### `core2026.08.2`

- `array`: `vortex.parquet.variant`, `vortex.variant`
- `dtype`: `vortex.uuid`

#### `core2026.08.3`

- `array`: `vortex.onpair`

#### `preview2025.05.0`

- `array`: `fastlanes.delta`

#### `unstable2026.02.0`
#### `preview2026.02.0`

- `array`: `vortex.zstd_buffers`

#### `unstable2026.04.0`
#### `preview2026.04.0`

- `array`: `vortex.parquet.variant`, `vortex.patched`, `vortex.tensor.cosine_similarity`,
`vortex.tensor.inner_product`, `vortex.tensor.l2_norm`, `vortex.tensor.normalized`
- `array`: `vortex.patched`, `vortex.tensor.cosine_similarity`, `vortex.tensor.inner_product`,
`vortex.tensor.l2_norm`, `vortex.tensor.normalized`
- `dtype`: `vortex.tensor.fixed_shape_tensor`, `vortex.tensor.vector`

#### `unstable2026.06.0`
#### `preview2026.06.0`

- `array`: `vortex.onpair`
- `layout`: `vortex.list`

#### `spatial2026.08.0`
Expand Down
13 changes: 13 additions & 0 deletions vortex-datafusion/src/persistent/format.rs
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,19 @@ impl VortexFormatFactory {
}
}

/// Creates a factory with an explicit session and session-driven table options.
///
/// Formats created by this factory start from the DataFusion session's `vortex` options,
/// falling back to [`VortexTableOptions::default`]. Table-level `OPTIONS(...)` are still
/// applied last.
pub fn new_with_session(session: VortexSession) -> Self {
Self {
session,
options: None,
expression_convertor: None,
}
}

/// Creates a factory with an explicit session and factory-level options.
///
/// The supplied options become the complete starting value for every
Expand Down
5 changes: 4 additions & 1 deletion vortex-datafusion/src/persistent/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ use vortex::array::arrays::VarBinArray;
use vortex::array::validity::Validity;
use vortex::buffer::Buffer;
use vortex::buffer::buffer;
use vortex::editions::CORE_2026_08_3;
use vortex::editions::EditionSessionExt;
use vortex::file::OpenOptionsSessionExt;
use vortex::file::WriteOptionsSessionExt;
use vortex::io::VortexWrite;
Expand Down Expand Up @@ -497,8 +499,8 @@ async fn arrow_uuid_extension_roundtrip() -> anyhow::Result<()> {
use vortex_arrow::ArrowSessionExt;

let ctx = TestSessionContext::default();
// Default vortex session has importer/exporter for Arrow UUID
let session = VortexSession::default();
session.enable_edition(CORE_2026_08_3)?;

let mut uuid_field = Field::new("id", DataType::FixedSizeBinary(16), false);
uuid_field.try_with_extension_type(Uuid)?;
Expand Down Expand Up @@ -565,6 +567,7 @@ async fn arrow_uuid_extension_roundtrip_nested_struct() -> anyhow::Result<()> {

let ctx = TestSessionContext::default();
let session = VortexSession::default();
session.enable_edition(CORE_2026_08_3)?;

let mut inner_uuid_field = Field::new("id", DataType::FixedSizeBinary(16), false);
inner_uuid_field.try_with_extension_type(Uuid)?;
Expand Down
7 changes: 7 additions & 0 deletions vortex-duckdb/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,11 @@ use std::sync::OnceLock;

use vortex::VortexSessionDefault;
use vortex::cloud::Registry;
use vortex::editions::CORE_2026_08_3;
use vortex::editions::EditionSessionExt;
use vortex::error::VortexExpect;
use vortex::error::VortexResult;
use vortex::error::vortex_err;
use vortex::io::runtime::BlockingRuntime;
use vortex::io::runtime::current::CurrentThreadRuntime;
use vortex::io::session::RuntimeSessionExt;
Expand Down Expand Up @@ -48,6 +51,10 @@ static RUNTIME: LazyLock<CurrentThreadRuntime> = LazyLock::new(CurrentThreadRunt
static REGISTRY: LazyLock<Registry> = LazyLock::new(Registry::new);
static SESSION: LazyLock<VortexSession> = LazyLock::new(|| {
let session = VortexSession::default().with_handle(RUNTIME.handle());
session
.enable_edition(CORE_2026_08_3)
.map_err(|error| vortex_err!("{error}"))
.vortex_expect("DuckDB-supported draft core edition is registered");
vortex_spatial::initialize(&session);
session
});
Expand Down
8 changes: 8 additions & 0 deletions vortex-jni/src/session.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ use jni::EnvUnowned;
use jni::objects::JClass;
use jni::sys::jlong;
use vortex::VortexSessionDefault;
use vortex::editions::CORE_2026_08_3;
use vortex::editions::EditionSessionExt;
use vortex::error::VortexExpect;
use vortex::error::vortex_err;
use vortex::io::runtime::BlockingRuntime;
use vortex::io::session::RuntimeSessionExt;
use vortex::session::VortexSession;
Expand All @@ -17,6 +21,10 @@ use crate::RUNTIME;
/// an opaque pointer that Java must pass to [`Java_dev_vortex_jni_NativeSession_free`].
pub(crate) fn new_session() -> Box<VortexSession> {
let session = VortexSession::default().with_handle(RUNTIME.handle());
session
.enable_edition(CORE_2026_08_3)
.map_err(|error| vortex_err!("{error}"))
.vortex_expect("JNI-supported draft core edition is registered");
vortex_parquet_variant::initialize(&session);
vortex_spatial::initialize(&session);
Box::new(session)
Expand Down
4 changes: 2 additions & 2 deletions vortex-python/src/io.rs
Comment thread
robert3005 marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,7 @@ impl PyVortexWriteOptions {
/// >>> vx.io.VortexWriteOptions.default().write(sprl, "chonky.vortex")
/// >>> import os
/// >>> os.path.getsize('chonky.vortex')
/// 215716
/// 215684
///
/// Wow, Vortex manages to use about two bytes per integer! So advanced. So tiny.
///
Expand All @@ -323,7 +323,7 @@ impl PyVortexWriteOptions {
///
/// >>> vx.io.VortexWriteOptions.compact().write(sprl, "tiny.vortex")
/// >>> os.path.getsize('tiny.vortex')
/// 54920
/// 54888
///
/// Random numbers are not (usually) composed of random bytes!
#[staticmethod]
Expand Down
8 changes: 7 additions & 1 deletion vortex-sqllogictest/bin/sqllogictests-runner.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ use sqllogictest::harness::Arguments;
use sqllogictest::harness::Failed;
use sqllogictest::harness::Trial;
use sqllogictest::strict_column_validator;
use vortex::VortexSessionDefault;
use vortex::editions::CORE_2026_08_3;
use vortex::editions::EditionSessionExt;
use vortex::session::VortexSession;
use vortex_datafusion::VortexFormatFactory;
use vortex_datafusion::VortexTableOptions;
use vortex_sqllogictest::duckdb::DuckDB;
Expand Down Expand Up @@ -64,7 +68,9 @@ fn drive_datafusion(path: &Path, work_dir: &Path, mode: Mode) -> anyhow::Result<
let rt = build_runtime()?;
rt.block_on(async {
let config = SessionConfig::default().with_option_extension(VortexTableOptions::default());
let factory = Arc::new(VortexFormatFactory::new());
let vortex_session = VortexSession::default();
vortex_session.enable_edition(CORE_2026_08_3)?;
let factory = Arc::new(VortexFormatFactory::new_with_session(vortex_session));
let session_state_builder = SessionStateBuilder::new()
.with_config(config)
.with_default_features()
Expand Down
14 changes: 12 additions & 2 deletions vortex-test/compat-gen/src/adapter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -87,11 +87,21 @@ pub fn write_compressed(
pub fn write_compressed_to_bytes(
chunk: ArrayRef,
strategy: Arc<dyn LayoutStrategy>,
) -> VortexResult<ByteBuffer> {
write_compressed_to_bytes_with_session(&VortexSession::default(), chunk, strategy)
}

/// Write a `.vortex` file into memory using a caller-provided session and layout strategy.
pub fn write_compressed_to_bytes_with_session(
session: &VortexSession,
chunk: ArrayRef,
strategy: Arc<dyn LayoutStrategy>,
) -> VortexResult<ByteBuffer> {
let stream = ArrayStreamAdapter::new(chunk.dtype().clone(), stream::iter([Ok(chunk)]));
let session = session.clone();

runtime()?.block_on(async {
let session = VortexSession::default().with_tokio();
runtime()?.block_on(async move {
let session = session.with_tokio();
let mut bytes = Vec::new();
let _summary = session
.write_options()
Expand Down
33 changes: 21 additions & 12 deletions vortex-test/compat-gen/src/fixtures/arrays/datasets/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,15 @@ pub fn fixtures() -> Vec<Box<dyn DatasetFixture>> {

#[cfg(test)]
mod tests {
use vortex::VortexSessionDefault;
use vortex::compressor::BtrBlocksCompressorBuilder;
use vortex::editions::CORE_2026_08_3;
use vortex::editions::EditionSessionExt;
use vortex::file::WriteStrategyBuilder;
use vortex_array::array_session;
use vortex::session::VortexSession;
use vortex_arrow::ArrowSessionExt;
use vortex_error::VortexResult;
use vortex_error::vortex_err;

use super::fixtures;
use crate::adapter;
Expand All @@ -29,28 +34,32 @@ mod tests {
}

#[test]
fn roundtrip_non_clickbench_fixtures_to_bytes() {
let session = array_session();
fn roundtrip_non_clickbench_fixtures_to_bytes() -> VortexResult<()> {
let session = VortexSession::default();
session
.enable_edition(CORE_2026_08_3)
.map_err(|error| vortex_err!("{error}"))?;
for dataset in fixtures()
.into_iter()
.filter(|fixture| !is_clickbench_fixture(fixture.name()))
{
let array = dataset.build(&session.arrow()).unwrap();
let regular_bytes = adapter::write_compressed_to_bytes(
let array = dataset.build(&session.arrow())?;
let regular_bytes = adapter::write_compressed_to_bytes_with_session(
&session,
array.clone(),
WriteStrategyBuilder::default().build(),
)
.unwrap();
let _regular = adapter::read_file(regular_bytes).unwrap();
)?;
let _regular = adapter::read_file(regular_bytes)?;

let compact_bytes = adapter::write_compressed_to_bytes(
let compact_bytes = adapter::write_compressed_to_bytes_with_session(
&session,
array,
WriteStrategyBuilder::default()
.with_btrblocks_builder(BtrBlocksCompressorBuilder::default().with_compact())
.build(),
)
.unwrap();
let _compact = adapter::read_file(compact_bytes).unwrap();
)?;
let _compact = adapter::read_file(compact_bytes)?;
}
Ok(())
}
}
9 changes: 6 additions & 3 deletions vortex/src/editions/core/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,14 @@
pub mod v2025_05;
pub mod v2025_06;
pub mod v2025_10;
pub mod v2026_07;
pub mod v2026_08;
pub mod v2026_08_2;
pub mod v2026_08_3;

pub use v2025_05::CORE_2025_05_0;
pub use v2025_06::CORE_2025_06_0;
pub use v2025_10::CORE_2025_10_0;
pub use v2026_07::CORE_2026_07_0;
pub use v2026_08::CORE_2026_08;
pub use v2026_08::CORE_2026_08_0;
pub use v2026_08::CORE_2026_08_1;
pub use v2026_08_2::CORE_2026_08_2;
pub use v2026_08_3::CORE_2026_08_3;
25 changes: 18 additions & 7 deletions vortex/src/editions/core/v2026_08.rs
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright the Vortex contributors

//! The August 2026 core edition adding canonical Map arrays and zoned layouts.
//! The frozen August 2026 core editions.

use vortex_edition::Edition;
use vortex_edition::EditionDeclaration;
use vortex_edition::EditionId;
use vortex_edition::EditionMember;

/// The August 2026 core edition containing canonical Map arrays and zoned layouts.
pub const CORE_2026_08: EditionId = EditionId::new("core", 2026, 8, 0);
/// The August 2026 core edition containing zoned layouts.
pub const CORE_2026_08_0: EditionId = EditionId::new("core", 2026, 8, 0);

/// The declaration of [`CORE_2026_08`] and the components that join the family at it.
/// The declaration of [`CORE_2026_08_0`] and the components that join the family at it.
///
/// The aggregates are the set the default writer records in zone maps. A strategy asking for an
/// aggregate outside this set fails the write instead of producing zone maps an older reader would
Expand All @@ -20,13 +20,12 @@ pub const CORE_2026_08: EditionId = EditionId::new("core", 2026, 8, 0);
/// `vortex.sum` is deliberately not a member: zone maps prune, a zone sum does not, and its
/// null-on-empty semantics were changed and reverted within a single week. The writer no longer
/// records it, so the two stay consistent.
pub static DECLARATION: EditionDeclaration = EditionDeclaration {
pub static DECLARATION_0: EditionDeclaration = EditionDeclaration {
edition: Edition {
id: CORE_2026_08,
id: CORE_2026_08_0,
min_vortex_version: Some("0.84.0"),
},
added: &[
EditionMember::array(&"vortex.map"),
EditionMember::layout(&"vortex.zoned"),
EditionMember::aggregate(&"vortex.bounded_max"),
EditionMember::aggregate(&"vortex.bounded_min"),
Expand All @@ -36,3 +35,15 @@ pub static DECLARATION: EditionDeclaration = EditionDeclaration {
EditionMember::aggregate(&"vortex.null_count"),
],
};

/// The second August 2026 edition of the `core` family, adding OnPair arrays.
pub const CORE_2026_08_1: EditionId = EditionId::new("core", 2026, 8, 1);

/// The declaration of [`CORE_2026_08_1`] and the components that join the family at it.
pub static DECLARATION_1: EditionDeclaration = EditionDeclaration {
edition: Edition {
id: CORE_2026_08_1,
min_vortex_version: Some("0.84.0"),
},
added: &[EditionMember::array(&"vortex.onpair")],
};
Loading
Loading