Tags: vortex-data/vortex
Tags
Split `TableStrategy` dispatcher from `StructStrategy` (#8638) Splits the `TableStrategy` writer into a dtype and override dependent dispatcher, and a struct writer. - **`StructStrategy`** — a *structural* writer that only shreds struct streams. It transposes a struct chunk stream into one ordered stream per field (plus validity when nullable) and writes each through a child strategy resolved **by direct field name** (`field_writers` entry, else `default`; validity through `validity`). It is intentionally dtype-unaware and knows nothing about nested dtypes or field-path overrides; it bails on non-struct input. - **`TableStrategy`** — now a *dispatcher*. `write_stream` routes on the stream's dtype and configured per-field overrides. `TableStrategy` continues to own the field-path overrides and resolves the per-field child strategies before handing them to `StructStrategy`. This lets the writer dispatch different layout kinds per dtype — e.g. a list layout for list columns when a list strategy is configured — instead of `TableStrategy` hard-coding struct shredding as its only nested behavior. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
feat(geo): add native `LineString`, `MultiPoint`, `MultiLineString` t… …ypes (#8679) Complete the set of OGC simple-feature geometries alongside the existing Point/Polygon/MultiPolygon: - vortex.geo.linestring - List<Struct<x, y[, z][, m]>> - vortex.geo.multipoint - List<Struct<x, y[, z][, m]>> - vortex.geo.multilinestring - List<List<Struct<x, y[, z][, m]>>> Each mirrors the existing Polygon/MultiPolygon extension types. Wire them through the DuckDB integration so they surface as GEOMETRY like the existing native types: the dtype -> GEOMETRY mapping, per-type WKB exporters, and native-geometry-column recognition for predicate pushdown. Tests cover dtype validation across all four dimensions, Arrow field import/export (including that the shape-identical types resolve by name), and WKB-literal decode round-trips. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
feat(vortex-geo): native Point extension type and GeoDistance scalar … …function (#8372) ## Summary This PR adds a native point type to `vortex-geo`. Points are by far the most common geometry in analytical datasets, and a columnar representation makes their coordinates directly accessible without parsing WKB. It also adds the scalar function: point-to-point distance with PostGIS `ST_Distance` semantics (planar/Euclidean, results in CRS units). ## API Changes Adds to `vortex-geo`, all registered through `vortex_geo::initialize`: - Extension type `Point` (`vortex.geo.point`): a location stored as `Struct<x, y, z?, m?>` of non-nullable `f64`, where `z?` is an optional elevation and `m?` an optional measure. - `Coordinate`: the internal value a point scalar unpacks to. - Scalar function `GeoDistance` (`vortex.geo.distance`): per-row distance between two equal-length point columns; either or both operands may be constant, in which case the query point is decoded once and broadcast. ## Testing Unit tests cover dtype validation for every GeoArrow dimension (and rejection of invalid storage), round-tripping a point column through scalar execution back to the original coordinates, WKT display for all four dimensions, and distance over all operand shapes: column-to-constant (either side), column-to-column, and constant-to-constant. --- Supersedes #8342 (same change, moved from my fork to an in-repo branch). --------- Signed-off-by: Nemo Yu <zyu379@wisc.edu> Signed-off-by: Nemo Yu <zhenghong@spiraldb.com> Signed-off-by: Nemo Yu <83347615+HarukiMoriarty@users.noreply.github.com> Signed-off-by: "Nemo Yu" <zhenghong@spiraldb.com> Co-authored-by: Joe Isaacs <joe.isaacs@live.co.uk>
TurboQuant: better centroid initialization (#8116) ## Summary Tracking issue: #7830 Followup of #8076 Changes the centroid initialization for `vortex-turboquant` to use `+- sqrt(bit_width) * sigma` instead of just [-1, 1] ## Testing ```sh uv run benchmarks/vector-search-bench/scripts/plot-turboquant-distortion.py \ --dataset cohere-small-100k:single \ --dataset openai-medium-500k:single \ --dataset bioasq-medium-1m \ --dataset glove-small-100k \ --dataset gist-small-100k \ --dataset sift-small-500k \ --output ~/Downloads/distortion-sqrt-sigma.png ``` <img width="2816" height="950" alt="distortion-3-sigma" src="https://github.com/user-attachments/assets/cb2db4eb-13a7-4083-949b-ee7f04ab7428" /> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Avoid re-building vortex-duckdb twice (#8035) ## Summary Currently, if you build something like `duckdb-bench` cargo will rebuild `vortex-duckdb` once before it'll be satisfied that it doesn't need to rebuild it. The root cause is that cargo's fingerprinting can't accurately track files that are created **during** the build. This change is fine because we still track our own changes, the only external change will be something like a duckdb version upgrade which will change `build.rs` and re-trigger the build as expected. Signed-off-by: Adam Gutglick <adam@spiraldb.com>
bench: bit-packed compare-constant baseline (#8012) ## Summary - Adds a divan benchmark `bitpack_compare` in `vortex-fastlanes` that compares an `Operator::Eq` / `Operator::Lt` against an out-of-range constant on a `BitPackedData` array vs. an explicit "decompress, then Arrow compare" baseline that materialises the unpacked `PrimitiveArray` first. - The constant is chosen as `1 << BW`, just past the packable range, so a future compare-constant kernel can recognise it and short-circuit. Today both arms decompress; this PR establishes a baseline for that follow-up to land against. - Grid sized for fast runs: `len ∈ {1024, 65536}`, `bit_width ∈ {4, 16}`, Eq + Lt. The follow-up optimization (out-of-range fast path on `BitPacked`, plus the `bitpack_constant` analytical encoder) is in #PR2-PLACEHOLDER, stacked on this branch. Splitting the bench out lets the speedup PR show concrete numbers against this measured baseline. ## Test plan - [x] `cargo check -p vortex-fastlanes --benches` - [ ] `cargo bench -p vortex-fastlanes --bench bitpack_compare` records the slow baseline numbers prior to the follow-up landing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Skip validity dispatch for non-nullable arrays (#7748) Two small optimizations: - Skip the `is_invalid` vtable dispatch when `dtype` is non-nullable - Demote the dtype-equality post-check to debug assertion. It's an encoding-correctness invariant, not runtime input validation Signed-off-by: Baris Palaska <barispalaska@gmail.com>
Split up the monster `ci.yml` workflow (#7472) ## Summary `ci.yml` is reaching 1000 lines of YAML, which no one wants. This PR splits into 5 files: 1. ci.yml - now only contains linters and test runs 2. codspeed.yml - just codspeed benchmarks 3. cuda.yaml - all the cuda incantations 4. publish-dry-runs.yaml - all the dark magic that we have to try and make sure we can actually publish 5. rust-instrumented.yaml - rust-specific jobs that require nightly and some extra instrumentations (coverage, miri, etc) The real magic here is that there are no changes to the job names, so all rules still apply! This also makes github retries much easier, as github only allows retries after the whole workflow finishes. --------- Signed-off-by: Adam Gutglick <adam@spiraldb.com>
PreviousNext