Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions .bumpversion.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,6 @@ filename = "crates/lance-csr/Cargo.toml"
search = 'version = "{current_version}"'
replace = 'version = "{new_version}"'

[[tool.bumpversion.files]]
filename = "crates/lance-csr-catalog/Cargo.toml"
search = 'version = "{current_version}"'
replace = 'version = "{new_version}"'

[[tool.bumpversion.files]]
filename = "crates/lance-csr-benches/Cargo.toml"
search = 'version = "{current_version}"'
Expand Down
11 changes: 5 additions & 6 deletions .github/workflows/python-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -217,8 +217,8 @@ jobs:

publish-crates:
# After PyPI, so the two registries never disagree. If only this job fails, rerun it on
# its own; the wheels don't need rebuilding. The crates.io trusted publishers for both
# crates are registered for this workflow and environment `crates-io`.
# its own; the wheels don't need rebuilding. The crates.io trusted publisher for the
# crate is registered for this workflow and environment `crates-io`.
needs: publish
runs-on: ubuntu-24.04
timeout-minutes: 60
Expand All @@ -237,14 +237,13 @@ jobs:
sudo apt-get install -y protobuf-compiler

# Verifying compiles lance, so do it before getting the token, which lasts 30 minutes.
# Cargo packages lance-csr-catalog first, since lance-csr depends on it.
- name: Package and verify the crates
run: cargo package --locked -p lance-csr-catalog -p lance-csr
- name: Package and verify the crate
run: cargo package --locked -p lance-csr

- uses: rust-lang/crates-io-auth-action@v1
id: auth

- name: Publish to crates.io
run: cargo publish --locked --no-verify -p lance-csr-catalog -p lance-csr
run: cargo publish --locked --no-verify -p lance-csr
env:
CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }}
3 changes: 1 addition & 2 deletions .github/workflows/python-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,7 @@ jobs:
shared-key: "lance-csr-python"
# maturin builds the bindings crate, which sits outside the root
# workspace. The root workspace is listed too so the cache key hashes
# the manifests of lance-csr and lance-csr-catalog, which the bindings
# use by path; otherwise a dependency or feature change there keeps
# the manifest of lance-csr, which the bindings use by path; otherwise a dependency or feature change there keeps
# restoring a stale cache. Its target directory is never built here.
workspaces: |
. -> target
Expand Down
1 change: 0 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@

## Project Structure & Module Organization
- `crates/lance-csr/` hosts the Rust Cypher engine; keep new modules under `src/` and co-locate helpers inside `query/` or feature-specific submodules.
- `crates/lance-csr-catalog/` provides the graph source catalog and `DirNamespace` (Unity Catalog and Delta Lake support were removed).
- `crates/lance-csr-benches/` contains performance benchmarks in a dedicated unpublished crate; add new benchmarks here rather than in the main crate.
- `crates/lance-csr-python/src/` contains the PyO3 bridge; `python/python/lance_csr/` holds the pure-Python facade and packaging metadata.
- `python/python/tests/` stores functional tests; mirror new features with targeted cases here and in the corresponding Rust module.
Expand Down
15 changes: 0 additions & 15 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
[workspace]
members = [
"crates/lance-csr",
"crates/lance-csr-catalog",
"crates/lance-csr-benches",
]
# Python package needs to be built by maturin.
Expand Down
17 changes: 7 additions & 10 deletions RELEASE.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Releasing lance-csr

A release publishes the Python package to [PyPI](https://pypi.org/project/lance-csr/) and the
`lance-csr-catalog` and `lance-csr` crates to [crates.io](https://crates.io/crates/lance-csr). PyPI publishing is
`lance-csr` crate to [crates.io](https://crates.io/crates/lance-csr). PyPI publishing is
automated from a GitHub release, and the crates follow once PyPI has the new version.

The whole release takes about 50 minutes: the macOS wheel build, then the crates. Don't add steps that build the
Expand All @@ -26,9 +26,9 @@ cargo update --workspace --manifest-path crates/lance-csr-python/Cargo.toml
(cd python && uv lock)
```

`bump-my-version` updates `.bumpversion.toml`, the four crate `Cargo.toml` files and `python/pyproject.toml`. It
`bump-my-version` updates `.bumpversion.toml`, the three crate `Cargo.toml` files and `python/pyproject.toml`. It
doesn't update the three lockfiles that also carry the version, which is what the other three commands are for.
The diff should be 9 files and 15 lines each way, touching only version lines. If `uv lock` or `cargo update`
The diff should be 8 files and 11 lines each way, touching only version lines. If `uv lock` or `cargo update`
moves a dependency, undo that and update only the workspace packages. Add `--allow-dirty` to `bump-my-version` if
untracked files in your checkout stop it.

Expand Down Expand Up @@ -68,12 +68,12 @@ Publishing it starts `.github/workflows/python-publish.yml`:
| `build-macos` | about 35 min | macOS arm64 wheel, built natively on `macos-15` |
| `test` | under 1 min each | each wheel on Python 3.12, 3.13 and 3.14 |
| `publish` | seconds | checks the tag matches `python/pyproject.toml`, then uploads through PyPI trusted publishing |
| `publish-crates` | about 10 to 15 min | after `publish`: packages and verifies `lance-csr-catalog` and `lance-csr`, then uploads them through crates.io trusted publishing |
| `publish-crates` | about 10 to 15 min | after `publish`: packages and verifies `lance-csr`, then uploads it through crates.io trusted publishing |

The build jobs run in parallel. Nothing reaches PyPI unless every build and test job passes, and nothing
reaches crates.io unless PyPI publishing succeeds. The `pypi` and `crates-io` environments only accept `v*`
tags. The PyPI trusted publisher is registered for workflow `python-publish.yml` and environment `pypi`, and
the crates.io trusted publishers for both crates for the same workflow and environment `crates-io`, so don't
the crates.io trusted publisher for `lance-csr` for the same workflow and environment `crates-io`, so don't
rename any of them.

**Don't run a separate `mode=dry_run` before a release.** The release run builds and tests the same wheels
Expand All @@ -94,11 +94,8 @@ If a build or test job failed, nothing has been published, so nothing needs undo
PyPI already has the release, so don't release again. Rerun just that job with `gh run rerun <run id> --failed`;
the wheels aren't rebuilt.

Cargo publishes `lance-csr-catalog` first, since `lance-csr` depends on it. If it published the catalog and
then failed on `lance-csr`, a rerun fails because the catalog version already exists. Publish the rest by hand
from the release tag: a crate owner runs `cargo publish -p lance-csr`. If **Require trusted publishing** is on
for the crate, crates.io rejects API tokens, so turn it off for the manual publish and back on afterwards.
`lance-csr-python` and `lance-csr-benches` are `publish = false`.
`lance-csr-python` and `lance-csr-benches` are `publish = false`. Up to 0.7.1, the catalog types shipped
in a separate `lance-csr-catalog` crate; it is no longer published.

## 4. Check the release

Expand Down
24 changes: 0 additions & 24 deletions crates/lance-csr-catalog/Cargo.toml

This file was deleted.

1 change: 0 additions & 1 deletion crates/lance-csr-catalog/LICENSE

This file was deleted.

1 change: 0 additions & 1 deletion crates/lance-csr-catalog/NOTICE

This file was deleted.

34 changes: 0 additions & 34 deletions crates/lance-csr-catalog/README.md

This file was deleted.

38 changes: 0 additions & 38 deletions crates/lance-csr-catalog/src/lib.rs

This file was deleted.

3 changes: 0 additions & 3 deletions crates/lance-csr-catalog/src/namespace/mod.rs

This file was deleted.

14 changes: 0 additions & 14 deletions crates/lance-csr-python/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion crates/lance-csr/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ datafusion-expr = "54"
datafusion-sql = "54"
datafusion-functions-aggregate = "54"
futures = "0.3"
lance-csr-catalog = { path = "../lance-csr-catalog", version = "0.7.1" }
lance = "13.0.0"
lance-linalg = "13.0.0"
lance-namespace = "13.0.0"
Expand Down
11 changes: 6 additions & 5 deletions crates/lance-csr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,12 +157,13 @@ clauses, patterns, expressions and functions the engine runs.
- `query` – High level `CypherQuery` API and runtime.
- `csr_index`, `csr_expand` – The CSR adjacency index and the traversal operators that use it.
- `error` – `GraphError` and result helpers.
- `namespace` – Namespace helpers (re-exported from `lance-csr-catalog`).
- `source_catalog` – Catalog helpers for looking up table metadata (re-exported from `lance-csr-catalog`).
- `mapping` – `GraphMapping`, the JSON graph descriptor that maps labels to tables, and
`GraphSnapshot`, which pins each table to a Lance version and opens them for `CypherEngine`.
- `namespace` – `DirNamespace`, a Lance namespace for a directory of datasets.
- `source_catalog` – `GraphSourceCatalog`, which resolves labels to DataFusion table sources.

`lance-csr` re-exports the catalog and namespace types from the `lance-csr-catalog` crate for
API compatibility. You can depend on `lance-csr-catalog` directly if you only need catalog or
namespace utilities.
Up to 0.7.1 the `namespace` and `source_catalog` types lived in a separate `lance-csr-catalog` crate.
They now live here, at the same root paths (`lance_csr::DirNamespace` and so on).

## Error Handling

Expand Down
2 changes: 1 addition & 1 deletion crates/lance-csr/src/apply.rs
Original file line number Diff line number Diff line change
Expand Up @@ -506,7 +506,7 @@ mod tests {
#[test]
fn call_bypasses_snapshot_templates_in_every_reading_stage() {
let (config, _) = graph();
let catalog = Arc::new(lance_csr_catalog::InMemoryCatalog::new());
let catalog = Arc::new(crate::InMemoryCatalog::new());
for text in [
"CALL { RETURN 1 AS n } RETURN n",
"WITH 1 AS x CALL { WITH x RETURN x AS n } RETURN n",
Expand Down
2 changes: 1 addition & 1 deletion crates/lance-csr/src/datafusion_planner/config_helpers.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ use super::analysis::PlanningContext;
use super::DataFusionPlanner;
use crate::config::{NodeMapping, RelationshipMapping};
use crate::error::Result;
use lance_csr_catalog::GraphSourceCatalog;
use crate::GraphSourceCatalog;
use std::sync::Arc;

impl DataFusionPlanner {
Expand Down
2 changes: 1 addition & 1 deletion crates/lance-csr/src/datafusion_planner/join_ops.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ use crate::ast::{PropertyValue, RelationshipDirection};
use crate::case_insensitive::qualify_column;
use crate::config::{NodeMapping, RelationshipMapping};
use crate::error::Result;
use crate::GraphSourceCatalog;
use datafusion::common::Column;
use datafusion::logical_expr::{
col, BinaryExpr, Expr, JoinType, LogicalPlan, LogicalPlanBuilder, Operator, TableSource,
};
use lance_csr_catalog::GraphSourceCatalog;
use std::collections::HashMap;
use std::sync::Arc;

Expand Down
2 changes: 1 addition & 1 deletion crates/lance-csr/src/datafusion_planner/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ pub(crate) use scalar_functions::sql_expr;
use crate::config::GraphConfig;
use crate::error::Result;
use crate::logical_plan::LogicalOperator;
use crate::GraphSourceCatalog;
use datafusion::logical_expr::LogicalPlan;
use lance_csr_catalog::GraphSourceCatalog;
use std::sync::Arc;

/// Planner abstraction for graph-to-physical planning
Expand Down
2 changes: 1 addition & 1 deletion crates/lance-csr/src/datafusion_planner/scan_ops.rs
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ impl DataFusionPlanner {
// No catalog attached - create empty source fallback for flexibility
// This allows planners created with DataFusionPlanner::new() to work
// without requiring a catalog, though they won't have actual data sources
let empty_source = Arc::new(lance_csr_catalog::SimpleTableSource::empty());
let empty_source = Arc::new(crate::SimpleTableSource::empty());
let normalized_label = label.to_lowercase();
let builder =
LogicalPlanBuilder::scan(&normalized_label, empty_source, None).map_err(|e| {
Expand Down
4 changes: 2 additions & 2 deletions crates/lance-csr/src/datafusion_planner/test_fixtures.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@

use crate::config::GraphConfig;
use crate::logical_plan::LogicalOperator;
use crate::{InMemoryCatalog, SimpleTableSource};
use arrow_schema::{DataType, Field, Schema};
use lance_csr_catalog::{InMemoryCatalog, SimpleTableSource};
use std::sync::Arc;

pub fn person_schema() -> Arc<Schema> {
Expand All @@ -17,7 +17,7 @@ pub fn person_schema() -> Arc<Schema> {
]))
}

pub fn make_catalog() -> Arc<dyn lance_csr_catalog::GraphSourceCatalog> {
pub fn make_catalog() -> Arc<dyn crate::GraphSourceCatalog> {
let person_src = Arc::new(SimpleTableSource::new(person_schema()));
let knows_schema = Arc::new(Schema::new(vec![
Field::new("src_person_id", DataType::Int64, false),
Expand Down
5 changes: 3 additions & 2 deletions crates/lance-csr/src/engine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -511,8 +511,9 @@ fn config_error(message: String) -> GraphError {
}
}

/// Open a Lance dataset. `version = None` means the latest version.
async fn open_lance_dataset(
/// Open the Lance dataset at `uri`, pinned to `version`, or at its latest version when
/// `version` is `None`. Storage options go to the object store, as in Lance itself.
pub async fn open_lance_dataset(
uri: &str,
version: Option<u64>,
storage_options: &HashMap<String, String>,
Expand Down
Loading
Loading