feat: rust examples for creating and querying Paimon tables - #648
feat: rust examples for creating and querying Paimon tables#648ganesh-skumar wants to merge 7 commits into
Conversation
|
Hi, While checking the CI failure, the rust examples require adding I was wondering whether it would be more appropriate to place these examples in a separate example crate to avoid introducing these additional dependencies into the core paimon crate. Or is updating the generated DEPENDENCIES.rust.tsv files the expected approach in this case? Thanks! |
|
|
||
| // Create new database | ||
| catalog | ||
| .create_database("my_db", false, HashMap::new()) |
There was a problem hiding this comment.
Could we make this setup example safely rerunnable? With ignore_if_exists = false, a second invocation would fail. Perhaps the example can drop/recreate its sample objects everytime the example is run for a clean start.
There was a problem hiding this comment.
got it, updated to ignore db if exists and creates fresh table.
| datafusion = "54.0.0" | ||
| datafusion-ffi = "54.0.0" | ||
| paimon = { version = "0.4.0", path = "crates/paimon" } | ||
| paimon-datafusion = { path = "crates/integrations/datafusion" } |
There was a problem hiding this comment.
Would it be better to place datafusion_query.rs under the paimon-datafusion crate?
There was a problem hiding this comment.
good idea, created integrations/datafusion/examples/datafusion_query.rs
|
|
||
| pub async fn create_catelog() -> Result<Arc<dyn Catalog>, Box<dyn Error>> { | ||
| let mut options = Options::new(); | ||
| options.set(CatalogOptions::WAREHOUSE, "/path-to/testdata"); |
There was a problem hiding this comment.
Could the path be provided as a cli argument, sharable by both examples. Since the user's would need to edit this code to wire up their real warehouse file otherwise.
There was a problem hiding this comment.
updated to pass path via cli
56754f4 to
f72b493
Compare
leaves12138
left a comment
leaves12138
left a comment
There was a problem hiding this comment.
I ran both examples end to end, including rerunning the setup and querying the resulting table. The functional flow works, but there is one destructive default behavior that should be addressed.
|
|
||
| // if exists, drop and create fresh table and data | ||
| if table_exists { | ||
| catalog.drop_table(&identifier, false).await?; |
There was a problem hiding this comment.
Could we avoid dropping an existing table by default? The warehouse is supplied as an arbitrary CLI path, so a user who points this example at an existing warehouse containing my_db.users would permanently delete that table and all of its data. Please fail when the table already exists, require an explicit destructive flag such as --overwrite, or use a clearly isolated example namespace/warehouse.
There was a problem hiding this comment.
Thanks, added table recreation based on flag.
| Ok(()) | ||
| } | ||
|
|
||
| pub async fn create_catelog(warehouse: String) -> Result<Arc<dyn Catalog>, Box<dyn Error>> { |
There was a problem hiding this comment.
Nit: please rename create_catelog to create_catalog. The same typo also appears in datafusion_query.rs.
| ) | ||
| })?; | ||
|
|
||
| let overwrite = args.any(|arg| arg == "--overwrite"); |
There was a problem hiding this comment.
Could we make --overwrite order-independent, or document the required order? The first argument is always treated as the warehouse, so --overwrite <warehouse-path> fails; only <warehouse-path> --overwrite works. At minimum, please update the usage string to <warehouse-path> [--overwrite].
There was a problem hiding this comment.
Thanks! Updated the usage doc with the correct argument order. I also tried making it order-independent, but it made the code a bit messy for a beginner-friendly create_table example.
ArnavBalyan
left a comment
ArnavBalyan
left a comment
There was a problem hiding this comment.
Thanks for the review @QuakeWang and thanks for the PR @ganesh-skumar!
4 participants
Purpose
Add Rust examples demonstrating how to create a Paimon table with sample data and query it using DataFusion.
Brief change log
create_table.rsexample to create a Paimon table and insert sample data.datafusion_query.rsexample demonstrating the DataFusion DataFrame API.paimon-datafusionas a workspace dependency for the examples.