RFC: Re-work some DataFrame APIs

[ion-elgreco]

Some API's feel a bit un-intuitive, I think Polars has really excelled at this area. My suggestion is we re-use some of those APIs or take some inspiration of them, changes I am proposing (I am happy to work on these areas especially with datafusion-ray becoming a thing):

• - DataFrame.cache() -> DataFrame ===> DataFrame.collect() -> DataFrame
• - DataFrame.collect() -> list[pyarrow.RecordBatch] ===> DataFrame.to_batches() -> list[pyarrow.RecordBatch]
• - DataFrame.join ===> DataFrame.join(right: DataFrame, on: str | sequence[str] | None, left_on: str | sequence[str] | None, right_on: str | sequence[str] | None
• - DataFrame.schema -> pyarrow.Schema ===> DataFrame.schema -> datafusion.Schema Map Rust arrow types to dafusion-py types
• - DataFrame.with_column ===> DataFrame.with_columns Allow multiple inputs as exprs or key value pairs
• - DataFrame.with_column_renamed ===> DataFrame.rename() a simple rename is clear enough and should allow a dict as input
• - DataFrame.aggregate ===> DataFrame.group_by().agg() this feels more natural coming from PySpark/Polars/Pandas

Can remove these:

• - DataFrame.select_columns already covered by DataFrame.select

Missing APIs:

• - DataFrame.cast to cast on top level a single or multiple columns
• - DataFrame.drop to drop columns, instead of writing a very verbose select
• - DataFrame.fill_null/fill_nan to fill null or nan values
• - DataFrame.interpolate interpolate values per col
• - Asof join missing in df api?
• - Join on (inequality join)
• - DataFrame.head/tail
• - DataFrame.pivot
• - DataFrame.unpivot

Optional but useful:

• - DataFrame.with_row_idx

[emgeee]

These proposals generally sound good to me. I do think care should be taken around the first two points since Dataframe cache() and collect() methods shadow the underlying rust library and renaming those methods at the python level would be immensely confusing for those coming from the rust library or those seeking to better understand the python layer.

The other suggestion I might add is to keep Datafusion.with_column() but make it a simple wrapper around Datafusion.with_columns().

[ion-elgreco]

Asof joins are pending: apache/datafusion#318

[HairstonE]

Hi @ion-elgreco! I'd like to work on implementing the remaining incomplete items from this RFC. Since it's been over a year since the last update, is this issue still a priority?

@timsaucer @andygrove - would appreciate your input on whether these DataFrame API improvements are still worth pursuing given the current project direction.

I'm ready to tackle these systematically:

• head/tail, drop, to_batches
• fill_null/fill_nan, interpolate
• asof join
• collect/schema changes

Happy to start with incremental PRs if this aligns with current priorities. Thanks!

[ion-elgreco]

Hi @ion-elgreco! I'd like to work on implementing the remaining incomplete items from this RFC. Since it's been over a year since the last update, is this issue still a priority?

@timsaucer @andygrove - would appreciate your input on whether these DataFrame API improvements are still worth pursuing given the current project direction.

I'm ready to tackle these systematically:

• head/tail, drop, to_batches
• fill_null/fill_nan, interpolate
• asof join
• collect/schema changes

Happy to start with incremental PRs if this aligns with current priorities. Thanks!

Just my 2 cents but I believe these things are still valuable to add, however I have no decision power here :p