Update PyO3 to 0.29 and explore abi3t support

[ngoldbaum]

I'm working on free-threaded support in polars with @kumaraditya303: pola-rs/polars#27955. Datafusion shows up in the polars dependency tree.

Updating to PyO3 0.29 enables builds with the new Python 3.15 free-threaded stable ABI, which should allow datafusion to enable free-threaded support without an ongoing need to upload version-specific wheels every time Python does a release.

[timsaucer]

Thanks for the issue! We have to update pyo3 in lock step with arrow-rs. I just merged a PR to updated it to 0.29, so it will be the next major arrow release (60). Then we need datafusion upstream to update, which it generally does very quickly. After that we'll follow suit.

Can you point out where datafusion shows up in the dependency tree for polars? That's new to me.

FWIW our next release of datafusion-python does have free threaded support with 3.13 (experimental) and 3.14 (stable) python.

[kumaraditya303]

Can you point out where datafusion shows up in the dependency tree for polars? That's new to me.

polars has dependency on deltalake which in turn depends on datafusion. It is heavily used by polars for io.

[ngoldbaum]

OK great. Just an FYI though that PyO3 0.29 dropped 3.13t support. See the release notes and the free-threading guide for more detail about that. I'd suggest simply never shipping 3.13t wheels. 3.13t never gained support for a few key packages and that makes it difficult to use 3.13t with a nontrivial set of dependencies. 3.14t is supported much more widely so we're trying to get people to start support there.

[timsaucer]

Sounds good! I will exclude them when I do the upload! Thanks for the feedback!

[timsaucer]

DataFusion-54.0.0 is released with free threaded Python wheels available. Do you want to keep this issue open? We will auto-upgrade pyo3 as our dependencies update.

[ngoldbaum]

Let's close it if it's not useful for you.