Repository navigation
Drop the Arrow dependency, document that this is Feather V1 - #25
Merged
Merged
Conversation
FeatherFiles reached into the old Arrow.jl for ArrowVector, DictEncoding, unsafe_isnull and unsafe_getvalue. That package is gone — its registry slot was taken over at 0.3.0 by the unrelated Apache Arrow.jl — so `Arrow = "0.2.1 - 0.4"` resolved to a package with none of those names and precompilation failed. FeatherLib now vendors that code and exports what is needed, so point the four references in missing-conversion.jl at FeatherLib.ArrowCompat and drop the dependency entirely. No logic change. Removing the old Arrow dependency also lifts the CategoricalArrays <= 0.8 cap it imposed on every FeatherFiles and Queryverse user, which is the substance of #23. The README now says plainly that this package handles Feather V1 only. V1 and V2 are different formats — V2 is exactly the Arrow IPC file format, is what every current tool writes whether the file is named .feather or .arrow, and is deprecated-for-removal upstream as of Arrow 25.0.0. Readers hitting a modern .feather file need Arrow.jl, not this package. julia compat moves to 1.10 so the CI matrix only covers versions that actually run the testitem tooling. Note for CI: this needs FeatherLib 1.0, which is not registered yet, so the build here stays red until that release is tagged. Verified locally against the FeatherLib branch — all 25 tests pass on Julia 1.12 with CategoricalArrays 1.1.1 and no Arrow in the manifest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Feather stores dates and times as the Arrow wire types Datestamp, Timestamp
and TimeOfDay. Those leaked all the way to users: DataFrame(load("x.feather"))
gave a column of FeatherLib.ArrowCompat.Datestamp rather than Date. This is
Feather.jl#99, still open upstream. Unwrap them here, in the layer users
actually see, rather than in FeatherLib -- FeatherLib is deliberately a
faithful low-level mirror of the file layout.
Doing so uncovered a second bug. getiterator decided whether a column was
nullable with `col_eltype.b <: Missing`, but Julia does not order the members
of a Union predictably: for Union{Datestamp,Missing} the non-Missing half is
`.a`, so that test silently skipped the column and it came back neither
DataValue-wrapped nor converted. Use Base.nonmissingtype instead, which is
correct whatever the ordering.
get_columns_copy_using_missing copies every column out, so it now close!s its
ResultSet before returning and the file is immediately deletable -- no GC
required, which is what matters on Windows. getiterator cannot do the same:
its rows read straight through the memory mapping, so the mapping has to
outlive the iterator.
That distinction is why the long-commented-out rm() in the first testitem
stays commented out. It sits after a getiterator call, and it is that
mapping, not the one from get_columns_copy_using_missing, that holds the
file. The comment there now says so. The deterministic half has its own
testitem, which does rm() with no GC at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Displaying a feather file left it memory mapped, so on Windows merely
running `display(load("x.feather"))` in the REPL kept the file locked
against deletion or overwriting until the GC happened to run.
Unlike getiterator, which hands the iterator to the caller and therefore
cannot close anything, the show methods consume the rows they need and
return. All three TableShowUtils printers are eager -- printtable and
printHTMLtable collect or count a bounded prefix, printdataresource
iterates and writes -- so the ResultSet can be closed as soon as the
printer returns.
Factor the read-and-convert step out of getiterator into read_converted,
and add a withtable helper that opens a ResultSet, runs the printer, and
close!s in a finally. getiterator keeps its existing behaviour; its comment
now points callers who want the file released at
get_columns_copy_using_missing, which copies and then closes.
Verified with no GC.gc() anywhere: after each of the three show methods the
file is immediately deletable, and the rendered output is unchanged (the
existing testitem asserts the exact sprint(show, ...) string).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The "Date and time columns" testitem checked that a non-temporal column survives untouched with `typeof(rows[1].n) == Int64`, but the column is written as [1, 2], whose eltype is Int -- and Int is Int32 on the x86 CI legs. All five failures in the last run were this one assertion, on ubuntu-x86 and windows-x86; every x64 leg passed. Verified on 32-bit Julia locally (julia +release~x86, Sys.WORD_SIZE=32, Int=Int32) rather than only on x64: 45/45 tests pass there, and the x64 run still passes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to queryverse/FeatherLib.jl#24. Together they make FeatherFiles installable again.
Why it was broken
FeatherFiles reached into the old Arrow.jl for
ArrowVector,DictEncoding,unsafe_isnullandunsafe_getvalue. That package is gone — its registry slot (UUID69666777-…) was taken over at version 0.3.0 by the unrelated Apache Arrow.jl — soArrow = "0.2.1 - 0.4"resolved to a package with none of those names, and precompilation failed.What this does
FeatherLib now vendors that code and exports what is needed, so the four references in
missing-conversion.jlpoint atFeatherLib.ArrowCompatand the dependency goes away. No logic change.Dropping the old Arrow dependency also lifts the
CategoricalArrays ≤ 0.8cap it imposed on every FeatherFiles and Queryverse user — the substance of #23. (Leaving that issue open for you to close.)The README now says plainly that this package handles Feather V1 only. V1 and V2 are different formats: V2 is exactly the Arrow IPC file format, is what every current tool writes whether the file is named
.featheror.arrow, and Apache deprecated V1 reading and writing at Arrow 25.0.0 with removal planned. Anyone hitting a modern.featherfile needs Arrow.jl, not this package.juliacompat moves to 1.10 so the CI matrix only covers versions that actually run the testitem tooling — a deliberate deviation from the ecosystem-wide "restore pre-migration compat" decision, for this package.Also:
showno longer locks the fileDisplaying a feather file left it memory mapped, so on Windows
display(load("x.feather"))in the REPL kept it locked against deletion until the GC happened to run. The threeshowmethods consume the rows they need and return — all three TableShowUtils printers are eager — so they nowclose!theResultSetas soon as the printer is done.The read-and-convert step is factored out of
getiteratorintoread_converted, with awithtablehelper that opens, prints and closes.getiteratoris unchanged: it hands the iterator to the caller, so it cannot close anything, and its comment now points callers who want the file released atget_columns_copy_using_missing.Measured, with no
GC.gc()anywhere:get_columns_copy_using_missing— whatDataFrame(load(f))usesshow/ HTML / dataresourcegetiteratorthencollectVerification
No longer blocked — FeatherLib v1.0.0 is released and in the General registry, so this resolves.
Verified against the registered FeatherLib 1.0.0 rather than a
deved path: a clean environment picks up FeatherLib v1.0.0 with CategoricalArrays 1.1.1, FlatBuffers 0.6.2 and no Arrow in the manifest. All 6 testitems pass on Julia 1.12, and the renderedshowoutput is unchanged (the existing testitem asserts the exact string).Also verified across both packages that format routing works purely by magic bytes, with no FileIO registry change: a V1 file named
.featherresolves toformat"Feather"and loads through FeatherFiles, while a V2 file named.featherresolves toformat"Arrow".🤖 Generated with Claude Code