ReadStat 2.0: full C-API coverage, rich metadata, value labels, dates, parallel and streaming reads, write support - #98
Open
davidanthoff wants to merge 14 commits into
Open
davidanthoff wants to merge 14 commits into
davidanthoff wants to merge 14 commits into
Conversation
Replace src/C_interface.jl with a ReadStat.CAPI submodule that wraps the readstat 1.1.9 parser-side C API one-to-one: proper Julia enums for every C enum (including all 42 error codes), opaque typed pointers plus C getters for every struct except readstat_value_t, and @ccall-based wrappers covering the metadata, variable, value, and missing-range accessors, handler and custom-IO setters, encodings, and row limit/offset. readstat_value_t remains the one by-value struct mirror. Its layout is now verified behaviorally against the C library by a new ABI test item that round-trips missing ranges through by-value returns; that test showed MinGW GCC uses MS-compatible bitfield layout, so the struct is 24 bytes on Windows (bitfields in a fresh 4-byte unit at offset 16) and 16 bytes elsewhere. The parser is now freed via try/finally so a throwing handler cannot leak it. Version 2.0.0-DEV; minimum Julia raised to 1.12. Reader behavior unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reading now produces a ReadStatTable: concretely typed DataValueVector columns reached by index or name, a ReadStatMeta with the full file-level metadata the C library exposes (creation and modified time, format version, 64-bit flag, compression, endianness, table name, file label, encoding, notes, fweight, and all value-label sets), and a ReadStatVarMeta per column (label, format, raw type and type class, value-label set name, widths, measure, alignment, and SPSS missing-value ranges). Value-label sets are parsed into typed dictionaries keyed by Int32/Float64/String, with labeled tagged missing values keyed by their tag Char. Parsing goes through typed column buffers (raw Vector plus Bool NA mask, grouped by element type and dispatched via a small typecode branch), so the value path is type-stable without touching any DataValues internals; the buffers are wrapped into DataValueVectors only at the end. The NA mask starts all-missing over defined backing storage, so cells the parser never delivers surface as NA instead of undefined memory. Handler @cfunction pointers are created once in __init__ and shared; every handler captures Julia exceptions and aborts the parse instead of unwinding through C. ReadStatDataFrame remains as a deprecated alias whose 1.x field access is reconstructed through getproperty shims with deprecation warnings, so existing consumers such as StatFiles.jl keep working until their follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every reader (and the new extension-sniffing `readstat` dispatcher) now accepts `usecols` (Symbol/index/vectors/Regex/predicate — skipped columns are skipped inside the C library via READSTAT_HANDLER_SKIP_VARIABLE, and the value path indexes by index_after_skipping), `row_limit`/`row_offset` (wired to the C setters; buffers are preallocated for exactly the selected range, and a zero limit is handled Julia-side because the C library treats 0 as unlimited), `file_encoding`/`handler_encoding` (iconv names), and `progress` (a callback that can stop the parse and keep the rows read so far, trimmed to complete rows). The note, fweight, and error handlers are now always registered: notes and the frequency-weight variable land in ReadStatMeta, and messages the C library reports through its error handler surface as Julia warnings after the parse. `read_meta` reads only metadata: the returned table has zero rows but full file- and variable-level metadata, value labels, and notes, with the true row count (the C library caps the reported count at any row limit in effect, so read_meta sets none). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tagged missing values (Stata/SAS .a-.z) still read as NA, but their tags are now recorded per column and exposed through the new missingtags accessor; label sets that label a tagged missing value keep the tag as a Char key. SPSS user-defined missing values collapse to NA by default as before, and the new user_missing=:keep keyword keeps the codes as data instead, with the declared rules always available in varmetadata(tbl, col).missing_ranges. Test fixtures are sourced from ReadStatTables.jl (MIT) and pyreadstat (Apache-2.0), verified against the upstream blob SHAs and attributed in test/data/README.md; pyreadstat ships the expected values for both user-missing modes as CSVs, which the new tests assert against. read_por is now tested for the first time (#96) using the known-good sample.por; StatFiles.jl's types.por fixture turns out to carry a timestamp readstat 1.1.9 rejects, which the tests pin down as a clean error (context for queryverse/StatFiles.jl#32). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Value-labeled columns can now be viewed through a LabeledArray: a zero-copy wrapper around the raw code column plus the shared label dictionary, whose elements (DataValue-wrapped LabeledValues, so NA propagates in the usual DataValues way) display as their labels but compute as their codes — ==, isless, and hash act on the code, while comparison against a string compares the label. Unlabeled codes display as string(code), so partial label sets and labels on continuous variables lose nothing. Obtain the view per column with labeled(tbl, col) or table-wide with the new apply_value_labels=true keyword; rawvalues/getvaluelabels/unwrap/valuelabel take it apart again. This addresses the long-standing value-label requests (#74, #47). SAS value labels are now reachable too: read_sas7bcat parses a .sas7bcat catalog into label sets keyed by format name, and read_sas7bdat gained a catalog keyword that merges those labels into the table, where they line up with the vallabel names the variables already carry. Fixtures for the catalog pair and a string-keyed label set are sourced from pyreadstat and ReadStatTables.jl like the rest of test/data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Columns whose display format is a date/time format now come back as
DataValueVector{Date}, {DateTime}, or {HMS} columns; convert_datetime=false
keeps the raw numbers. The format tables cover Stata (%tc/%tC milliseconds,
%td/%d days, %tw/%tm/%tq/%th counts with Stata week semantics, %ty calendar
years; epoch 1960-01-01), SAS (DATETIME*/E8601* seconds, DATE/MMDDYY/
DDMMYY*/YYMMDD*/... days, TIME/HHMM/HOUR/E8601TM times; epoch 1960-01-01),
and SPSS (DATETIME/YMDHMS, DATE/ADATE/EDATE/JDATE/SDATE, TIME/DTIME;
seconds since 1582-10-14), with width and decimal suffixes normalized away.
Value-labeled columns are never converted — they hold codes.
The new HMS type carries times of day as raw seconds with unbounded hours,
since SAS and SPSS time values can exceed 24 hours or be negative, which
Dates.Time cannot represent; it displays as H:MM:SS[.fff] and converts to
Time when within a calendar day.
Tested against the pyreadstat-derived sample fixtures in all five formats
plus DATETIME13 and Stata %tc/%td/%tw/%tm/%tq/%th/%ty edge cases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every reader, the readstat dispatcher (with an explicit format), and read_meta now accept an IO containing the complete file in addition to a path. Julia implementations of the C library open/close/seek/read handler quartet drive the parse over the stream; non-seekable streams are buffered into an IOBuffer first, since the parsers seek backwards routinely. Exceptions raised by the stream are captured on the IO context and rethrown after the parse instead of unwinding through C. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The new ntasks keyword (auto-enabled for large files) splits the selected row range into contiguous chunks, one C parser per Threads.@Spawn task. All tasks write directly into shared, preallocated column buffers: chunk obs_index values are relative to each parser''s row_offset (verified by the row-selection tests), so every task fills a disjoint region of plain Vectors, which is thread-safe without locks — no per-task buffers, no merge copy, and no chained columns. Only the rare tagged-missing tag vectors are per-task (lazy shared allocation would race) and get combined after the join, and chunk variable handlers skip column setup entirely via a preassigned-columns mode. A metadata pre-pass without a row limit supplies the true row count, the schema for preallocation, and the value labels and notes: experimentation showed the sav parser does not reliably deliver value labels while a row limit is in effect, so chunk parses cannot be trusted for metadata. The same finding fixes row_limit=0 reads, which previously used a 1-row C limit and could lose sav value labels. Formats without a recorded row count (xport, por) and IO input fall back to a serial parse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…aming ReadStatSource is the interface a display layer or query engine plans against: constructing one parses nothing; schema() caches a metadata-only table; colnames/coltypes/nrows answer planning questions (coltypes reflects date/time conversion and value-label application per the source options, and nrows is missing when the format records no count); supports() reports which pushdowns the format can honor (:projection, :row_range, :row_count, :parallel). read(src; usecols=..., rows=...) pushes a column projection and a 1-based row range into the C parse, and works repeatedly off one stream for IO-backed sources. chunks(src; chunksize, usecols, rows) streams the file as complete ReadStatTables in a single pass: the parse runs on its own task and hands finished chunks through a bounded channel (backpressure blocks the parser), deliberately not one re-parse per chunk, which would be quadratic on compressed formats — so it also works for formats with unknown row counts. A metadata pre-pass supplies every chunk with the file's complete metadata, so value labels and date/time conversion apply from the first chunk on (dta stores label sets after the data). Closing the iterator early releases the parse task; already-buffered chunks still drain. This is the enabling interface for the planned StatFiles.jl follow-up (show via read(src; rows=...), iteration via chunks — StatFiles #4, #15) and the future query-execution engine (ReadStat.jl #29, #39). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ions ReadStat.jl can now write every format the C library writes. The CAPI submodule gains the complete writer surface (label sets including tagged values, variables with all attributes and SPSS missing declarations, notes, string refs, file-level metadata, compression, validation, and the row API). On top sits a low-level Writer following the C call order one-to-one and streaming output bytes into any Julia IO through the data-writer callback — public but unexported, for full control including Stata strL columns via string refs. The exported write_dta/write_sav/write_por/write_sas7bdat/write_xport take columns plus names (or a ReadStatTable, carrying its metadata across) and canonicalize onto the storage types each format supports: Bool and wide integers narrow sensibly, Date/DateTime/HMS/Time re-encode into the format''s native representation with a matching display format, LabeledArray columns bring their value labels along, and NA/tagged-missing cells write as the right kind of missing. write_sas7bcat writes value-label catalogs (numeric sets only — a readstat 1.1.9 limitation, documented). Two C quirks handled: the por writer silently writes no rows under lowercase variable names (names are uppercased for por, matching what a por read returns) and only handles doubles (numerics widen there). Round trips are tested for all five data formats plus the catalog, covering labels, tags, notes, user-missing rules, compression (zsav and row compression), IO destinations, and a 50k-row file read back with ntasks=4 and via chunks. This supersedes the stale write-support PRs #42 and #46. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
read_txt reads a fixed-width text data file described by a schema file — a Stata dictionary, SAS commands with an INPUT statement, or SPSS DATA LIST syntax — through the C library''s schema parsers and readstat_parse_txt, with the usual usecols/row_limit/row_offset/handler_encoding/ convert_datetime keywords (date/time formats interpret per the schema''s producer). The text parser ignores readstat_set_row_offset, so the offset is emulated by shifting the handler''s row base. The CAPI submodule gains the five schema functions; a Stata dictionary fixture exercises the path end to end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The README walks through reading (kwargs, labels, date/time, missing semantics), lazy sources with streaming chunks, writing, and 1.x migration; NEWS gets the full 2.0.0 release notes; and the documentation gains an introduction plus an auto-generated API reference over the now fully docstringed public surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #98 +/- ##
==========================================
+ Coverage 83.33% 88.39% +5.06%
==========================================
Files 2 21 +19
Lines 156 1370 +1214
==========================================
+ Hits 130 1211 +1081
- Misses 26 159 +133
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The 2.0 rewrite turned out to use only one post-1.10 language feature: the `public` keyword. It is now declared via eval on Julia 1.11+ only, and the compat floor returns to 1.10 (matching the rest of the Queryverse); the full test suite passes on 1.10.12. On 32-bit unix, glibc''s off_t and time_t are 32-bit (the jll build uses no large-file/time64 opt-ins), but the wrappers hardcoded Int64: the seek callback''s argument stack shifted, handing it a garbage io context — the linux-x86 CI segfaults in the IO, source/chunks, and writer round-trip tests. readstat_off_t and time_t now size per platform (64-bit on Windows, which uses _off64_t and 64-bit time_t throughout, hence the green windows-x86 leg). Also adds the docstrings whose absence broke the documentation build''s cross-references (ValueLabelDict and the low-level writer''s row and file-metadata functions); the docs now build cleanly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The windows-x86 legs on Julia 1.10/1.11 showed timestamps with garbage in the high half: this MinGW build has 32-bit time_t on i686 (the 1.12-x86 passes were luck in what the second register held). readstat_off_t stays 64-bit on all of Windows (readstat.h typedefs it to _off64_t there, and the IO tests pass on every x86 leg), so only Ctime_t changes to follow the word size on every platform. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A ground-up modernization of ReadStat.jl on the readstat C library v1.1.9, staying fully in the Queryverse world (DataValues.jl columns, no Tables.jl dependency). Minimum Julia stays 1.10 (the only post-1.10 feature used is the
publickeyword, declared conditionally); version bumps to 2.0.0-DEV.Reading
ReadStatTable: concretely typedDataValueVectorcolumns by name/index plus complete metadata — file level (filemetadata: counts, creation/modified time, format version, 64-bit flag, compression, endianness, table name, file label, encoding, notes, fweight, all value-label sets) and per variable (varmetadata: label, display format, raw type/class, value-label set, widths, measure, alignment, SPSS missing rules).ReadStatDataFramestays as a deprecated alias with 1.xgetpropertyshims, so StatFiles.jl keeps working unchanged.usecols(projection inside the C parse viaREADSTAT_HANDLER_SKIP_VARIABLE),row_limit/row_offset(addresses Add support for row_offset and row_limit from ReadStat #67, Read partial sas data #39),ntasks,file_encoding/handler_encoding,user_missing,convert_datetime,apply_value_labels,catalog,progress(cancellable, returns the rows read so far).readstat(extension dispatch),read_meta(metadata-only, true row count),read_sas7bcat,read_txt(fixed-width text via SAS/SPSS/Stata schema files), and every reader accepts anIO(custom C IO handlers; non-seekable streams are buffered).Date/DateTime/HMSby default from the Stata/SAS/SPSS format tables;HMSrepresents times beyond 24 hours..dtawith value labels #74, How to extract data and labels of column from .SAV file? #47): always parsed into typed dictionaries; the lazyLabeledArray/LabeledValueview (labeled(tbl, col)orapply_value_labels=true) displays labels but computes on raw codes — a partial dictionary encoding, deliberately not a categorical type. SAS labels come from.sas7bcatcatalogs via thecatalogkeyword.missingtags), including labeled tags; SPSS user-defined missing values collapse to NA by default or stay data withuser_missing=:keep, with the rules always in metadata (addresses User missing values vs. system missing values #82).ntasksparallel reads split the row range across one C parser per task, all writing disjoint regions of shared preallocated buffers — no per-task copies, no chained columns; a metadata pre-pass supplies the schema and labels (the sav parser drops value labels under a row limit, found empirically).Pushdown + streaming (prep for the query engine; addresses #29, #39)
ReadStatSourceis a lazy handle:schema/colnames/coltypes/nrows/supportsfor planning,read(src; usecols, rows)for projected range reads, andchunks(src; chunksize)for single-pass streaming through a bounded channel (works for unknown-row-count formats; O(n), not per-chunk re-parses). This is the interface the StatFiles.jl follow-up will build on (StatFiles #4, #15) and the future query execution engine can push into.Writing (supersedes #42 and #46; enables StatFiles #1)
write_dta/write_sav/write_por/write_sas7bdat/write_xport/write_sas7bcatfrom columns+names or aReadStatTable(metadata carried across): value labels (including labeled tagged missings), tags, SPSS missing declarations, notes, file metadata, compression (:rows,:binary→ zsav), and date/time re-encoding per target format.ReadStat.Writermirrors the C writer API one-to-one and writes to anyIO, including Stata strL via string refs.Correctness fixes
readstat_value_t) is verified behaviorally against the library in CI — which immediately caught that MinGW GCC uses MS bitfield layout, so the struct is 24 bytes on Windows, not 16.read_poris tested for the first time (read_por is untested and has no fixture #96) against a known-good fixture; StatFiles'types.porturns out to carry a timestamp readstat 1.1.9 rejects (kept as an error-path test; context for StatFiles WIP Use BinaryProvider #32).Tests
14 test items, all green on Julia 1.12 with 4 threads: ABI verification, per-format reads with full metadata assertions, all reader kwargs, missing-value semantics against pyreadstat's published expected values, value labels (numeric/string/tagged/catalog), date/time across all five formats, IO input, multi-task ≡ serial equivalence, source/chunks, write→read round trips for all six formats (plus a 50k-row file re-read with
ntasks=4and viachunks), and the schema text path. Fixtures are sourced from ReadStatTables.jl (MIT) and pyreadstat (Apache-2.0), verified against upstream blob SHAs and attributed intest/data/README.md.A StatFiles.jl follow-up PR will route
.xpt/.por, adoptReadStatSource/chunks, expose metadata, and addsaveon top of the new writers (StatFiles #1, #4, #12, #15, #19, #32).🤖 Generated with Claude Code