spatialdata.write fails on Xenium sdata in env with pandas 3.0.0

[EmielGeeraerts]

I had some headaches trying to write a Xenium dataset as spatialdata object using write.

Error message:

anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.arrays.ArrowStringArray'> into <class 'zarr.core.group.Group'>
Error raised while writing key '_index' of <class 'zarr.core.group.Group'> to /tables/table/obs

Turns out, with pandas 3.0.0 the default string type changed and a new writer is missing.

Resetting to pandas to "old behavior" did fix things for me, but just letting you know.

import pandas as pd
pd.options.mode.string_storage = "python"

import anndata as ad
ad.settings.allow_write_nullable_strings = True

I'm on spatialdata-io==0.5.1 and anndata==0.12.6 at the time of writing

[LucaMarconato]

Thanks for reporting and for sharing the workaround. I see there is a test failing for pandas 3.0.0 here https://github.com/scverse/integration-testing/actions/runs/21613728467/job/62287938878 (tests/io/test_readwrite.py::test_sdata_with_nan_in_obs), it may be related.

Could you please share which dataset you are using? If you could reproduce with a public one, or find a subset of the sdata object that leads to the bug it would be helpful.

[EmielGeeraerts]

I was able to reproduce the issue with the following public dataset, which is also created with Xenium Onboard Analysis 4.0.0
https://www.10xgenomics.com/datasets/xenium-protein-ffpe-human-renal-carcinoma

curl -O https://s3-us-west-2.amazonaws.com/10x.files/samples/xenium/4.0.0/Xenium_V1_Human_Kidney_FFPE_Protein_updated/Xenium_V1_Human_Kidney_FFPE_Protein_updated_outs.zip
curl -O https://cf.10xgenomics.com/samples/xenium/4.0.0/Xenium_V1_Human_Kidney_FFPE_Protein_updated/Xenium_V1_Human_Kidney_FFPE_Protein_updated_xe_outs.zip

Below is the full error trace when I run the following:

import spatialdata_io as sio
sdata = sio.xenium(xenium_dir, cells_as_circles=True)
sdata.write(out_path, overwrite=True)

error_spatialdata-io_write.txt

pyproject.toml dependencies

"geosketch==1.3",
"harmonypy==0.2.0",
"igraph==0.11.8",
"pip>=23.2",
"popv==0.5.2",
"scanpy==1.12.0",
"scikit-misc==0.5.1",
"setuptools>=65.5",
"spatialdata[extra]==0.7.1",
"squidpy==1.8.0",
"wheel>=0.46.3"

Hope this helps!

[AlbertoFabbri93]

I don't think this error is raised only for Xenium, I am encountering the same error when saving CosMx data.

[Tomatokeftes]

Possibly useful from a different direction -- we hit the same error writing
non-Xenium data, so this may or may not match your case exactly.

Unless I am misreading the history, this looks like it is already fixed in
anndata 0.13.0: anndata
#2133 ("Support pandas 3.0
upcoming changes") shipped there and closed
#2221. The milestone on #2221
still reads 0.14.0, which is what had me convinced for a while that it was
still open and a long way off, so I suspect I am not the only one who has read
it that way.

I checked 0.13.0 itself rather than just the latest patch, but only in my own
environment (pandas 3.0.5, pyarrow 25.0.0, zarr 3.3.0, Windows), so please
treat the numbers below as a data point rather than a general claim.

If upgrading anndata is not an option, a settings-only workaround worked for me
on pandas 3 -- though it needed both of these, and neither did anything alone:

pd.set_option("mode.string_storage", "python")
ad.settings.allow_write_nullable_strings = True

As far as I can tell that is because anndata 0.12.x has a StringArray writer
gated behind that setting and no ArrowStringArray writer at all, so you have
to both leave Arrow-backed storage and unlock nullable-string writing. What I
measured, writing an obs frame through ad.io.write_elem on anndata 0.12.2:

mode.string_storage	allow_write_nullable_strings	result
default (arrow)	False	IORegistryError on ArrowStringArray
default (arrow)	True	IORegistryError on ArrowStringArray
python	False	RuntimeError on allow_write_nullable_strings
python	True	wrote fine

Values, index and categorical dtype all came back intact for me on that last
row. Two things I would flag rather than recommend it outright: it is global
process state, which seems fine in an application but probably not something a
library should set on its users' behalf; and it did not help under
future.infer_string on pandas 2, where the arrays are the *NumpySemantics
subclasses instead.

One small thing that cost me time, in case it helps: a string index, a string
column, and a string-backed categorical each fail independently, so a partial
fix can look like progress and still break. The categorical one is the easiest
to miss and is always in play here, since spatialdata requires region to be
categorical.

Happy to be corrected on any of this if it does not match what you are seeing.