Skip to content

Atera: aligned H&E and IF images, xenium table layout, writer support for table_controls, review fixes - #445

Open
timtreis wants to merge 29 commits into
scverse:atera-readerfrom
timtreis:atera-on-444
Open

timtreis wants to merge 29 commits into
scverse:atera-readerfrom
timtreis:atera-on-444

Conversation

@timtreis

@timtreis timtreis commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

@LucaMarconato, thanks for taking on the Atera review in #444. This PR puts our fixes on top of your branch and keeps your zarr < 3.3 helpers, colors, manifest extras and missing-bundle error.

Changes:

  • atera() reads the post-run H&E and IF images, placed by their alignment CSV. An H&E image without an alignment CSV, as in the WTA preview bundles, is placed by an affine fit to its keypoints.
  • The table follows the xenium() layout, with genes in table and the control and unassigned features in a second table, table_controls. The writer reads both tables. If you'd rather keep one table, we can drop the layout change.
  • The obs index is unnamed, which covers xinaesthete's sdata.write() point on feat: Updates for 10x Genomics Atera bundles #426. Without it, bounding_box_query fails with "cannot insert barcode, already exists".
  • read_transcripts_for_genes and read_cell_boundaries take positions from the bundle instead of a possibly filtered table, and empty selections return empty elements.
  • Images are read by TIFF tile, so the dask graph pickles and runs under dask.distributed. The pyramid is built with scale_factors instead of read from the stored levels, which bounding_box_query cropped wrongly.

The planar RGB and keypoint changes to xenium.py are also open against main as #446 and #447, and they drop out of this diff once main is merged into atera-reader.

Tests: the three Atera test files give 66 passed, against 31 on ec169cf, and the Xenium tests this branch shares with #446 and #447 pass in tests/test_xenium.py. The 10x tiny sample round-trips through the writer unchanged, and mypy passes on the changed source files.

To do: the full test suite and a rerun on the WTA kidney bundle.

Tim Treis added 29 commits November 20, 2026 17:39
…affine against keypoints.csv

Atera bundles ship he_image.ome.tif with he_imagealignment.csv (3 x 3 image px -> morphology px, the Xenium
convention) and keypoints.csv. The reader reuses the Xenium aligned-image parser (aligned_images=True), and warns
when the affine maps the keypoints with a median residual above 10 px.
On the Atera preview bundles a similarity fit to 10x's keypoints leaves 21.8 and 22.7 px residuals (about 0.2%
anisotropy in the scans); a least-squares affine brings every keypoint under 7.2 px. On the v1 bundles, where 10x's
alignment CSV is the similarity fit, the affine stays within about 3 px of it.

Folds f712d90 (the similarity-fit fallback this commit replaces), which is not ported on its own.
dask_image reads a planar RGB OME-TIFF as (1, 3, y, x), which the 4-D branch of xenium_aligned_image rejected
(it expected (1, y, x, 3)). The Atera v1 Onboard 1.1.1 H&E images are planar (breast, CRC, brain); the 1.0.4
kidney is interleaved.
…de column

spatialdata's table joins call obs.reset_index() (spatialdata 0.8.0
_core/query/relational_query.py:259), which fails with "cannot insert barcode,
already exists" when the index carries the name of an existing column. The column
keeps the values.
… transcripts store shared by all tasks

Every reader opened its zip in a try/finally; _open_zip replaces 11 of them. read_transcripts_for_genes
copied the tile decode of _read_transcript_tile; both now call _transcript_rows. _get_points and
atera_to_parquet open transcripts.zarr.zip once instead of once per partition (the review measured 70.8 s
against 12.4 s for 200 partitions); the ZipStore pickles and reopens, so the graph still runs under the
processes scheduler.

Adapted to ec169cf helpers: zarr lookups go through _get_array/_get_group (zarr < 3.3).
Replaces the hand-built _read_x_chunk/_lazy_x. Each chunk now reads only its slice of indptr (anndata
opens the CSR with should_cache_indptr=False), where every task read the whole indptr before, and the
unused zarr_key argument goes away. read_elem_lazy needs anndata 0.12.

Adapted to ec169cf helpers: X and indptr are looked up with _get_group/_get_array (zarr < 3.3).
…one bbox test; drop unused keys

- _get_labels was a copy of xenium._get_labels.
- The OME pixel-size fallback now goes through parse_physical_size, so a size in nm is converted to um.
- One vectorised _bbox_overlaps serves the three bbox filters.
- _decode_packed_cell_id is the little-endian int64 view of the uint32 pair; the sentinel views as -1.
- Six AteraKeys members were never read.

Adapted to ec169cf helpers: the bbox lookup uses _get_array; keeps ec169cf's VARM_GROUP, COLOR and MANIFEST_* keys (the writer uses them) and drops only CATEGORIES/CODES.
…not by patching zarr

_patched_ragged_vlen_chunk_decode swapped V2Codec._decode_single for the whole process. Two threads
reading at once left the patch installed or removed it mid-read, any short 1-D chunk was accepted and
broadcast, and FusedCodecPipeline bypassed it. _PadRaggedVlenStore pads only the trailing chunk of a 1-D
vlen-utf8 array, and only when it holds exactly the remaining items, at the store level, so it holds
under both codec pipelines and needs no global state.
…kip macOS ._ files

The fallback took any *keypoints.csv in the root and fitted it unchecked. Other columns raised a
KeyError, a NaN row a LinAlgError, fewer than 3 rows gave a NaN affine, and S2_keypoints.csv placed
S1's H&E. It now reads <sample>_keypoints.csv of the same sample as the image and needs the four
Xenium Explorer columns, at least 3 finite rows and non-collinear points. Otherwise it warns and
keeps the image at Identity, as main does. The aligned-image glob skips ._ sidecars, which crashed
atera() (aligned_images=True by default). Real bundles name the files <sample>_he_image.ome.tif and
<sample>_keypoints.csv (all six Atera v1 bundles checked).
… level 0

The H&E (aligned_images=True by default) went through dask_image, so every chunk of every scale
waited on one task that decoded the whole plane; the four public Atera v1 H&E files are 2.8 to
7.3 GB. The morphology images read tifffile's stored levels: scale1 and below kept coordinates in
their own pixels (a bounding_box_query cropped 2^i too wide at scale i), scale_factors other than 2
were ignored, and the graph held tifffile's open file handle, so it failed under the processes
scheduler or any dask.distributed client, sdata.write() included.

H&E, IF and morphology images now read level 0 through _TiffTiles: each dask task opens the TIFF and
reads one tile of the file's tile grid, so the graph pickles. Image2DModel/Image3DModel.parse then
build the pyramid with the standard scale_factors. The aligned images keep their affine and the
keypoint fallback. 2D and 3D channels are both named from the OME-XML (3D used the file name).
morphology_3d_images=True always raised with the default arguments: the (1, 4096, 4096) chunks meant
for (c, y, x) reached Image3DModel.parse for (c, z, y, x) ('zip() argument 2 is shorter than argument
1' on the 10x tiny sample). The 3D parse now gets one z plane per chunk, and the scale factors
downsample y and x only, since a z-stack of a few planes cannot be halved four times.
…not the in-memory table

read_transcripts_for_genes took a gene's position in gene_offset from sdata.tables['table'].var, and
read_cell_boundaries took cell ids for polygon positions from its obs. After an HVG subset, the first
returned another gene's transcripts under the requested name; after a cell filter or reorder, the
second attached other cells' ids or raised IndexError (also in the documented pairing with
read_table_for_cells(bbox)). Both now read the positions from the bundle's own var and obs, so they
work with a filtered table or none.

Adapted to the ec169cf table layout (69723bd and b908dff not ported here): keeps ec169cf's cast(float, pixel_size), and the test picks GENE_B by var["feature_name"], since var is indexed by feature_id.
…_ids, feature_types)

The Atera bundles index var by feature_id (Ensembl) and name the type column feature_type; the xenium reader,
and downstream gene search and QC, read symbols from the var index, Ensembl ids from gene_ids and the type from
feature_types. The type values are already the Xenium ones. Applies to all three table readers.
pd.Index(series, name=None) keeps the Series' name, so var was indexed by a
'feature_name' index beside the 'feature_name' column. Once a duplicate symbol
is suffixed the two differ and anndata refuses to write (hub conversions hit it).
…obsm

n_genes counted every stored feature, the negative control probes and codewords and the unassigned
codewords included: on the 10x tiny sample 79 of 132 features are not genes, and n_genes exceeded
the gene count in 6 of 13 cells. As in xenium() (gex_only=True), X and var now hold the Gene
Expression features. The counts of all other features go to obsm['control_counts'], a DataFrame with
one column per feature (read from the CSC copy, where columns are cheap), and n_genes counts genes
only. On the tiny sample: X 13 x 53, control_counts 13 x 79, n_genes equal to the nonzeros of X,
and the DataFrame survives sdata.write() and read_zarr().

The three table readers now share _read_obs_var and _assemble_table, so read_table_for_cells has
n_genes, control_counts and obsm['spatial'] too, and read_var returns var in the same layout as the
table (indexed by symbol), so its documented recipe adds a column by name.

Adapted to ec169cf helpers (indptr via _get_array); keeps ec169cf's category colors and manifest extras in the table's uns. Ported only because the commits after it build on it; c3f7e70 replaces obsm["control_counts"] with table_controls.
…rywhere

Only _xenium_var made a repeated feature_name unique. Every other path used the raw names as
categorical categories or as a lookup: atera(transcripts=True) and atera_to_parquet raised
'Categorical categories must be unique', read_table_for_genes raised InvalidIndexError for every gene,
and read_transcripts_for_genes returned only the first duplicate's rows. All of them now use the same
anndata-style unique names (G, G-1), so G-1 addresses the second feature.

Adapted to ec169cf helpers: atera_to_parquet keeps _get_array/_get_group; _get_table keeps the uns colors and manifest extras.
read_transcripts_for_genes for a gene without transcripts (NegControlProbe_00009_2 on the 10x tiny
sample) fell back to an object-dtype frame that PointsModel rejects; it now returns an empty points
element with the usual dtypes. read_table_for_cells with no cells (cell_ids=[] or an empty bbox) built
a 0-row table that TableModel rejects; it returns the empty table without a region, since spatialdata
cannot validate an empty annotating table. spatialdata has no empty shapes element (ShapesModel
rejects an empty GeoDataFrame), so read_cell_boundaries over an empty bbox raises a ValueError that
says so instead of an IndexError.
shapely needs 3 vertices for a ring, so a single such polygon in polygon_sets made from_ragged_array
raise ('A linearring requires at least 4 coordinates') and aborted the default atera() read (cell and
nucleus boundaries are on by default). Such polygons are now kept as empty polygons, in place, with a
warning that counts them. XOA does emit 2- and 3-vertex polygons; whether Atera bundles do is not
known.
…s read

cells_table=False was overridden whenever labels, boundaries or transcripts were read, because the
polygons took cell ids from the table and the transcripts took feature names from it. Both now read
them from the bundle, so cells_table=False gives no table. With neither cell_labels nor
cell_boundaries read (only nucleus elements, or none), the table annotated the absent cell_labels and
spatialdata warned; it now annotates no element, since nucleus labels and polygons are numbered per
nucleus, not per cell.
…e when complete

ParquetWriter closed in a finally, so any failure left a truncated but valid parquet file, and a
failed rerun had already truncated the earlier complete one at open.

Adapted to ec169cf helpers: the zarr lookups use _get_array/_get_group.
The module-level logging.warning/info calls went to the root logger, and the first of them ran
logging.basicConfig as a side effect in the user's process.
The transcripts are partitioned by row range of a tile, not by tile; atera()'s X is lazy, not eager;
the module note listed what no longer lives where it said. (The aligned_images entry was updated with
the keypoint fallback.)
No new mypy errors in the changed files against 5827d7b plus our 7 commits, with the repo's mypy hook.

Adapted to ec169cf helpers: X and indptr stay on _get_group/_get_array, keys stay wrapped in str() (7fc9d3c is not ported).
On the 10x tiny sample, read_elem returns var/feature_name as a numpy StringDType array, which
np.asarray(..., dtype=str) cannot cast, so atera() failed in _read_feature_names.
The dense obsm['control_counts'] DataFrame took 10.58 GB on the Atera v1 WTA kidney (399,120 cells x
6,625 non-gene features), and anndata cannot write a pandas sparse DataFrame. atera() now returns
table_controls next to table: a sparse CSR X with the stored dtype, the negative control probes and
codewords and the unassigned codewords in var with their feature_types, and the same obs index,
instance ids and annotated element as table. table keeps the Gene Expression features only, and
read_table_for_cells returns those.

Adapted to ec169cf: the gene table keeps the category colors and manifest extras in uns.
…ture_matrix

atera() splits the features into table (Gene Expression) and table_controls, with var in the xenium
layout. The writer now appends table_controls (same cells in the same order, checked) and maps
feature_types/gene_ids/the symbol index back to feature_type/feature_id/feature_name. Features are
written genes first, then controls, which is the 10x tiny Atera bundle's own order (53 Gene Expression
features first). A single table in the bundle's var layout is still written as it is.

The reader keys the var category colors by the column it names (feature_types_colors) and puts them on
both tables.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant