Repository navigation
Atera reader (code review of #426) - #444
Draft
LucaMarconato wants to merge 29 commits into
Draft
LucaMarconato wants to merge 29 commits into
LucaMarconato wants to merge 29 commits into
Conversation
Adds a native reader for 10x Genomics Atera datasets: cell-by-gene expression table, cell/nucleus segmentation labels and boundary polygons, per-transcript locations, and morphology images. - Table (`cell_feature_matrix.zarr.zip`/`csc_cell_feature_matrix.zarr.zip`) is read directly with `anndata`'s own zarr IO (`anndata.io.read_elem`/ `anndata.io.sparse_dataset`), with `X` lazily backed by dask. - Cell/nucleus labels are lazily backed by dask; boundary polygons are read via `shapely.from_ragged_array`, with an optional tiled/pyramided partial-read path (`read_cell_boundaries`) for large bundles. - Transcripts are read lazily per spatial tile via dask-delayed. - Morphology images are read via `tifffile`'s native OME-TIFF tile grid, avoiding materializing full-resolution planes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A tile can hold 100M+ transcripts. Each dask partition was a whole tile, so writing a large dataset ran many multi-GB partitions at once and swapped to disk. atera_to_parquet likewise decoded a whole tile per batch. - _read_transcript_tile takes start/stop and reads only those rows; the per-row gene index comes from searchsorted over gene_offset. - _get_points makes one partition per 5M-row chunk of a tile. - atera_to_parquet reads and writes each tile in the same chunks. - Add tests that chunked reads match whole-tile reads across gene boundaries, and that partitions are bounded. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Removed code for calculating the number of genes detected per cell as this is now in the atera loader
- Replace zarr.Group.get_array/get_group (zarr >= 3.3 only) with small helpers that work on any zarr 3.x; we don't pin zarr>=3.3 for now. - Raise a clear FileNotFoundError when the bundle directory doesn't exist, instead of a misleading pixel-size error.
|
Notebooks changed:
Review visual diffs and comment on any notebook cell. All comments sync back to this PR. Powered by ReviewNB |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #444 +/- ##
==========================================
+ Coverage 66.60% 68.57% +1.97%
==========================================
Files 26 30 +4
Lines 3345 4290 +945
==========================================
+ Hits 2228 2942 +714
- Misses 1117 1348 +231
🚀 New features to boost your workflow:
|
| "### 2. Clone the branch with the Atera reader\n", | ||
| "\n", | ||
| "```bash\n", | ||
| "git clone --branch atera-reader https://github.com/10XGenomics/spatialdata-io\n", |
Collaborator
There was a problem hiding this comment.
This will need to be updated
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Atera support is implemented by @stephenwilliams22 in #426. Such PR is from a fork form an organization, and this doesn't allow spataldata-io maintainers to edit it directly. I will therefore work on this branch. It can then be merged into the #426, or merged into main (closing #426).
This PR implements code changes arising from the code review of #426.