Skip to content

Avoid structured data in ux.open_dataset, ux.open_mfdataset, and from_xarray - #1887

Open
Sevans711 wants to merge 7 commits into
mainfrom
sevans/deprecate-structured-open_dataset
Open

Sevans711 wants to merge 7 commits into
mainfrom
sevans/deprecate-structured-open_dataset

Conversation

@Sevans711

@Sevans711 Sevans711 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1854

Overview

EDIT: as per messages below, this PR has been updated from its original form (clarify/warn if using structured inputs in ux.open_dataset or ux.open_mfdataset) to instead fully remove support for structured data in ux.open_dataset, ux.open_mfdataset, UxDataset.from_xarray, and UxDataArray.from_xarray. This expands the PR's scope beyond #1854 which only discussed ux.open_dataset.

Fully removing support for structured data in from_xarray also means deleting most of the contents of test_from_xarray_with_grid_from_latlon which was introduced by PR #1569. With my current understanding of the uxarray API, I believe the "correct" solution to the underlying issue solved by that PR (#1410) would have instead been to simply recommend the use of the existing UxDataset.from_structured() method.

Additional expansions of scope:

  1. Fixed small typo in UxDataArray.from_xarray(). For clarity, also renamed corresponding internal variable from ds to da.
  2. Improved docstrings of open_dataset() and open_mfdataset(), especially the description of grid_filename_or_obj.
  3. Removed the recently-introduced unnecessary lines in _map_dataset_to_ugrid which are specific to structured grids and not backed by tests. For more details see: ux.open_dataset currently supports structured data #1854 (comment)

Misc. notes:

PR Checklist

General

  • An issue is created and linked
  • Added appropriate labels (if your uxarray repo permissions allow it)
  • Filled out Overview and Expected Usage (if applicable) sections

Testing & Benchmarking

  • There is adequate test coverage of changes from this PR (add new tests if needed)
  • [N/A] If this PR could affect performance, ran ASV benchmarks and confirmed they show expected behavior (add a new benchmark if necessary)

Documentation and Examples

  • Docstrings updated with any function changes, and included in all new functions
  • [N/A] User (public) functions added to docs/api.rst; internal (private) function names start with an underscore (_)
  • [N/A] If touched any notebook files, cleared the output of all cells before committing
  • [N/A] If added new notebook files, put into appropriate directories and referenced in appropriate files

AI Disclosure

AI Usage: just GitHub Copilot's inline code suggestions.

  • I have tested and take responsibility for all AI-generated content in my PR.

open_mfdataset, too. Also, further improves docstrings for those functions.

Also, removes `if grid.source_grid_spec == "Structured"` branch in `_map_dataset_to_ugrid`.
@Sevans711 Sevans711 added the bug Something isn't working label Nov 9, 2026
@Sevans711
Sevans711 requested a lite review from Copilot November 9, 2026 22:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active issues remain, including a critical unguarded dimension-mapping path that can raise UnboundLocalError, an invalid xarray.open_mf_dataset API reference, and misleading API documentation.

3 open issues
What changed in this PR

This pull request clarifies that ux.open_dataset and ux.open_mfdataset are intended for unstructured data, warns about forthcoming structured-input deprecation, and updates related documentation and tests.

Changes:

  • Added structured-grid detection and FutureWarnings to both dataset-opening APIs.
  • Improved documentation and migration guidance for grid and data inputs.
  • Added warning regression tests and removed an obsolete structured-grid mapping shortcut.

This pull request requires changes.

File Review
uxarray/​core/​utils.py Critical (3 votes): Restore a guard for datasets containing neither structured source dimension; otherwise _map_dims_to_ugrid can raise UnboundLocalError.
uxarray/​core/​api.py Moderate (4 votes): Correct the nonexistent xarray.open_mf_dataset reference to xarray.open_mfdataset. Nit (4 votes): Keep grid_filename_or_obj documented as the grid source rather than a data-file path.
test/​core/​test_api.py Adds warning coverage for structured inputs to open_dataset and open_mfdataset.

Review effort: Lite

Comment thread uxarray/core/utils.py Resolved
Comment thread uxarray/core/api.py Resolved
Comment thread uxarray/core/api.py Resolved
@Sevans711

Sevans711 commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

From internal discussion, @erogluorhan confirmed that from_xarray() is not intended to support structured (lat/lon) data. Its current docstring clearly mentions that it expects unstructured inputs, so the accidental support for structured data in this particular case can be removed without any need for a deprecation warning (and, should be removed as part of this PR).

And, further clarification: Orhan's preference is to also proceed without a deprecation warning in the ux.open_dataset case as well. I will update the code appropriately and then send a message here once it is done to clarify that the PR is once again ready for reviews. For now it needs updates before it should be reviewed.

@erogluorhan erogluorhan left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Sevans711 , thanks for considering clarifications! Also, thank you for communicating the internal discussion!

Let me bring additional important parts of our internal discussion here for our user & contributor community's reference:

UXarray has native support for only unstructured grids, not for structured grids. Therefore, any support for structured grids should be explicit through only a few specialized APIs, e.g. from_structured(). The core API members such as open_dataset() shouldn't support structured grids in any way.

That said, this PR's title, description, and the work need to be updated accordingly:

  • I’d rather NOT have open_dataset()’s docstrings mention anything about structured grids.
    • Maybe a short phrase similar to from_xarray()’s, i.e. “... data residing on an unstructured grid”, or something similar about the grid being an unstructured grid can be added though.

Comment thread uxarray/core/api.py Resolved
Comment thread uxarray/core/api.py Resolved
@Sevans711 Sevans711 changed the title Clarify/warn to avoid structured data in ux.open_dataset and ux.open_mfdataset Avoid structured data in ux.open_dataset and ux.open_mfdataset Nov 11, 2026
@Sevans711 Sevans711 changed the title Avoid structured data in ux.open_dataset and ux.open_mfdataset Avoid structured data in ux.open_dataset, ux.open_mfdataset, and from_xarray Nov 11, 2026
@Sevans711

Copy link
Copy Markdown
Collaborator Author

@erogluorhan Thank you for providing the additional context here. I believe all your points have been addressed. I also updated the PR title and the PR overview message to clarify the new scope and intentions here. Please let me know if you find anything I missed!

@erogluorhan erogluorhan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Sevans711 ! This covers everything I had pointed out before.

I only have a couple inline comments, plus Claude helped me find a few points as below (Despite issue 1, I can see the docs built successfully though, so please take them all with a pinch of salt):

1. ⛔ This breaks the user guide and will fail the docs build

The PR description states "Nothing in the docs currently uses from_xarray() with structured data." This isn't quite right — docs/user-guide/structured.ipynb has a whole section, "Pairing a Structured Grid with an Existing Dataset" (md cell 13, code cells 14 and 16), built on exactly this:

uxgrid = ux.Grid.from_structured(lon=ds_air_temp.lon, lat=ds_air_temp.lat)
uxds = ux.UxDataset.from_xarray(ds_air_temp, uxgrid=uxgrid)   # now raises DimensionError
regional = uxds["air"][0].subset.bounding_circle((-100.0, 40.0), 15)  # cascades

I ran this on the PR branch and it raises DimensionError. The notebook is listed in docs/userguide.rst:136 and is not in nb_execution_excludepatterns (docs/conf.py:233), so it executes at docs build time.

That section is the documented rendering of #1410 — the same workflow the deleted test covered. It should be rewritten to the UxDataset.from_structured(ds) equivalent (and the prose about "UXarray records the structured source dimensions" dropped) as part of this PR.


2. ⛔ The UxDataArray remedy in the error message doesn't work

Consider UxDataset.from_structured(arr.to_dataset()).to_dataarray() instead.

I verified this: it returns a plain xr.DataArray with a spurious variable dim of size 1 — the uxgrid is gone, which is the opposite of what the user wanted. It also fails outright for unnamed arrays.

The working form is:

UxDataset.from_structured(da.to_dataset())[da.name]   # -> UxDataArray, uxgrid attached

Error messages that route users to a broken API are worse than no suggestion at all.


3. The dims reported in the error message are the wrong dims

...received a grid with source_grid_spec='Structured' and dims ['n_node', 'n_face', 'n_max_face_nodes']

uxgrid._ds.dims are the post-conversion UGRID dims — they're identical for every grid, so they carry no diagnostic signal and actively mislead ("I passed lat/lon, why is it talking about n_node?"). It also surfaces a private attribute in user-facing text.

Either drop the clause, or print uxgrid._source_dims_dict (e.g. {'n_face': ('lon', 'lat')}), which actually names what the user passed in.


4. open_grid() is left as an unguarded hole

g = ux.open_grid(structured_ds)   # still silently succeeds: spec='Structured', n_face=32

open_grid is as much a core entry point as open_dataset, and the agreed principle here is that core API shouldn't support structured grids in any way. Post-PR it returns a Grid that is now a dead end — data can no longer be attached to it through any public path.

Guarding open_grid (not Grid.from_dataset, which Grid.from_structured(ds=...) delegates to) would close the issue properly. If deferring is preferred, let's open an explicit follow-up issue rather than leaving it silent.


5. Backwards compatibility should be called out in the release notes

This is a hard break on a path that #1569 added deliberately and covered with a regression test for #1410, in a released version. Skipping the deprecation cycle was agreed, so no objection to the decision itself — but it should be called out in the release notes.

Note that this also makes items 2 and 3 more than cosmetic: the error message is the only migration guidance users will get.


6. Is DimensionError the right error type?

DimensionError is documented as "wrong size(s), name(s), or number of dimensions," which isn't really what happened — the user supplied an unsupported grid kind. GridInvalidError ("data does not correspond to a valid uxarray.Grid") looks like a closer fit, and both subclass ValueError so it's not a compatibility concern.

Minor, but given the effort that went into a curated errors.py (#1556), worth picking precisely.

must record its source dimensions so ``UxDataset.from_xarray`` flattens the
structured (lon, lat) data variables onto ``n_face``."""
ds = xr.tutorial.open_dataset(ds_name)
def test_from_xarray_with_grid_from_latlon():

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about adding a "_fails_with" to this test case similar to the one in test_api? The current name reads a bit like expected usage, not a failure case.

Comment thread uxarray/core/api.py
filename_or_obj, chunks=corrected_chunks, **kwargs
)

if _represents_structured_data(uxgrid):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple points:

  1. If we need to implement it this way: This is okay, but I am wondering if a simple, one-line check wherever this was needed (without defining a newer internal function) as the following could suffice:

if uxgrid.source_grid_spec == "Structured":

  1. Maybe more importantly, could we implement this in a way that the unintentional leakage of structured grids this far into the core API could be avoided in the first place (rather than catching them here after all)? For instance, the support for ingestion of structured grids could be separated from the main flow a bit?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ux.open_dataset currently supports structured data

3 participants