Skip to content

Skip unknown nested fields when resolving Avro read schemas - #4289

Open
haizhou-zhao wants to merge 1 commit into
apache:mainfrom
haizhou-zhao:fix/skip-unknown-nested-avro-fields
Open

haizhou-zhao wants to merge 1 commit into
apache:mainfrom
haizhou-zhao:fix/skip-unknown-nested-avro-fields

Conversation

@haizhou-zhao

Copy link
Copy Markdown

Closes #4288

Rationale for this change

Avro fields are resolved by ID, so a field that exists only in the file should be read and dropped. Primitives already work this way. A file-only map, list, or struct instead raised ResolveError: File/read schema are not aligned for map, got None, because SchemaPartnerAccessor raised whenever the partner was None.

With this change, the accessor returns None for the children of a field that has no partner, and the existing None handling in ReadSchemaResolver reads and drops them. A real type mismatch, such as a map in the file and a string in the read schema, still raises.

Are these changes tested?

Yes. test_read_manifest_entry_with_unknown_nested_fields writes a v2 manifest entry whose data_file has an unknown map, list, and struct (with a nested map) placed before known fields. It then checks that the entry reads back unchanged. The test fails on main with the ResolveError above.

Are there any user-facing changes?

Yes, a bug fix: PyIceberg can read manifests and other Avro files that contain nested fields missing from the read schema.

Fields are resolved by ID, so a file field that is not in the read schema
should be read and dropped. This already worked for primitives, but a
file-only map, list, or struct raised ResolveError, e.g. when a writer adds
extra map fields to data_file in manifests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@AP24110011695 AP24110011695 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff against ReadSchemaResolver and ran the new test locally (Windows, Python 3.12, pure-Python decoder fallback): test_read_manifest_entry_with_unknown_nested_fields passes.

I also probed the mismatch guarantees that the new None pass-through must not weaken — all four still raise on this branch, including the field-level paths the modified accessors serve (file map vs read string, file list vs read map, both top-level and nested inside a struct).

One suggestion: none of the tests here pin a field-level mismatch as still raising — the existing not-aligned tests in test_resolver.py all resolve at the top level, which bypasses field_partner + the container accessors this PR touches. Since the change widens what those accessors accept, a small regression test would lock the boundary:

def test_field_level_nested_mismatch_still_raises() -> None:
    file_struct = StructType(
        NestedField(10, "x", MapType(11, IntegerType(), 12, BinaryType()), required=False)
    )
    read_struct = StructType(NestedField(10, "x", StringType(), required=False))
    with pytest.raises(ResolveError):
        resolve_reader(file_struct, read_struct)

Also confirmed the fallback direction reads correctly: inside a file-only parent there is no read partner to mismatch against, so nested children resolving to read-and-drop is the right call.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avro read fails with ResolveError when the file has a map/list/struct field not in the read schema

2 participants