Repository navigation
Skip unknown nested fields when resolving Avro read schemas - #4289
haizhou-zhao wants to merge 1 commit into
Conversation
Fields are resolved by ID, so a file field that is not in the read schema should be read and dropped. This already worked for primitives, but a file-only map, list, or struct raised ResolveError, e.g. when a writer adds extra map fields to data_file in manifests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AP24110011695
left a comment
There was a problem hiding this comment.
Reviewed the diff against ReadSchemaResolver and ran the new test locally (Windows, Python 3.12, pure-Python decoder fallback): test_read_manifest_entry_with_unknown_nested_fields passes.
I also probed the mismatch guarantees that the new None pass-through must not weaken — all four still raise on this branch, including the field-level paths the modified accessors serve (file map vs read string, file list vs read map, both top-level and nested inside a struct).
One suggestion: none of the tests here pin a field-level mismatch as still raising — the existing not-aligned tests in test_resolver.py all resolve at the top level, which bypasses field_partner + the container accessors this PR touches. Since the change widens what those accessors accept, a small regression test would lock the boundary:
def test_field_level_nested_mismatch_still_raises() -> None:
file_struct = StructType(
NestedField(10, "x", MapType(11, IntegerType(), 12, BinaryType()), required=False)
)
read_struct = StructType(NestedField(10, "x", StringType(), required=False))
with pytest.raises(ResolveError):
resolve_reader(file_struct, read_struct)Also confirmed the fallback direction reads correctly: inside a file-only parent there is no read partner to mismatch against, so nested children resolving to read-and-drop is the right call.
Closes #4288
Rationale for this change
Avro fields are resolved by ID, so a field that exists only in the file should be read and dropped. Primitives already work this way. A file-only map, list, or struct instead raised
ResolveError: File/read schema are not aligned for map, got None, becauseSchemaPartnerAccessorraised whenever the partner wasNone.With this change, the accessor returns
Nonefor the children of a field that has no partner, and the existingNonehandling inReadSchemaResolverreads and drops them. A real type mismatch, such as a map in the file and a string in the read schema, still raises.Are these changes tested?
Yes.
test_read_manifest_entry_with_unknown_nested_fieldswrites a v2 manifest entry whosedata_filehas an unknown map, list, and struct (with a nested map) placed before known fields. It then checks that the entry reads back unchanged. The test fails onmainwith theResolveErrorabove.Are there any user-facing changes?
Yes, a bug fix: PyIceberg can read manifests and other Avro files that contain nested fields missing from the read schema.