Check current-schema-id before combining Iceberg REST cleanup and schema changes
When cleanup and column addition share a REST request, the current schema reference must still resolve to a retained definition.
Readable metadata is the acceptance boundary
A metadata update should be accepted only after its current schema ID resolves to a retained schema definition, rather than merely after JSON is produced. When an Iceberg REST request combines cleanup with a column addition, a small data contract can check this connection separately from the response status. The reference check in the Truthound Core Python SDK compares ID tables extracted by the operator. [1, 2, 3]
A failed response can follow a pointer change
Apache Iceberg issue #18677 reports that, on the main development branch, TableMetadata.Builder.removeSchemas removes IDs from schemas but leaves them in schemasById. An add-schema in the same REST request can find that stale ID, leaving the new definition out of the array while changing current-schema-id. The report shows current-schema-id=1 with only 0 retained in schemas, followed by a failing loadTable after the commit. These are the conditions of a public report; this article does not present an engine reproduction or a verified fix. [1]
Keep request and table version boundaries
Compare current-schema-id with the schemas ID set in metadata from before and after a request that combines cleanup and a schema change. If the reference breaks, stop subsequent cleanup or writes. Record the table identity, metadata locations on both sides, request ID, current-schema-id, and schemas.id. Mixing a schemas list from another table or another point in time can make a broken connection appear valid. [1, 3]
Exercise one ID connection in an illustrative fixture
Define illustrative metadata with schemas=[0,1] and current-schema-id=1: the current ID resolves. With schemas=[0] and current-schema-id=1, it does not. The first case passes the reference-presence condition because ID 1 is retained. The second can be represented as JSON but has no schema to read, so it must be rejected. This expected result follows from the declared ID set; it is not copied from observed output. [1, 3]
Reference presence does not verify schema contents
ReferentialIntegrityValidator in Truthound Core Python SDK 3.3.2 can receive a current-schema-id column and an operator-prepared schemas.id reference table. It checks whether values in the specified column exist in the reference column of reference_data. Whether the definition under the same ID matches the approved column structure needs a separate check. The SDK does not repair REST commit atomicity or the builder index. [1, 2, 3]
Do not generalize across catalog implementations
The report excludes InMemoryCatalog and JdbcCatalog because they apply each operation to freshly built metadata. It instead concerns REST servers using the CatalogHandlers commit loop or a copy of it. The similar partition spec path is reported to reject the request rather than commit unreadable metadata. Keeping the schema and spec outcomes distinct makes the response scope more precise. [1]
Separate the operations when combined acceptance lacks evidence
If cleanup and the change do not need to share a request, split them into separate approval stages and check an actual loadTable after each. A passing reference check does not establish the new column type, field ID, or compatibility. Treat a resolved reference as a necessary condition for proceeding to those checks, rather than as assurance for the entire change. [1, 3]
Application conditions
Extract current-schema-id and schemas.id before and after the same REST request.
Alternatives
Separate cleanup and the schema change, checking loadTable after each.
Limitations
ID-reference presence does not establish schema contents or REST commit atomicity.
Evidence
- Event checked
- Subject checked
- Modified
