Styxis
Company
Knowledge

Check Hudi narrow-integer write, read, and statistics readiness separately

Check write, read, and statistics readiness separately for narrow-integer bulk_insert. Keep ShortType and ByteType paths distinct; file presence or a successful ordinary insert cannot approve statistics-dependent consumption.

A file does not approve every path

A created file does not establish every narrow-integer path is compatible. Truthound Core can inspect an operator-prepared readiness record while write success, reader values, and statistics availability remain separate conditions. [1, 2, 3]

The narrow-integer annotation report

Hudi issue #20503 reports master on Spark 3.5 writing ShortType bulk_insert as INT32+INT(16), unlike other write paths. Vectorized reads fail and column statistics are skipped; ByteType fails during writing. These are reported paths, not a universal failure of all integer data. [1]

Join operation and representation

Record input dtype, write operation, physical annotation, reader status, and stats collection. Test ShortType and ByteType independently. Do not count a skipped statistics file as a completed stats-dependent ingestion merely because storage exists. [1, 3]

A required-statistics prerequisite

Declare small-column statistics a required input for the illustrative downstream task. stats_available=true passes that readiness prerequisite; false fails and blocks stats-based pruning approval. The flag does not prove the statistics or the row values are correct. [1, 3]

Readiness is not added support

ColumnPairValidator in Truthound Core Python SDK 3.3.2 can compare prepared non-null availability flags with true. The operator examines files and stats records. The SDK does not add Hudi narrow-integer writer or reader support. [1, 2, 3]

Show partial alternatives honestly

Verify the reported insert control and nonvectorized reader on the real version, checking values and statistics separately. An alternative that restores reads but leaves required stats absent is only a partial response. [1, 3]

Bind results to the write path

Keep each readiness result attached to its write path. Otherwise a successful insert control can incorrectly approve bulk_insert, or a successful read can conceal a missing input to later pruning. [1, 3]

Application conditions

Prepare narrow dtype, write operation, Parquet annotation, and read/stats states.

Alternatives

Check insert and nonvectorized controls on that version separately for values and statistics.

Limitations

  • Statistics presence proves neither value correctness nor every pruning decision.

Evidence

Event checked
Subject checked
Modified