Styxis
Company
Knowledge

SPSS to Parquet: preserving metadata instead of clearing attrs to make a write succeed

Deleting metadata to make a write succeed is different from completing a preservation task. Specify contracts for both values and descriptive metadata.

SPSS to Parquet: preserving metadata instead of clearing attrs to make a write succeed

Clearing attrs to make a Parquet write succeed can retain numeric rows while discarding the information needed to interpret them. A migration required to preserve variable labels and creation times cannot finish merely because a file was created. The handover needs a way to read the table and its metadata together. [1, 2, 3]

What the primary source establishes

A report describes creation and modification datetime metadata passed through read_spss causing Parquet's JSON metadata construction to fail. Clearing attrs allows writing, but the report does not conclusively assign responsibility to one package. [1]

The handover decision

Preserve variable labels and the approved serialized forms of creation and modification times in a separate manifest. Link that manifest to the reread table before declaring the migration complete. Record dataset_id, metadata_key, source_type, and serialized_value so the destination remains interpretable. [1, 3]

A reference defined before the output

In the illustrative migration contract, creation_time is a required metadata key. A present key satisfies that presence requirement; an absent key fails it even when every data row was written successfully. The expected key comes from the preservation policy, not from whichever metadata the output happens to retain. [1, 3]

The scoped Truthound check

To apply ColumnExistsValidator in Truthound Core 3.3.2, pivot the sidecar's required keys into actual columns for each dataset_id. Presence of creation_time and validity of its timestamp are different checks. Passing an attribute dictionary does not automatically validate meaning at every metadata level. [1, 2, 3]

Limits and alternative choices

Presence does not prove that a timestamp is the real creation time or that SPSS and Parquet preserve every semantic detail. Store JSON-compatible metadata in a versioned sidecar with the data's identity. If the approved preservation policy excludes that metadata, its absence alone does not constitute a migration failure. [1, 3]

Application conditions

Pivot required metadata into a wide table by dataset_id, making creation_time an actual column linked to the original sidecar.

Alternatives

Store JSON-compatible metadata in a versioned sidecar with the data's identity.

Limitations

  • Presence does not prove that a timestamp is the real creation time or that SPSS and Parquet preserve every semantic detail.

Evidence

Event checked
Subject checked
Modified