Styxis
Company
Knowledge

Preserve mixed-case Hadoop option keys through Delta DataFrame maps

Link each case-sensitive Hadoop option key to its effective Configuration lookup. A forwarded string alone cannot establish that the I/O setting took effect; keep credential values out of the record.

Map convenience and key identity

Passing through a case-insensitive option map must not erase the identity of a case-sensitive Hadoop key. Truthound Core can compare prepared effective-key resolution states before I/O configuration is approved. [1, 2, 3]

Lowercased fs and dfs keys

Delta issue #8016 reports master with Spark 4.1 lowercasing mixed-case fs and dfs DataFrame options. The original key then resolves to null in Hadoop Configuration and cache matching can differ. The issue identifies both Scala and V2 map-conversion boundaries. [1]

Keep names without secrets

Record nonsensitive original key names, transmitted names, and effective Configuration lookup results. If the original key disappears, withhold approval for that configuration. Do not put credential values into the inspection table. [1, 3]

Resolve the original key

For illustrative fs.azure.skipUserGroupMetadataDuringInitialization, the expected lookup value is true. Normalize a missing original-key lookup as resolved=false. resolved=true passes; false fails even if a lowercase spelling holds a value elsewhere. [1, 3]

Comparison does not change settings

ColumnPairValidator in Truthound Core Python SDK 3.3.2 can compare non-null prepared resolution flags. The operator inspects effective configuration and preserves key case. The SDK does not modify Hadoop settings or validate credentials. [1, 2, 3]

Check the required I/O behavior

Review a supported path that preserves original keys, then verify the required behavior with a small I/O operation. Key preservation alone does not establish external connectivity or every option’s behavioral effect. [1, 3]

Effective keys explain the boundary

The useful completion evidence names the effective key, not merely the configuration text the application submitted. This distinguishes a lost option from a service-side failure after a correctly configured request. [1, 3]

Application conditions

Prepare original/transmitted key names and effective lookup results without secrets.

Alternatives

After a case-preserving supported path, check required behavior with a small I/O operation.

Limitations

  • Key preservation does not prove connectivity, every option’s behavior, or credential validity.

Evidence

Event checked
Subject checked
Modified