Preserve mixed-case Hadoop option keys through Delta DataFrame maps
Link each case-sensitive Hadoop option key to its effective Configuration lookup. A forwarded string alone cannot establish that the I/O setting took effect; keep credential values out of the record.
Map convenience and key identity
Passing through a case-insensitive option map must not erase the identity of a case-sensitive Hadoop key. Truthound Core can compare prepared effective-key resolution states before I/O configuration is approved. [1, 2, 3]
Lowercased fs and dfs keys
Delta issue #8016 reports master with Spark 4.1 lowercasing mixed-case fs and dfs DataFrame options. The original key then resolves to null in Hadoop Configuration and cache matching can differ. The issue identifies both Scala and V2 map-conversion boundaries. [1]
Keep names without secrets
Record nonsensitive original key names, transmitted names, and effective Configuration lookup results. If the original key disappears, withhold approval for that configuration. Do not put credential values into the inspection table. [1, 3]
Resolve the original key
For illustrative fs.azure.skipUserGroupMetadataDuringInitialization, the expected lookup value is true. Normalize a missing original-key lookup as resolved=false. resolved=true passes; false fails even if a lowercase spelling holds a value elsewhere. [1, 3]
Comparison does not change settings
ColumnPairValidator in Truthound Core Python SDK 3.3.2 can compare non-null prepared resolution flags. The operator inspects effective configuration and preserves key case. The SDK does not modify Hadoop settings or validate credentials. [1, 2, 3]
Check the required I/O behavior
Review a supported path that preserves original keys, then verify the required behavior with a small I/O operation. Key preservation alone does not establish external connectivity or every option’s behavioral effect. [1, 3]
Effective keys explain the boundary
The useful completion evidence names the effective key, not merely the configuration text the application submitted. This distinguishes a lost option from a service-side failure after a correctly configured request. [1, 3]
Application conditions
Prepare original/transmitted key names and effective lookup results without secrets.
Alternatives
After a case-preserving supported path, check required behavior with a small I/O operation.
Limitations
Key preservation does not prove connectivity, every option’s behavior, or credential validity.
Evidence
- Event checked
- Subject checked
- Modified
