Styxis
Company
Knowledge

pandas read_html: keeping nested-table headers out of observation rows

Record the boundary between DOM headers and body rows in the ingestion contract. Otherwise, a rising number of collected rows can be mistaken for an increase in observations.

pandas read_html: keeping nested-table headers out of observation rows

A web-table import can contain valid numeric rows while also admitting a nested heading as an observation. Removing every string that resembles a title can then delete legitimate data. Preserve each row's role within the selected table to decide what actually needs to be collected again. [1, 2, 3]

What the primary source establishes

A report about nested HTML tables shows the A and B header entering the data once more when the table is read through bs4 or html5lib. The expected output in the report contains only the numeric body row. [1]

The handover decision

The collector should record the selected table ID and the tbody row inventory. The aggregation owner can then compare suspected header rows with their original DOM positions and determine which material needs reloading. Keep table_id, dom_row_role, and parsed_row_role together so that value filtering does not replace provenance. [1, 3]

A reference defined before the output

The illustrative table contains a header row and a body row, with only the latter defined as an observation. A body role is admissible; a header role is not. Do not make the header pass by coercing its text into numbers. A valid data row can happen to contain the same words as a heading, so string equality alone is not a safe deletion rule. [1, 3]

The scoped Truthound check

Truthound Core 3.3.2 offers InSetValidator to check whether a row role independently extracted from the DOM belongs to the allowed body set. The collector remains responsible for understanding HTML structure and selecting the table. An incorrectly extracted role can therefore pass the check while concealing header contamination. [1, 2, 3]

Limits and alternative choices

Deleting rows merely because their text matches the heading can remove valid observations. Compare a parser selected by explicit table ID with a DOM-based extraction of the same table. If the source genuinely includes those strings as body data, matching the text is insufficient evidence of header contamination. [1, 3]

Application conditions

Prepare the DOM row role and parsed row role for the same selected table and source row.

Alternatives

Compare a parser selected by explicit table ID with a DOM-based extraction of the same table.

Limitations

  • Deleting rows merely because their text matches the heading can remove valid observations.

Evidence

Event checked
Subject checked
Modified