normalize

normalize

Bring a long-format signal frame to the dtypes the detectors compute with.

Data read from parquet, databases or CSV arrives in many dtypes: pandas nullable types (Int64, Float64, boolean, string), pyarrow types (int64[pyarrow], bool[pyarrow]), or numbers stored as text. pandas arithmetic on these differs from NumPy in ways that change results without an error: a comparison with a missing value is <NA> instead of False, cumsum is not defined for bool[pyarrow], and text numbers concatenate instead of adding up.

:func:normalize_signal_frame converts once, up front, so every detector sees the same NumPy-backed dtypes:

• numeric value columns become int64 (no missing values) or float64;
• numbers stored as text in value_double / value_integer are parsed, with a :class:~ts_shape.errors.DataQualityWarning counting the values that are not numbers;
• boolean value columns become bool (no missing values) or object holding True / False / NaN, as pandas builds them from NumPy data;
• string columns (value_string, the uuid column) become object;
• a pyarrow timestamp time column becomes datetime64.

unique_text

unique_text(series: Series) -> set[str]

set(series.dropna().astype(str).unique()), from the unique values.

Never converts the whole column to text: only its distinct values are turned into strings (a column with mixed non-text values falls back to the row-wise conversion).

text_mask

text_mask(series: Series, text: str) -> np.ndarray

Rows whose value, as text, equals text (missing values never match).

The same rows as series.astype(str) == text, compared on the unique values and mapped back with isin.

text_keys

text_keys(series: Series) -> np.ndarray

series.astype(str).to_numpy() for a column without missing values.

Built from the unique values: rows with the same value share one str.

normalize_signal_frame

normalize_signal_frame(
    frame: DataFrame,
    *,
    time_column: str = "systime",
    uuid_column: str = "uuid"
) -> pd.DataFrame

Return frame with NumPy-backed dtypes (see the module docstring).

Only the time column, the uuid column and value_* columns are touched; other columns keep their dtype. frame is changed in place and returned.

Parameters:

Name	Type	Description	Default
frame	DataFrame	Long-format signal data (a copy owned by the caller).	required
time_column	str	Timestamp column.	'systime'
uuid_column	str	Signal identifier column.	'uuid'

Returns:

Type	Description
DataFrame	The same frame object, with converted columns.