harmonization

harmonization

DataHarmonizer

DataHarmonizer(
    dataframe: DataFrame,
    time_column: str = "systime",
    uuid_column: str = "uuid",
    value_column: str = "value_double",
)

Bases: Base

Data Harmonization for multi-signal timeseries.

Provides utilities to pivot, resample, align, and fill gaps across multiple UUID-keyed signals stored in long (stacked) format.

Methods: - pivot_to_wide: Pivot long-format to wide-format (one column per UUID). - resample_to_uniform: Resample to a uniform time grid with interpolation. - detect_gaps: Identify time gaps per UUID exceeding a threshold. - fill_gaps: Fill detected gaps using various strategies. - align_asof: Align two UUID signals using merge_asof. - merge_multi_signals: End-to-end harmonization pipeline.

Keep a time-sorted copy of dataframe.

Parameters:

Name	Type	Description	Default
dataframe	DataFrame	Long-format input.	required
time_column	str	Timestamp column. Text and pyarrow timestamps are converted; an empty untyped column becomes datetime64.	'systime'
uuid_column	str	Signal identifier column.	'uuid'
value_column	str	Column holding the values to harmonize.	'value_double'

Raises:

Type	Description
ColumnNotFoundError	If one of the three columns is missing.

pivot_to_wide

pivot_to_wide(aggfunc: str = 'first') -> pd.DataFrame

Pivot long-format DataFrame to wide-format with one column per UUID.

Memory: the wide frame is dense, one row per distinct timestamp of any signal and one column per UUID, so it holds timestamps x signals cells (8 bytes each for float values) however sparse the long frame is. Signals sampled at their own instants barely share rows: 100 such signals of 100,000 samples each (10 million long rows, about 0.5 GB) pivot to 10 million x 100 cells, 8 GB, mostly NaN. Select the signals you need and resample or round the timestamps onto a common grid before pivoting large data (:meth:resample_to_uniform pivots too, then resamples).

Parameters:

Name	Type	Description	Default
aggfunc	str	Aggregation function for duplicate timestamps ('first', 'mean', 'last').	'first'

Returns:

Type	Description
DataFrame	DataFrame with systime as index and one column per UUID. A signal without any value is kept as an all-NaN column.

resample_to_uniform

resample_to_uniform(
    window: str,
    method: str = "linear",
    fill_limit: int | None = None,
) -> pd.DataFrame

Resample to a uniform time grid with interpolation.

Pivots first, so the full wide frame of :meth:pivot_to_wide (one row per distinct timestamp, see its memory note) is built before it is reduced to the grid.

Parameters:

Name	Type	Description	Default
window	str	Pandas frequency string (e.g. '1s', '100ms', '1min').	required
method	str	Interpolation method ('linear', 'time', 'nearest', 'quadratic', 'cubic').	'linear'
fill_limit	int | None	Maximum number of consecutive NaNs to fill.	None

Returns:

Type	Description
DataFrame	DataFrame with uniform DatetimeIndex.

detect_gaps

detect_gaps(threshold: str = '10s') -> pd.DataFrame

Identify time gaps per UUID exceeding the threshold.

Parameters:

Name	Type	Description	Default
threshold	str	Minimum gap duration as a pandas Timedelta string.	'10s'

Returns:

Type	Description
DataFrame	DataFrame with one row per gap: uuid, gap_start, gap_end, gap_duration (Timedelta) and the canonical start, end, duration_seconds (the same gap). Empty with these columns, typed, when there is no gap.

fill_gaps

fill_gaps(
    strategy: str = "interpolate",
    max_gap: str | None = None,
    fill_value: float | None = None,
) -> pd.DataFrame

Fill gaps in the wide-format data using the specified strategy.

Parameters:

Name	Type	Description	Default
strategy	str	One of 'interpolate', 'ffill', 'bfill', 'constant'.	'interpolate'
max_gap	str | None	Maximum gap size to fill (pandas Timedelta string), measured from the valid sample before a run of missing values to the valid sample after it. Larger gaps remain NaN. None (default) fills every gap the strategy can fill, however long.	None
fill_value	float | None	Value used when strategy='constant'.	None

Returns:

Type	Description
DataFrame	Wide-format DataFrame with gaps filled.

align_asof

align_asof(
    left_uuid: str,
    right_uuid: str,
    tolerance: str = "1s",
    direction: str = "nearest",
) -> pd.DataFrame

Align two UUID signals using merge_asof with configurable tolerance.

Parameters:

Name	Type	Description	Default
left_uuid	str	UUID of the left (reference) signal.	required
right_uuid	str	UUID of the right signal.	required
tolerance	str	Maximum time difference for matching.	'1s'
direction	str	One of 'nearest', 'backward', 'forward'.	'nearest'

Returns:

Type	Description
DataFrame	DataFrame with systime, value_left, value_right.

Warns:

Type	Description
DataQualityWarning	When a uuid is not in the (non-empty) frame; the result then has no matches for it.

merge_multi_signals

merge_multi_signals(
    uuids: list[str] | None = None,
    window: str | None = None,
    method: str = "linear",
) -> pd.DataFrame

End-to-end harmonization: pivot, filter, resample, interpolate.

The selected signals are pivoted before resampling: the memory of :meth:pivot_to_wide applies to them (pass uuids to bound it).

Parameters:

Name	Type	Description	Default
uuids	list[str] | None	Optional list of UUIDs to include. None means all.	None
window	str | None	Optional resample frequency. None means no resampling.	None
method	str	Interpolation method for resampling.	'linear'

Returns:

Type	Description
DataFrame	Wide-format DataFrame ready for cross-signal analytics.