parquet_loader

parquet_loader

ParquetLoader

ParquetLoader(base_path: str)

This class provides class methods to load parquet files from a specified directory structure.

Example

Files are named <uuid>.parquet in YYYY/MM/DD/HH folders:

import tempfile from pathlib import Path import ts_shape from ts_shape import ParquetLoader tmp = tempfile.TemporaryDirectory() folder = Path(tmp.name, "2025", "01", "01", "00") folder.mkdir(parents=True) ts_shape.make_timeseries(["temp"], n_points=3, freq="1min", noise=0).to_parquet( ... folder / "temp.parquet") ParquetLoader.load_by_uuid_list(tmp.name, ["temp"])[ ... ["systime", "uuid", "value_double"]] systime uuid value_double 0 2025-01-01 00:00:00 temp 100.0 1 2025-01-01 00:01:00 temp 100.0 2 2025-01-01 00:02:00 temp 100.0 tmp.cleanup()

Initialize the ParquetLoader with the base directory path.

Parameters:

Name	Type	Description	Default
base_path	str	The base directory where parquet files are stored.	required

Raises:

Type	Description
LoaderError	If base_path does not exist or is not a directory.

load_all_files classmethod

load_all_files(base_path: str) -> pd.DataFrame

Loads all parquet files in the specified base directory into a single pandas DataFrame.

Parameters:

Name	Type	Description	Default
base_path	str	The base directory where parquet files are stored.	required

Returns:

Type	Description
DataFrame	pd.DataFrame: A DataFrame containing all the data from the parquet files.

load_by_time_range classmethod

load_by_time_range(
    base_path: str, start: Timestamp, end: Timestamp
) -> pd.DataFrame

Loads parquet files that fall within a specified time range based on the directory structure.

The directory structure is expected to be in the format YYYY/MM/DD/HH (UTC). The hour folder that contains start is included; tz-aware bounds are converted to UTC, naive ones are taken as UTC.

Parameters:

Name	Type	Description	Default
base_path	str	The base directory where parquet files are stored.	required
start	Timestamp	The start timestamp.	required
end	Timestamp	The end timestamp.	required

Returns:

Type	Description
DataFrame	pd.DataFrame: A DataFrame containing the data from the parquet files within the time range.

load_by_uuid_list classmethod

load_by_uuid_list(
    base_path: str, uuid_list: list
) -> pd.DataFrame

Loads parquet files that match any UUID in the specified list.

A file matches when its name without .parquet equals a UUID (exact match, not a substring).

Parameters:

Name	Type	Description	Default
base_path	str	The base directory where parquet files are stored.	required
uuid_list	list	A list of UUIDs to filter the files.	required

Returns:

Type	Description
DataFrame	pd.DataFrame: A DataFrame containing the data from the parquet files with matching UUIDs.

load_files_by_time_range_and_uuids classmethod

load_files_by_time_range_and_uuids(
    base_path: str,
    start: Timestamp,
    end: Timestamp,
    uuid_list: list,
) -> pd.DataFrame

Loads parquet files that fall within a specified time range and match any UUID in the list.

The directory structure is expected to be in the format YYYY/MM/DD/HH (UTC; the hour folder that contains start is included), and a file matches when its name without .parquet equals a UUID.

Parameters:

Name	Type	Description	Default
base_path	str	The base directory where parquet files are stored.	required
start	Timestamp	The start timestamp.	required
end	Timestamp	The end timestamp.	required
uuid_list	list	A list of UUIDs to filter the files.	required

Returns:

Type	Description
DataFrame	pd.DataFrame: A DataFrame containing the data from the parquet files that meet both criteria.