s3proxy_parquet_loader

s3proxy_parquet_loader ¤

S3ProxyDataAccess ¤

S3ProxyDataAccess(
    start: str,
    end: str,
    uuids: list[str],
    s3_config: dict[str, str],
)

A class to access timeseries data via an S3 proxy. This class retrieves data for specified UUIDs within a defined time range, with the option to output data as Parquet files or as a single combined DataFrame.

Initialize the S3ProxyDataAccess object.

Parameters:

Name	Type	Description	Default
`start`	`str`	Start timestamp in "Year-Month-Day Hour:Minute:Second" format.	required
`end`	`str`	End timestamp in "Year-Month-Day Hour:Minute:Second" format.	required
`uuids`	`list[str]`	List of UUIDs to retrieve data for.	required
`s3_config`	`dict[str, str]`	Configuration dictionary for the S3 connection. Must contain `endpoint_url`, `key`, `secret`, `use_ssl`, `version_aware` and `s3_path_base`.	required

fetch_data_as_parquet ¤

fetch_data_as_parquet(output_dir: str) -> None

Retrieve timeseries data from S3 and save it as Parquet files.

Each file is saved as <output_dir>/year/month/day/hour/<uuid>.parquet.

Parameters:

Name	Type	Description	Default
`output_dir`	`str`	Base directory to save the Parquet files.	required

fetch_data_as_dataframe ¤

fetch_data_as_dataframe() -> pd.DataFrame

Retrieve timeseries data from S3 and return it as a single DataFrame.

Returns:

Type	Description
`DataFrame`	A combined DataFrame with data for all specified UUIDs and time slots.