_utils

_utils

Shared helpers for ts-shape loaders.

Loaders pull data from local disk, object stores (S3/Azure), databases and REST APIs. The two cross-cutting concerns are:

• input validation -- fail fast with a clear :class:~ts_shape.errors.LoaderError instead of silently returning an empty DataFrame when a source is missing or misconfigured.
• transient-failure handling -- network reads against object stores and APIs occasionally fail; :func:retry_call retries them with exponential backoff before giving up.

These helpers are intentionally backend-agnostic so every loader can reuse the same behaviour and error type.

validate_local_path

validate_local_path(
    path: str | Path, *, must_be_dir: bool = False
) -> Path

Validate that a local filesystem path exists.

Parameters:

Name	Type	Description	Default
path	str | Path	The path to check.	required
must_be_dir	bool	When True, also require the path to be a directory.	False

Returns:

Type	Description
Path	The path as a :class:pathlib.Path.

Raises:

Type	Description
LoaderError	If the path does not exist, or must_be_dir is set and the path is not a directory.

require_config

require_config(
    config: dict, keys: list[str], *, name: str = "config"
) -> None

Validate that config contains every key in keys.

Parameters:

Name	Type	Description	Default
config	dict	The configuration mapping to check.	required
keys	list[str]	Required keys.	required
name	str	Label used in the error message (e.g. "s3_config").	'config'

Raises:

Type	Description
LoaderError	If any key is missing, listing all of the missing keys.

retry_call

retry_call(
    func: Callable[[], T],
    *,
    attempts: int = 3,
    initial_delay: float = 0.5,
    backoff: float = 2.0,
    retry_exceptions: tuple[type[BaseException], ...] = (
        OSError,
    ),
    exclude: tuple[type[BaseException], ...] = (),
    description: str = "loader call",
    sleep: Callable[[float], None] = time.sleep
) -> T

Call func and retry on transient failures with exponential backoff.

Parameters:

Name	Type	Description	Default
func	Callable[[], T]	A zero-argument callable performing the I/O.	required
attempts	int	Maximum number of attempts (>= 1).	3
initial_delay	float	Seconds to wait before the second attempt.	0.5
backoff	float	Multiplier applied to the delay after each failed attempt.	2.0
retry_exceptions	tuple[type[BaseException], ...]	Exception types that trigger a retry.	(OSError,)
exclude	tuple[type[BaseException], ...]	Exception types that are re-raised immediately even when they are a subclass of retry_exceptions (e.g. FileNotFoundError, which means "no data" rather than a transient fault).	()
description	str	Human-readable label used in log messages.	'loader call'
sleep	Callable[[float], None]	Sleep function; injectable so tests run without real delays.	sleep

Returns:

Type	Description
T	Whatever func returns on the first successful attempt.

Raises:

Type	Description
ValueError	If attempts is less than 1.
Exception	Re-raises the last exception once all attempts are exhausted, or immediately for any exception listed in exclude.