irx.parquet_options

irx.parquet_options

Source: packages/irx/src/irx/parquet_options.py

title: Typed configuration for native Parquet sessions.

Functions

option_text(value: str) -> bytes

title: Reject NUL before passing option text across a C-string boundary.
parameters:
  value:
    type: str
returns:
  type: bytes

option_integer(value: int) -> None

title: Reject bool and integer truncation at the native ABI boundary.
parameters:
  value:
    type: int

validate_option_items(columns: tuple[str, ...], partition_columns: tuple[str, ...], row_groups: tuple[int, ...], predicates: tuple[ParquetPredicate, ...], encryption_key: bytes | None) -> None

title: Runtime-check every item of options collections via the IRx wrapper.
parameters:
  columns:
    type: tuple[str, Ellipsis]
  partition_columns:
    type: tuple[str, Ellipsis]
  row_groups:
    type: tuple[int, Ellipsis]
  predicates:
    type: tuple[ParquetPredicate, Ellipsis]
  encryption_key:
    type: bytes | None

validate_predicate(column: str, operation: Literal['eq', 'ne', 'lt', 'le', 'gt', 'ge', 'is_null', 'is_valid'], value: str | int | float | bool) -> None

title: Runtime-check all predicate fields with the owning package wrapper.
parameters:
  column:
    type: str
  operation:
    type: Literal[eq, ne, lt, le, gt, ge, is_null, is_valid]
  value:
    type: str | int | float | bool

Classes

ParquetPredicate

title: One typed comparison; multiple predicates are combined with AND.
attributes:
  column:
    type: str
  operation:
    type: Literal[eq, ne, lt, le, gt, ge, is_null, is_valid]
  value:
    type: str | int | float | bool

ParquetOptions

title: Options copied into a native Parquet session at open time.
summary: >-
  Empty columns/row_groups select all. Dataset evolution unions same-typed
  fields only. Atomic replacement and rewrite append are POSIX-local
  operations with advisory writer exclusion and optional fsync durability.
  Reconstruct preserves supported logical types over explicit physical
  storage for single files. Raw AES keys are excluded from repr; callers
  own key distribution and lifetime.
attributes:
  batch_size:
    type: int
  row_group_size:
    type: int
  data_page_size:
    type: int
  compression:
    type: str
  compression_level:
    type: int | None
  columns:
    type: tuple[str, Ellipsis]
  partition_columns:
    type: tuple[str, Ellipsis]
  row_groups:
    type: tuple[int, Ellipsis]
  predicates:
    type: tuple[ParquetPredicate, Ellipsis]
  use_threads:
    type: bool
  pre_buffer:
    type: bool
  dataset:
    type: bool
  hive:
    type: bool
  unify:
    type: bool
  normalize:
    type: bool
  atomic:
    type: bool
  durable:
    type: bool
  reconstruct:
    type: bool
  append:
    type: bool
  encryption_key:
    type: bytes | None

Methods

native(self, lib: ctypes.CDLL) -> Iterator[ctypes.c_void_p]
title: Create and release a temporary checked native options owner.
parameters:
  lib:
    type: ctypes.CDLL
returns:
  type: Iterator[ctypes.c_void_p]
Back to top