Parquet

2026-03-11

27 min read time

Applies to Linux

class pylibhipdf.io.parquet.ChunkedParquetReader(ParquetReaderOptions options, Stream stream=None, size_t chunk_read_limit=0, size_t pass_read_limit=1024000000)

Bases: object

Reads chunks of a Parquet file into a TableWithMetadata.

For details, see chunked_parquet_reader.

Parameters

optionsParquetReaderOptions

Settings for controlling reading behavior

chunk_read_limitsize_t, default 0

Limit on total number of bytes to be returned per read, or 0 if there is no limit.

pass_read_limitsize_t, default 1024000000

Limit on the amount of memory used for reading and decompressing data or 0 if there is no limit.

__init__(*args, **kwargs)

has_next(self) → bool

Returns True if there is another chunk in the Parquet file to be read.

Returns

True if we have not finished reading the file.

read_chunk(self) → TableWithMetadata

Read the next chunk into a TableWithMetadata

Returns

TableWithMetadata

The Table and its corresponding metadata (column names) that were read in.

stream

class pylibhipdf.io.parquet.ChunkedParquetWriterOptions

Bases: object

static builder(SinkInfo sink)

Create builder to create ChunkedParquetWriterOptions.

Parameters

sink: SinkInfo

The sink used for writer output

Returns

ChunkedParquetWriterOptionsBuilder

set_dictionary_policy(self, dictionary_policy policy) → void

Sets the policy for dictionary use.

Parameters

policyDictionaryPolicy

Policy for dictionary use

Returns

None

class pylibhipdf.io.parquet.ChunkedParquetWriterOptionsBuilder

Bases: object

build(self) → ChunkedParquetWriterOptions

Create a ChunkedParquetWriterOptions object

compression(self, compression_type compression) → ChunkedParquetWriterOptionsBuilder

Sets compression type.

Parameters

compressionCompressionType

The compression type to use

Returns

Self

key_value_metadata(self, list metadata) → ChunkedParquetWriterOptionsBuilder

Sets Key-Value footer metadata.

Parameters

metadatalist[dict[str, str]]

Key-Value footer metadata

Returns

Self

max_dictionary_size(self, size_t val) → ChunkedParquetWriterOptionsBuilder

Sets the maximum dictionary size, in bytes.

Parameters

valsize_t

Sets the maximum dictionary size, in bytes.

Returns

Self

max_page_size_bytes(self, size_t val) → ChunkedParquetWriterOptionsBuilder

Sets the maximum uncompressed page size, in bytes.

Parameters

valsize_t

Maximum uncompressed page size, in bytes to set

Returns

Self

max_page_size_rows(self, size_type val) → ChunkedParquetWriterOptionsBuilder

Sets the maximum page size, in rows.

Parameters

valsize_type

Maximum page size, in rows to set.

Returns

Self

metadata(self, TableInputMetadata metadata) → ChunkedParquetWriterOptionsBuilder

row_group_size_bytes(self, size_t val) → ChunkedParquetWriterOptionsBuilder

Sets the maximum row group size, in bytes.

Parameters

valsize_t

Maximum row group size, in bytes to set

Returns

Self

row_group_size_rows(self, size_type val) → ChunkedParquetWriterOptionsBuilder

Sets the maximum row group size, in rows.

Parameters

valsize_type

Maximum row group size, in rows to set

Returns

Self

stats_level(self, statistics_freq sf) → ChunkedParquetWriterOptionsBuilder

Sets the level of statistics.

Parameters

sfStatisticsFreq

Level of statistics requested in the output file

Returns

Self

write_arrow_schema(self, bool enabled) → ChunkedParquetWriterOptionsBuilder

Set to true if arrow schema is to be written.

Parameters

enabledbool

Boolean value to enable/disable writing of arrow schema.

Returns

Self

class pylibhipdf.io.parquet.ParquetReaderOptions

Bases: object

The settings to use for read_parquet For details, see cudf::io::parquet_reader_options

static builder(SourceInfo source)

Create a ParquetReaderOptionsBuilder object

For details, see cudf::io::parquet_reader_options::builder()

Parameters

sinkSourceInfo

The source to read the Parquet file from.

Returns

ParquetReaderOptionsBuilder

Builder to build ParquetReaderOptions

set_columns(self, list col_names) → void

Sets names of the columns to be read.

Parameters

col_nameslist

List of column names

Returns

None

set_filter(self, Expression filter) → void

Sets AST based filter for predicate pushdown.

Parameters

filterExpression

AST expression to use as filter

Returns

None

set_num_rows(self, size_type nrows) → void

Sets number of rows to read.

Parameters

nrowssize_type

Number of rows to read after skip

Returns

None

set_row_groups(self, list row_groups) → void

Sets list of individual row groups to read.

Parameters

row_groupslist

List of row groups to read

Returns

None

set_skip_rows(self, int64_t skip_rows) → void

Sets number of rows to skip.

Parameters

skip_rowsint64_t

Number of rows to skip from start

Returns

None

class pylibhipdf.io.parquet.ParquetReaderOptionsBuilder

Bases: object

allow_mismatched_pq_schemas(self, bool val) → ParquetReaderOptionsBuilder

Sets to enable/disable reading of matching projected and filter columns from mismatched Parquet sources.

Parameters

valbool

Boolean value whether to read matching projected and filter columns from mismatched Parquet sources.

Returns

ParquetReaderOptionsBuilder

build(self)

Create a ParquetReaderOptions object

columns(self, list col_names) → ParquetReaderOptionsBuilder

Sets names of the columns to be read.

Parameters

col_nameslist[str]

List of column names

Returns

ParquetReaderOptionsBuilder

convert_strings_to_categories(self, bool val) → ParquetReaderOptionsBuilder

Sets enable/disable conversion of strings to categories.

Parameters

valbool

Boolean value to enable/disable conversion of string columns to categories

Returns

ParquetReaderOptionsBuilder

filter(self, Expression filter) → ParquetReaderOptionsBuilder

Sets AST based filter for predicate pushdown.

Parameters

filterExpression

AST expression to use as filter

Returns

ParquetReaderOptionsBuilder

use_arrow_schema(self, bool val) → ParquetReaderOptionsBuilder

Sets to enable/disable use of arrow schema to read.

Parameters

valbool

Boolean value whether to use arrow schema

Returns

ParquetReaderOptionsBuilder

use_pandas_metadata(self, bool val) → ParquetReaderOptionsBuilder

Sets to enable/disable use of pandas metadata to read.

Parameters

valbool

Boolean value whether to use pandas metadata

Returns

ParquetReaderOptionsBuilder

class pylibhipdf.io.parquet.ParquetWriterOptions

Bases: object

static builder(SinkInfo sink, Table table)

Create builder to create ParquetWriterOptionsBuilder.

Parameters

sinkSinkInfo

The sink used for writer output

tableTable

Table to be written to output

Returns

ParquetWriterOptionsBuilder

set_column_chunks_file_paths(self, list file_paths) → void

Sets column chunks file path to be set in the raw output metadata.

Parameters

file_pathslist[str]

Vector of strings which indicate file paths.

Returns

None

set_max_dictionary_size(self, size_t size_bytes) → void

Sets the maximum dictionary size, in bytes.

Parameters

size_bytesint

Sets the maximum dictionary size, in bytes.

Returns

None

set_max_page_size_bytes(self, size_t size_bytes) → void

Sets the maximum uncompressed page size, in bytes.

Parameters

size_bytesint

Maximum uncompressed page size, in bytes to set

Returns

None

set_max_page_size_rows(self, size_type size_rows) → void

Sets the maximum page size, in rows.

Parameters

size_rowsint

Maximum page size, in rows to set.

Returns

None

set_partitions(self, list partitions) → void

Sets partitions.

Parameters

partitionslist[Partitions]

Partitions of input table in {start_row, num_rows} pairs.

Returns

None

set_row_group_size_bytes(self, size_t size_bytes) → void

Sets the maximum row group size, in bytes.

Parameters

size_bytesint

Maximum row group size, in bytes to set

Returns

None

set_row_group_size_rows(self, size_type size_rows) → void

Sets the maximum row group size, in rows.

Parameters

size_rowsint

Maximum row group size, in rows to set

Returns

None

class pylibhipdf.io.parquet.ParquetWriterOptionsBuilder

Bases: object

build(self) → ParquetWriterOptions

Create a ParquetWriterOptions from the set options.

Returns

ParquetWriterOptions

compression(self, compression_type compression) → ParquetWriterOptionsBuilder

Sets compression type.

Parameters

compressionCompressionType

The compression type to use

Returns

Self

dictionary_policy(self, dictionary_policy val) → ParquetWriterOptionsBuilder

Sets the policy for dictionary use.

Parameters

valDictionaryPolicy

Policy for dictionary use.

Returns

Self

int96_timestamps(self, bool enabled) → ParquetWriterOptionsBuilder

Sets whether timestamps are written as int96 or timestamp micros.

Parameters

enabledbool

Boolean value to enable/disable int96 timestamps

Returns

Self

key_value_metadata(self, list metadata) → ParquetWriterOptionsBuilder

Sets Key-Value footer metadata.

Parameters

metadatalist[dict[str, str]]

Key-Value footer metadata

Returns

Self

max_page_size_bytes(self, size_t val) → ParquetWriterOptionsBuilder

Sets the maximum uncompressed page size, in bytes.

Parameters

valsize_t

Maximum uncompressed page size, in bytes to set

Returns

Self

metadata(self, TableInputMetadata metadata) → ParquetWriterOptionsBuilder

Sets metadata.

Parameters

metadataTableInputMetadata

Associated metadata

Returns

Self

row_group_size_rows(self, size_type val) → ParquetWriterOptionsBuilder

Sets the maximum row group size, in rows.

Parameters

valsize_type

Maximum row group size, in rows to set

Returns

Self

stats_level(self, statistics_freq sf) → ParquetWriterOptionsBuilder

Sets the level of statistics.

Parameters

sfStatisticsFreq

Level of statistics requested in the output file

Returns

Self

utc_timestamps(self, bool enabled) → ParquetWriterOptionsBuilder

Set to true if timestamps are to be written as UTC.

Parameters

enabledbool

Boolean value to enable/disable writing of timestamps as UTC.

Returns

Self

write_arrow_schema(self, bool enabled) → ParquetWriterOptionsBuilder

Set to true if arrow schema is to be written.

Parameters

enabledbool

Boolean value to enable/disable writing of arrow schema.

Returns

Self

write_v2_headers(self, bool enabled) → ParquetWriterOptionsBuilder

Set to true to write V2 page headers, otherwise false to write V1 page headers.

Parameters

enabledbool

Boolean value to enable/disable writing of V2 page headers.

Returns

Self

pylibhipdf.io.parquet.is_supported_write_parquet(compression_type compression) → bool

Check if the compression type is supported for writing Parquet files.

For details, see is_supported_write_parquet().

pylibhipdf.io.parquet.merge_row_group_metadata(list metdata_list) → memoryview

Merges multiple raw metadata blobs that were previously created by write_parquet into a single metadata blob.

For details, see merge_row_group_metadata().

Parameters

metdata_listlist

List of input file metadata

Returns

memoryview

A parquet-compatible blob that contains the data for all row groups in the list

pylibhipdf.io.parquet.read_parquet(ParquetReaderOptions options, Stream stream=None)

Read from Parquet format.

The source to read from and options are encapsulated by the options object.

For details, see read_parquet().

Parameters

options: ParquetReaderOptions

Settings for controlling reading behavior

streamStream | None

CUDA stream used for device memory operations and kernel launches

pylibhipdf.io.parquet.write_parquet(ParquetWriterOptions options, Stream stream=None) → memoryview

Writes a set of columns to parquet format.

Parameters

optionsParquetWriterOptions

Settings for controlling writing behavior

streamStream | None

CUDA stream used for device memory operations and kernel launches

Returns

memoryview

A blob that contains the file metadata (parquet FileMetadata thrift message) if requested in parquet_writer_options (empty blob otherwise).