hipdf.io.parquet.read_parquet_metadata

19 min read time

Applies to Linux

hipdf.io.parquet.read_parquet_metadata(filepath_or_buffer) → tuple[int, int, list[Hashable], int, list[dict[str, int]]]

Read metadata and schema of a list of Parquet files

Parameters

pathsList of strings or path objects

Path of file(s) to be read

Returns

Total number of rows Total number of row groups List of column names Number of columns List of metadata of row groups

Examples

>>> import cudf
>>> num_rows, num_row_groups, names, num_columns, row_group_metadata = cudf.io.read_parquet_metadata(filename)
>>> df = [cudf.read_parquet(fname, row_group=i) for i in range(num_row_groups)]
>>> df = cudf.concat(df)
>>> df
  num1                datetime text
0  123 2018-11-13T12:00:00.000 5451
1  456 2018-11-14T12:35:01.000 5784
2  789 2018-11-15T18:02:59.000 6117

See Also

cudf.read_parquet