百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
P

parquet2

> 编程语言
开源

最快、最安全的 Rust 实现的 Parquet。无 `unsafe` 限制。已与 pyarrow 进行集成测试

385 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

最快、最安全的 Rust 实现的 Parquet。无 `unsafe` 限制。已与 pyarrow 进行集成测试

Parquet2

This is a re-write of the official parquet crate with performance, parallelism and safety in mind.

Checkout the guide for details on how to use this crate to read parquet.

The five main differentiators in comparison with parquet are:

  • it uses #![forbid(unsafe_code)]
  • delegates parallelism downstream
  • decouples reading (IO intensive) from computing (CPU intensive)
  • it is faster (10-20x when reading to arrow format)
  • supports async read and write.
  • It is integration-tested against pyarrow and (py)spark 3

The overall idea is to offer the ability to read compressed parquet pages and a toolkit to decompress them to their favourite in-memory format.

This allows this crate's iterators to perform minimal CPU work, thereby maximizing throughput. It is up to the consumers to decide whether they want to take advantage of this through parallelism at the expense of memory usage (e.g. decompress and deserialize pages in threads) or not.

This crate cannot be used directly to read parquet (except metadata). To read data from parquet, checkout arrow2.

Functionality implemented

  • Read dictionary pages
  • Read and write V1 pages
  • Read and write V2 pages
  • Compression and de-compression (all)

Functionality not (yet) implemented

  • Index pages
  • Bit-packed (Deprecated)
  • Byte Stream Split

The parquet format has multiple encoding strategies for the different physical types. This crate currently reads from almost all of them, and supports encoding to a subset of them. They are:

Supported decoding

  • PLAIN
  • RLE dictionary
  • RLE hybrid
  • Delta-encoding
  • Delta length byte array
  • Delta strings

Delta-encodings are still experimental, as I have been unable to generate large pages encoded with them from spark, thereby hindering robust integration tests.

Encoding

  • PLAIN
  • RLE hybrid
  • Delta-encoding
  • Delta length byte array

Organization

  • read: read metadata and pages
  • write: write metadata and pages
  • encoding: encoders and decoders of the different parquet encodings
  • page: page declarations
  • metadata: parquet files metadata (e.g. FileMetaData)
  • schema: types metadata declaration (e.g. ConvertedType)
  • types.rs: physical type declaration (i.e. how things are represented in memory).
  • statistics: deserialized representation of a parquet page
  • compression: compressors and decompressors compression (e.g. Gzip)
  • error: errors declaration

Run integration tests

There are integration tests against parquet files generated by pyarrow. To run then, you will need to run

bash
python3 -m venv venv
venv/bin/pip install pip --upgrade
venv/bin/pip install pyarrow==7
venv/bin/python tests/write_pyarrow.py
cargo test

before. This is only needed once (per change in the tests/write_pyarrow.py).

How to implement page readers

The in-memory format used to consume parquet pages strongly influences how the pages should be deserialized. As such, this crate does not commit to a particular in-memory format. Consumers are responsible for converting pages to their target in-memory format.

This git repository contains a serialization to a simple in-memory format in integration, that is used to validate integration with other implementations.

There is also an implementation for the arrow format here.

Higher Parallelism

Typically, converting a page into memory is expensive and thus consider how to distribute work across threads. E.g.

rust
let handles = vec![];
for column in columns {
    let column_meta = metadata.row_groups[row_group].column(column);
    let compressed_pages = get_page_iterator(column_meta, &mut file, file)?.collect()?;
    // each compressed_page has a buffer; cloning is expensive(!). We move it so that the memory
    // is released at the end of the processing.
    handles.push(thread::spawn move {
        page_iter_to_array(compressed_pages.into_iter())
    })
}
let columns_from_all_groups = handles.join_all();

this will read the file as quickly as possible in the main thread and send CPU-intensive work to other threads, thereby maximizing IO reads (at the cost of storing multiple compressed pages in memory; buffering is also an option here).

Decoding flow

Generally, a parquet file is read as follows:

  1. Read metadata
  2. Seek a row group and column
  3. iterate over (compressed) pages within that (group, column)

This is IO-intensive, requires parsing thrift, and seeking within a file.

Once a compressed page is loaded into memory, it can be decompressed, decoded and deserialized into a specific in-memory format. All of these operations are CPU-intensive and are thus left to consumers to perform, as they may want to send this work to threads.

read -> compressed page -> decompressed page -> decoded bytes -> deserialized

License

Licensed under either of

  • Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
  • MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Rustparallelismparquetrustsafe

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言