parquet-wasm

WASM Parquet

WebAssembly bindings to read and write the Apache Parquet format to and from Apache Arrow using the Rust parquet and arrow crates.

This is designed to be used alongside a JavaScript Arrow implementation, such as the canonical JS Arrow library.

Including read and write support and all compression codecs, the brotli-compressed WASM bundle is 1.2 MB. Refer to custom builds for how to build a smaller bundle. A minimal read-only bundle without compression support can be as small as 456 KB brotli-compressed.

Install

parquet-wasm is published to NPM. Install with

yarn add parquet-wasm

or

npm install parquet-wasm

API

Parquet-wasm has both a synchronous and asynchronous API. The sync API is simpler but requires fetching the entire Parquet buffer in advance, which is often prohibitive.

Sync API

Refer to these functions:

• readParquet: Read a Parquet file synchronously.
• readSchema: Read an Arrow schema from a Parquet file synchronously.
• writeParquet: Write a Parquet file synchronously.

Async API

• readParquetStream: Create a ReadableStream that emits Arrow RecordBatches from a Parquet file.
• ParquetFile: A class for reading portions of a remote Parquet file. Use fromUrl to construct from a remote URL or fromFile to construct from a File handle. Note that when you're done using this class, you'll need to call free to release any memory held by the ParquetFile instance itself.

Both sync and async functions return or accept a Table class, an Arrow table in WebAssembly memory. Refer to its documentation for moving data into/out of WebAssembly.

Entry Points

ESM

The esm entry point is the primary entry point. It is the default export from parquet-wasm, and is also accessible at parquet-wasm/esm and parquet-wasm/esm/parquet_wasm.js (for symmetric imports directly from a browser).

Note that when using the esm bundles, you must manually initialize the WebAssembly module before using any APIs. Otherwise, you'll get an error TypeError: Cannot read properties of undefined. There are multiple ways to initialize the WebAssembly code:

Asynchronous initialization

The primary way to initialize is by awaiting the default export.

import wasmInit, {readParquet} from "parquet-wasm";

await wasmInit();

Without any parameter, this will try to fetch a file named 'parquet_wasm_bg.wasm' at the same location as parquet-wasm. (E.g. this snippet input = new URL('parquet_wasm_bg.wasm', import.meta.url);).

Note that you can also pass in a custom URL if you want to host the .wasm file on your own servers.

import wasmInit, {readParquet} from "parquet-wasm";

// Update this version to match the version you're using.
const wasmUrl = "https://cdn.jsdelivr.net/npm/[email protected]/esm/parquet_wasm_bg.wasm";
await wasmInit(wasmUrl);

Synchronous initialization

The initSync named export allows for

import {initSync, readParquet} from "parquet-wasm";

// The contents of esm/parquet_wasm_bg.wasm in an ArrayBuffer
const wasmBuffer = new ArrayBuffer(...);

// Initialize the Wasm synchronously
initSync(wasmBuffer)

Async initialization should be preferred over downloading the Wasm buffer and then initializing it synchronously, as WebAssembly.instantiateStreaming is the most efficient way to both download and initialize Wasm code.

Bundler

The bundler entry point doesn't require manual initialization of the WebAssembly blob, but needs setup with whatever bundler you're using. Refer to the Rust Wasm documentation for more info.

Node

The node entry point can be loaded synchronously from Node.

const {readParquet} = require("parquet-wasm");

const wasmTable = readParquet(...);

Using directly from a browser

You can load the esm/parquet_wasm.js file directly from a CDN

const parquet = await import(
  "https://cdn.jsdelivr.net/npm/[email protected]/esm/+esm"
)
await parquet.default();

const wasmTable = parquet.readParquet(...);

This specific endpoint will minify the ESM before you receive it.

Debug functions

These functions are not present in normal builds to cut down on bundle size. To create a custom build, see Custom Builds below.

setPanicHook

setPanicHook(): void

Sets console_error_panic_hook in Rust, which provides better debugging of panics by having more informative console.error messages. Initialize this first if you're getting errors such as RuntimeError: Unreachable executed.

The WASM bundle must be compiled with the console_error_panic_hook feature for this function to exist.

Example

…

Published examples

(These may use older versions of the library with a different API).

• GeoParquet on the Web (Observable)
• Hello, Parquet-WASM (Observable)

Comparison to hyparquet

hyparquet is another Parquet reader for JavaScript. That project is written in pure JavaScript and has subtly different goals and comparing it to parquet-wasm provides benefits and costs.

tl;dr: if you can use Arrow in your app, use parquet-wasm for better performance and memory usage. Otherwise, or if you are latency-focused, hyparquet could be better for you.

Advantages of hyparquet:

• Smaller bundle size
• Pure JS, so easier to debug
• No WebAssembly, so no initialization step
• No WebAssembly, so no separate memory space.

Advantages of parquet-wasm:

• Faster for large files, as it uses a very high-performance Rust Parquet library compiled to WebAssembly
• Faster and more memory efficient because it loads data into Apache Arrow, a high-performance binary memory format. In comparison, hyparquet loads data to JS objects, which are much less memory efficient than Arrow buffers. This is especially true for large files.
• Even though parquet-wasm has a larger bundle size, the bandwidth savings of loading large amounts of Parquet can quickly make up for that overhead.

Conclusion

• If you only need to load the metadata of Parquet files, or if you have very small Parquet files, using hyparquet could be a good choice as hyparquet is smaller, and thus the overhead before loading the file could be smaller.

• If you need the absolute smallest bundle size, hyparquet may be better for your use case.

• Otherwise, since parquet-wasm:

  1. Uses a really high performance Rust library
  2. Is running in WebAssembly, and
  3. Converts to a high-performance binary memory format

  If you have large files and can use the resulting Arrow data directly without converting to JS objects, parquet-wasm should be significantly faster and more memory efficient.

Feel free to open an issue to discuss more!

Performance considerations

Tl;dr: When you have a Table object (resulting from readParquet), try the new Table.intoFFI API to move it to JavaScript memory. This API is less well tested than the Table.intoIPCStream API, but should be faster and have much less memory overhead (by a factor of 2). If you hit any bugs, please create a reproducible issue.

Under the hood, parquet-wasm first decodes a Parquet file into Arrow in WebAssembly memory. But then that WebAssembly memory needs to be copied into JavaScript for use by Arrow JS. The "normal" conversion APIs (e.g. Table.intoIPCStream) use the Arrow IPC format to get the data back to JavaScript. But this requires another memory copy inside WebAssembly to assemble the various arrays into a single buffer to be copied back to JS.

Instead, the new Table.intoFFI API uses Arrow's C Data Interface to be able to copy or view Arrow arrays from within WebAssembly memory without any serialization.

Note that this approach uses the arrow-js-ffi library to parse the Arrow C Data Interface definitions. This library has not yet been tested in production, so it may have bugs!

I wrote an interactive blog post on this approach and the Arrow C Data Interface if you want to read more!

Example

…

Compression support

The Parquet specification permits several compression codecs. This library currently supports:

• Uncompressed
• Snappy
• Gzip
• Brotli
• ZSTD
• LZ4_RAW
• LZ4 (deprecated)

LZ4 support in Parquet is a bit messy. As described here, there are two LZ4 compression options in Parquet (as of version 2.9.0). The original version LZ4 is now deprecated; it used an undocumented framing scheme which made interoperability difficult. The specification now reads:

It is strongly suggested that implementors of Parquet writers deprecate this compression codec in their user-facing APIs, and advise users to switch to the newer, interoperable LZ4_RAW codec.

It's currently unknown how wide