cryo

❄️ cryo ❄️

cryo is the easiest way to extract blockchain data to parquet, csv, json, or a python dataframe.

cryo is also extremely flexible, with many different options to control how data is extracted + filtered + formatted

cryo is an early WIP, please report bugs + feedback to the issue tracker

note that cryo's default settings will slam a node too hard for use with 3rd party RPC providers. Instead, --requests-per-second and --max-concurrent-requests should be used to impose ratelimits. Such settings will be handled automatically in a future release.

to discuss cryo, check out the telegram group

Contents

1. Example Usage
2. Installation
3. Data Schema
4. Code Guide
5. Documentation
   1. Basics
   2. Syntax
   3. Datasets

Example Usage

use as cryo <dataset> [OPTIONS]

Example	Command
Extract all logs from block 16,000,000 to block 17,000,000	cryo logs -b 16M:17M
Extract blocks, logs, or traces missing from current directory	cryo blocks txs traces
Extract to csv instead of parquet	cryo blocks txs traces --csv
Extract only certain columns	cryo blocks --include number timestamp
Dry run to view output schemas or expected work	cryo storage_diffs --dry
Extract all USDC events	cryo logs --contract 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48

For a more complex example, see the Uniswap Example.

cryo uses ETH_RPC_URL env var as the data source unless --rpc <url> is given

Installation

The simplest way to use cryo is as a cli tool:

Method 1: install from source

git clone https://github.com/paradigmxyz/cryo
cd cryo
cargo install --path ./crates/cli

This method requires having rust installed. See rustup for instructions.

Method 2: install from crates.io

cargo install cryo_cli

This method requires having rust installed. See rustup for instructions.

Make sure that ~/.cargo/bin is on your PATH. One way to do this is by adding the line export PATH="$HOME/.cargo/bin:$PATH" to your ~/.bashrc or ~/.profile.

Python Installation

cryo can also be installed as a python package:

Installing cryo python from pypi

(make sure rust is installed first, see rustup)

pip install maturin
pip install cryo

Installing cryo python from source

pip install maturin
git clone https://github.com/paradigmxyz/cryo
cd cryo/crates/python
maturin build --release
pip install --force-reinstall <OUTPUT_OF_MATURIN_BUILD>.whl

Data Schemas

Many cryo cli options will affect output schemas by adding/removing columns or changing column datatypes.

cryo will always print out data schemas before collecting any data. To view these schemas without collecting data, use --dry to perform a dry run.

Schema Design Guide

An attempt is made to ensure that the dataset schemas conform to a common set of design guidelines:

• By default, rows should contain enough information in their columns to be order-able (unless the rows do not have an intrinsic order).
• Columns should usually be named by their JSON-RPC or ethers.rs defaults, except in cases where a much more explicit name is available.
• To make joins across tables easier, a given piece of information should use the same datatype and column name across tables when possible.
• Large ints such as u256 should allow multiple conversions. A value column of type u256 should allow: value_binary, value_string, value_f32, value_f64, value_u32, value_u64, and value_d128. These types can be specified at runtime using the --u256-types argument.
• By default, columns related to non-identifying cryptographic signatures are omitted by default. For example, state_root of a block or v/r/s of a transaction.
• Integer values that can never be negative should be stored as unsigned integers.
• Every table should allow a chain_id column so that data from multiple chains can be easily stored in the same table.

Standard types across tables:

• block_number: u32
• transaction_index: u32
• nonce: u32
• gas_used: u64
• gas_limit: u64
• chain_id: u64
• timestamp: u32

JSON-RPC

cryo currently obtains all of its data using the JSON-RPC protocol standard.

dataset	blocks per request	results per block	method
Blocks	1	1	eth_getBlockByNumber
Transactions	1	multiple	eth_getBlockByNumber, eth_getBlockReceipts, eth_getTransactionReceipt
Logs	multiple	multiple	eth_getLogs
Contracts	1	multiple	trace_block
Traces	1	multiple	trace_block
State Diffs	1	multiple	trace_replayBlockTransactions
Vm Traces	1	multiple	trace_replayBlockTransactions

cryo use ethers.rs to perform JSON-RPC requests, so it can be used any chain that ethers-rs is compatible with. This includes Ethereum, Optimism, Arbitrum, Polygon, BNB, and Avalanche.

A future version of cryo will be able to bypass JSON-RPC and query node data directly.

Code Guide

• Code is arranged into the following crates:
  • cryo_cli: convert textual data into cryo function calls
  • cryo_freeze: core cryo code
  • cryo_python: cryo python adapter
  • cryo_to_df: procedural macro for generating dataset definitions
• Do not use panics (including panic!, todo!, unwrap(), and expect()) except in the following circumstances: tests, build scripts, lazy static blocks, and procedural macros

Documentation

1. cryo help
2. cryo syntax
3. cryo datasets

cryo help

(output of cryo help)

…

cryo syntax

(output of cryo help syntax)

…

cryo datasets

(output of cryo help datasets)

…