snarkOS

## Table of Contents * [1. Overview](#1-overview) * [2. Build Guide](#2-build-guide) * [2.1 Requirements](#21-requirements) * [2.2 Installation](#22-installation) * [3. Run an Aleo Node](#3-run-an-aleo-node) * [3.1 Run an Aleo Client](#31-run-an-aleo-client) * [3.2 Run an Aleo Validator](#32-run-an-aleo-validator) * [3.3 Run an Aleo Prover](#33-run-an-aleo-prover) * [4. FAQs](#4-faqs) * [5. Command Line Interface](#5-command-line-interface) * [6. Development Guide](#6-development-guide) * [6.1 Quick Start](#61-quick-start) * [6.2 Operations](#62-operations) * [6.3 Local Devnet](#63-local-devnet) * [6.4 Feature Flags](#64-feature-flags) * [6.5 Local Backups](#65-local-backups) * [7. Contributors](#7-contributors) * [8. License](#8-license) [comment]: <> (* [4. JSON-RPC Interface](#4-json-rpc-interface)) [comment]: <> (* [5. Additional Information](#5-additional-information)) ## 1. Overview __snarkOS__ is a decentralized operating system for zero-knowledge applications. This code forms the backbone of [Aleo](https://aleo.org/) network, which verifies transactions and stores the encrypted state applications in a publicly-verifiable manner. ## 2. Build Guide ### 2.1 Definitions The following snarkOS node types exist in the Aleo network: - **Validator**: Validator nodes participate in consensus and must be started with an account that is bonded into the committee. - **Client**: Clients do not participate in consensus but maintain a ledger. They are capable of providing information about the network as well as accepting solutions and transactions and communicating them to their peers. All clients run the same software, however, for the purposes of configuration management, this document defines two types of clients: - Core Client: Client node connected directly to a validator node. - Outer Client: Client node connected only to other clients or prover nodes. - **Prover**: Prover nodes are dedicated to solving the Aleo puzzle. They do not participate in consensus or maintain a copy of the ledger. ### 2.2 Requirements The following are the requirements to run an Aleo node: - **OS**: 64-bit architectures only, latest up-to-date for security - Clients: Ubuntu 22.04 (LTS), macOS Ventura or later, Windows 11 or later - Validators: Ubuntu 22.04 (LTS) - **CPU**: 64-bit architectures only, Latest Intel Xeon or Better - Clients: 24-cores (32-cores or larger preferred) - Validators: 64-cores (128-cores or larger preferred) - **RAM**: DDR4 or better - Clients: 128GiB of memory (192GiB or larger preferred) - Validators: 256GiB of memory (384GiB or larger preferred) - **Storage**: PCIe Gen 3 x4, PCIe Gen 4 x2 NVME SSD, or better - Clients: 2TB of disk space (4TB or larger preferred) - Validators: 4TB of disk space (6TB or larger preferred) - **Network**: Symmetric, commercial, always-on - Clients: 250Mbps of upload **and** download bandwidth - Validators: 500Mbps of upload **and** download bandwidth No explicit recommendations are made for proving nodes as proving hardware may be highly variable. If interested in running Aleo Provers nodes, please refer to resources published by the Aleo community. ### 2.3 Installation Before beginning, please ensure your machine has Rust installed, with at least [this version](rust-toolchain). Instructions to [install Rust can be found here.](https://www.rust-lang.org/tools/install) Start by cloning this GitHub repository: ``` git clone --branch mainnet --single-branch https://github.com/ProvableHQ/snarkOS.git ``` Next, move into the `snarkOS` directory: ``` cd snarkOS ``` **[For Ubuntu users]** A helper script to install dependencies is available. From the `snarkOS` directory, run: ``` ./build_ubuntu.sh ``` Lastly, install `snarkOS`: ``` cargo install --locked --path . ``` #### Optional: CUDA Acceleration for Provers > This CUDA build is optional. The current snarkOS puzzle does not leverage CUDA acceleration—it is a leftover from a previous event, although CUDA may become relevant again with ARC-43. > > The CUDA feature is considered unstable and experimental; expect breaking changes. For prover operators who want to experiment with GPU support: ``` cargo install --locked --path . --features cuda ``` Please ensure ports `4130/tcp` and `3030/tcp` are open on your router and OS firewall. ### 2.4 Port Configuration #### 2.4.1 For Core Clients | Port | Protocol | Allow/Deny | Source | Explanation | |----------|----------|------------|------------------------------|------------------------------------------------------------| | 4130/tcp | TCP | Allow | All IPv4/IPv6 | TCP traffic to peers | #### 2.4.2 For Outer Clients | Port | Protocol | Allow/Deny | Source | Explanation | |----------|----------|------------|------------------------------|------------------------------------------------------------| | 3030/tcp | TCP | Allow | All IPv4/IPv6 | REST server | | 4130/tcp | TCP | Allow | All IPv4/IPv6 | TCP traffic to peers | #### 2.4.3 For Validators | Port | Protocol | Allow/Deny | Source | Explanation | |----------|----------|------------|------------------------------|------------------------------------------------------------| | 4130/tcp | TCP | Allow | All IPv4/IPv6 | TCP traffic to peers | | 5000/tcp | TCP | Allow | Trusted Validator IPs | TCP traffic between validators for BFT communication | | 3000/tcp | TCP | Allow | Internal VPC or VPN | Metrics dashboard, should only be open within an internal VPC or VPN | | 3030/tcp | TCP | Deny | All IPv4/IPv6 | REST server. This should **always** be disabled for validators | | 9000/tcp | TCP | Allow | Internal VPC or VPN | Metrics export, should only be open within an internal VPC or VPN | | 9090/tcp | TCP | Allow | Internal VPC or VPN | Prometheus metrics, should only be open within an internal VPC or VPN | **Note:** Ensure that your open file limit is set to 16,384 or above. For the recommended setting run: ``` # Increase the open file limit for the current user (replace with your username) echo " - nofile 65536" | sudo tee -a /etc/security/limits.conf # Increase the default system open file limit sudo bash -c 'echo "DefaultLimitNOFILE=65536" >> /etc/systemd/system.conf' ``` ## 3. Run an Aleo Node ## 3.1 Run an Aleo Client Start by following the instructions in the [Build Guide](#2-build-guide). The guide below provides information on running `core` and `outer` clients (as defined in Section 2.2.) Aleo community members running validators are recommended to run 1-3 `core` clients as their exclusive client peers. This will ensure network traffic from the public internet is verified prior to reaching the validator. Any client **not** connected directly to a validator can be considered an `outer` client. ### 3.1.1 Run an Aleo Core Client The following command is recommended when starting a client node that is connected to a validator: `snarkos start --client --nodisplay --node 0.0.0.0:4130 --peers "validator_ip:4130,core_client_ip_1:4130,core_client_ip_2:4130,core_client_ip3:4130,outer_client_ip_1:4130,..." --verbosity 1 --norest` To start a core client node, you can also run the following command from the `snarkOS` directory: ``` ./scripts/run-core-client.sh ``` ### 3.1.2 Run an Aleo Outer Client The following command is recommended when starting a client node that is NOT connected to a validator: `snarkos start --client --nodisplay --node 0.0.0.0:4130 --peers "core_client_ip_1:4130,core_client_ip_2:4130,core_client_ip3:4130,outer_client_ip_1:4130,..." --verbosity 1 --rest 0.0.0.0:3030` To start an outer client node, you can also run the following command from the `snarkOS` directory: ``` ./scripts/run-outer-client.sh ``` Outer clients can be bootstrap clients that serve as accessible entry points for new nodes joining the network with publicly known or static IPs. For bootstrap clients, we also recommend the use of `--rotate-external-peers` to avoid the bootstrap peerlist from filling up. ## 3.2 Run an Aleo Validator Start by following the instructions in the [Build Guide](#2-build-guide). The following command is recommended when starting a validator node: `snarkos start --validator --nodisplay --bft 0.0.0.0:5000 --node 0.0.0.0:4130 --peers "validator_ip_1:4130,validator_ip_2:4130,...,core_client_ip_1:4130,core_client_ip_2:4130,..." --validators "validator_ip_1:5000,validator_ip_2:5000,..." --verbosity 1 --norest --private-key-file ~/snarkOS/privatekey` Instead of specifying a private key file (`--private-key-file` flag), the private key can also be defined explicitly (`--private-key` flag). To start a validator, you can also run the following command from the `snarkOS` directory: ``` ./scripts/run-validator.sh ``` ### 3.2.1 Validator Telemetry Metrics Validator telemetry allows you to track participation in consensus. This is enabled by default in the standard build. Once enabled, telemetry metrics are available through: 1. Node logs 2. REST API endpoints ``` // GET /{network}/validators/participation // GET /{network}/validators/participation?metadata={true} ``` Telemetry is part of the `metrics` feature, which is enabled by default. The `telemetry` feature flag is kept as an alias for `metrics`, so commands that name it explicitly still work: #### 1. [Installation](#2.3-installation) Use the default installation command: ``` cargo install --locked --path . ``` #### 2. `./run-validator.sh` Telemetry is included by default when running `./scripts/run-validator.sh`. ## 3.3 Run an Aleo Prover Start by following the instructions in the [Build Guide](#2-build-guide). Next, generate an Aleo account address: ``` snarkos account new ``` This will output a new Aleo account in the terminal. **Please remember to save the account private key and view key.** The following is an example output: ``` Attention - Remember to store this account private key and view key. Private Key APrivateKey1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <-- Save Me And Use In The Next Step View Key AViewKey1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <-- Save Me Address aleo1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <-- Save Me ``` Next, to start a proving node, from the `snarkOS` directory, run: ``` ./scripts/run-prover.sh ``` When prompted, enter your Aleo private key: ``` Enter the Aleo Prover account private key: APrivateKey1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ``` ## 4. FAQs ### 1. My node is unable to compile. - Ensure your machine has Rust installed, with at least [this version](rust-toolchain). Instructions to [install Rust can be found here.](https://www.rust-lang.org/tools/install) - If large errors appear during compilation, try running `cargo clean`. - Ensure `snarkOS` is started using `./scripts/run-client.sh` or `./scripts/run-prover.sh`. ### 2. My node is unable to connect to peers on the network. - Ensure ports `4130/tcp` and `3030/tcp` are open on your router and OS firewall. - Ensure `snarkOS` is started using `./scripts/run-client.sh` or `./scripts/run-prover.sh`. ### 3. I can't generate a new a