基于 Filecoin 和 IPFS 构建的多层文件存储 API
Powergate is a multitiered file storage API built on Filecoin and IPFS, and an index builder for Filecoin data. It's designed to be modular and extensible.
Join us on our public Slack channel for news, discussions, and status updates. Check out our blog for the latest posts and announcements.
Warning This project is still pre-release and is not ready for production usage.
To build from source, you need to have a Go version that is between 1.14.x through 1.17.x. Go 1.18.x and above include a libp2p version that is not currently supported.
Powergate is composed of different modules which can be used independently, and compose together to provide other higher-level modules.
Here's a high-level overview of the main components, and how Powergate interacts with IPFS and a Filecoin client:
Note in the diagram that the Lotus and Filecoin client node doesn't need to be in the same host where Powergate is running. They can, but isn't necessary.
The Deals module provides a lower layer of abstraction to a Filecoin client node. It provides simple APIs to store, watch, and retrieve data in the Filecoin network. Currently, it interacts with the Lotus client but we have plans to support other Filecoin clients.
Powergate builds three indexes related to on-chain and off-chain data.
The Miners index provides processed data regarding registered miners (on-chain and off-chain), such as: total miner power, relative power, online status, geolocation, and more!
The Ask index provides a fast-retrieval up to date snapshot of miner's asking prices for data storage.
The Faults index provides history data about miners faults while proving their storage on-chain.
Built on top of the previous indexes, a Reputation module constructs a weighted-scoring system that allows to sort miners considering multiple on-chain and off-chain data, such as: compared price to the median of the market, low storage-fault history, power on network, and external sources (soon!).
This module provides a multitiered file storage API built on Filecoin and IPFS. Storing data on IPFS and Filecoin is as easy as expressing your desired configuration for storing a Cid.
Want to know more about this Powergate module? Check out the FFS design document and our presentation and demo at the IPFS Pinning Summit:
Powergate exposes an API built from the various modules through gRPC endpoints.
You can explore our .proto files to generate your clients, or take advange of a ready-to-use Powergate Go and JS client.
We have a CLI that supports most of Powergate features.
To build and install the CLI, run:
$ make install-powThe binary will be placed automatically in $GOPATH/bin. You may have to set the Path variables using the below commands
$ export PATH=$PATH:$(go env GOPATH)/bin
$ export GOPATH=$(go env GOPATH)You can then run pow in your terminal.
You can read the generated CLI docs in this repo, or run pow with the --help flag to see the available commands:
…Powergate installation involves running external dependencies, and wiring them correctly with Powergate.
Powergate needs external dependencies in order to provide full functionality, in particular a synced Filecoin client and a IPFS node.
Currently, we support the Lotus Filecoin client but we plan to support other clients.
All described modules of Powergate need to comunicate with Lotus to build indices data, and provide storing and retrieving features in FFS. To install Lotus refer to its official documentation, taking special attention to its dependencies.
Fully syncing a Lotus node can take time, so be sure to check you're fully synced doing ./lotus sync status.
We also automatically generate a public Docker image targeting the master branch of Lotus. This image is a pristine version of Lotus, with a sidecar reverse proxy to provide external access to the containerized API. For more information, refer to textileio/lotus-build and its Dockerhub repository.
In short, a fully-synced Lotus node should be available with its API (127.0.0.1:1234, by default) port accessible to Powergate.
A running IPFS node is needed if you plan to use the FFS module.
If that's the case, you can refer here for installation instructions, or its Dockerhub repository if you want to run a contanerized version. Currently we're supporting v0.5.1. The API endpoint should be accessible to Powergate (port 5001, by default).
Since FFS HotStorage is pinning Cids in the IPFS node, Powergate should be the only party controlling the pinset of the node. Other systems can share the same IPFS node if can guarantee not unpinning Cids pinned by Powergate FFS instances.
Powergate needs an offline geo-location database to resolve miners country using their IP address. The same folder in which powd is executing, should have the Geolite2 database file GeoLite2-City.mmdb or you can pass the --maxminddbfolder flag to powd to specify the path of the folder containing GeoLite2-City.mmdb.
You can copy this file from the GitHub repo at iplocation/maxmind/GeoLite2-City.mmdb. If you run Powergate using Docker, this database is bundeled in the image so isn't necessary to have extra considerations.
To build and install the Powergate server, run:
make install-powdYou can run the -h flag to see the configurable flags:
…Having a fully synced Lotus node can take a considerable amount of time and effort to mantain. We have built lotus-devnet which runs a local network with a sectorbuilder mock. This provides a fast way to spinup a local network where the sealing process if mocked, but the rest of the node logic is the same as production The localnet supports both 2Kib and 512Kib sectors, and the speed of block production is configurable. Refer to lotus-devnet readme for more information.
If you're interested in running Powergate and experiment with the CLI, the fastest way is to replace the Lotus client dependency with a running localnet, which runs a local Lotus client connected to a network with local miners.
A simple docker-compose setup is available that will run Powergate connected to a Lotus local network with 512Mib sectors and allows to use the gRPC API or CLI without any extra config flags! Note: you will first need to install Docker compose in order to get started.
cd docker
make localnetThis will build Powergate powd, a Lotus local network with BIGSECTORS=true by default, an IPFS node and wire them correctly to be ready to use.
Note: Running BIGSECTORS=false make localnet will create the Lotus devent using 2Kib sectors. This may be more appropriate for certain development or testing scenarios.
Here is a full example of using the local network: Terminal 1:
cd docker
make localnetWait for seeing logs about the height of the chain increase in a regular cadence.
Terminal 2:
…In this example we created a random 700 bytes file for the test, but since the localnet supports 512Mib sectors you can store store bigger files.
A production setup is also provided in the docker folder. It launches powd connected to lotus and ipfs, plus a set of monitoring components:
Depending on which network you want to connect to, you have to run different commands:
make up, to connect to mainnet.Remember that you should wait for Lotus to be fully-synced which might take a long time; you can check your current node sync status running lotus sync status inside the Lotus container. We also provide automatically generated Dockerhub images of Powergate server, see textile/powergate.
If you're interested in a more detailed explanation about Powergate installation, please refer to the installation docs.
We have a big set of tests for covering most important Powergate features.
For integration tests, we leverage our textileio/lotus-devnet configured with 2Kib sectors to provide fast iteration and CI runs.
If you want to run tests locally:
make testIt will auto-download any necessary dependencies and run all tests.
There's a dedicated binary to run benchmarks against a Powergate server. For more information see the specific README.
Soon we'll add benchmark results against real miners in mainnet, so stay tuned. ⌛
This project is a work in progress. As such, there's a few things you can do right now to help out:
Before you get started, be sure to read our contributors guide and our contributor covenant code of conduct.
Changelog is published to Releases.
暂无开放 Issues,或尚未同步最近议题。