Blb 是一个分布式对象存储系统,专为集群计算环境中的裸机而设计。
Blb is a distributed object storage system designed for use on bare metal with spinning disks in cluster computing environments.
It's implemented in Go. The developer-facing API is a client library, also
implemented in Go, that exposes objects with an interface similar to Go files
(io.ReadWriteSeeker).
Blb was designed with the following ideals in mind, in roughly this priority:
Blb borrows many ideas from other storage systems, most prominently Google's GFS and Colossus, and Microsoft's FDS. There are also some similarities to Dropbox's Magic Pocket.
As in similar systems, blobs are divided into fixed-size "tracts" of 8MB. Tracts are spread across storage nodes in the cluster. Each storage node runs a "tractserver" that manages the storage on that node and exposes it to the network.
Locations of tracts are stored in metadata servers, called "curators". Locations are stored explicitly, not based on consistent hashing or a similar scheme. This allows for the most control over data placement and movement.
Curators store their metadata in an embedded database (Bolt DB) and use the Raft protocol to guarantee consistency of updates across replicas.
Objects (also called blobs) are identified with a fixed-length numeric ID (currently 64 bits, but will probably move to 128). Objects are assigned to curators by a mapping from the upper bits of their ID, called the partition.
The partition-to-curator mapping is stored in another set of metadata servers, called "masters". The masters also use Raft to keep that mapping consistent. Currently partitions cannot be migrated from one curator to another.
This basic design is expected to scale to very large clusters just by adding more curators. (Although the implementation certainly has various bottlenecks that will need to be addressed before that point.)
To manipulate an object, the client library talks with the masters, curators, and tractservers. Some metadata can be cached to reduce the number of round-trips required to complete an operation.
Tractservers store tracts as files in a regular filesystem on local disks (we've used ext4). (Future plans include raw disk storage, bypassing the filesystem, with optional support for host-managed SMR.)
The tractserver is designed to have low memory and CPU requirements, so it can easily coexist with other services in a "converged" storage configuration. (In our environment we observed it using 50-60MB per 4TB disk under moderate load.)
Curator and master metadata is stored in a local filesystem as well, in Bolt DB files, logs, and DB snapshots, and should be on SSDs for good performance.
Blb supports two schemes for ensuring durability: plain replication, and Reed-Solomon erasure coding. All objects are created with replicated storage and can be optionally transitioned to erasure coded storage, after which they cannot be written to anymore. (Future plans may allow transitioning back to replicated storage when writing is requested.)
The replication factor may be controlled by clients. For RS erasure coding, clients choose from one of several pre-configured sets of parameters.
All data on disk is written with embedded crc32 checksums to detect corruption and errors in storage devices.
When full or partial disk failures are detected, Blb re-replicates affected data on new disks in the cluster.
Tractservers slowly scrub all their data to proactively detect failures.
The main consistency guarantee provided is that after a successful write to a range of an object, all reads from that range will return the data that was written. If a write returns a failure, future reads of that range may return anything, though other ranges of the object are unaffected.
Blb assumes that it's running in a protected network and currently does no authentication of network requests. Data is not encrypted on disk. (Future plans include encrypting data on disk and managing encryption keys in the curator.)
Because Blb stores explicit locations for all data, adding storage to a cluster is simple and doesn't require migrating any data. (Future plans include rebalancing existing data across new disks in a slow controlled way.) Removing storage from a cluster is similarly simple: an operator can mark disks for draining, which will migrate data off of them in a slow controlled way.
Blb exports many detailed metrics about performance and other operational concerns in Prometheus format.
In general, Blb is designed to heal and recover from routine hardware failures and not bother ops staff. With proper configuration, it can even replace members of Raft groups automatically.
Blb uses the new Go module system as a build system, so you'll need Go 1.11 or later.
go get github.com/westerndigitalcorporation/blb/cmd/...To test things out quickly, you can use blbcli to start a cluster on your
local machine and interact with it. Here's a sample transcript, lightly edited
to remove irrelevant bits:
…Run ./testblb from the bin directory to run the suite of integration tests.
Unit tests can be run with the usual go test command.
master, curator, and tractserver are the main binaries that you'll need to
deploy. There are scripts in scripts that have examples for running them.
They'll need customization for your environment, of course. A basic cluster
would start with three or more tractservers, one master group with three
replicas, and one or more curator groups with three replicas each.
Several of the environment-specific dependencies of Blb were stubbed out in this open source release, including cluster sniffing ("What cluster am I running in?"), dynamic configuration, and most importantly, service discovery.
The stub service discovery implementation polls DNS, but for best results, you should replace it with something that uses whichever better mechanism is available in your environment (e.g. Consul, ZooKeeper, or a fancier DNS-based approach).
All the platform-specific stubs are in the platform directory.
The set of disks managed by the tractserver can be dynamic, allowing for hot
swapping without restarting the tractserver and interrupting traffic. The
interface is HTTP over an unix socket. Details are in
internal/tractserver/disk_controller.go.
Blb was originally developed at Upthere (a cloud storage service) as the intended storage system for most of our bulk data. It ran in production for several months, although not as the sole storage system. In September 2017, Upthere was acquired by Western Digital, and it was decided to pause development on Blb and move data to other storage systems.
Although Blb is not being actively developed as a production system, its authors plan to continue improving the system in their spare time as an educational project. If you'd like to contribute in any way, just file pull requests or issues here on GitHub. If you'd like to use it as a base for production services, we'd also like to hear from you.
Some projects we'd like to work on:
暂无开放 Issues，或尚未同步最近议题。