qfs

QFS version 2

Release 2.2.8

• Release 2.2.8 contains updated Java module, Java 9+ support, Hadoop 3 benchmark support, and minor improvements. Release notes.

Release 2.2.7

• Release 2.2.7 contains updated python module, python3 support, and minor improvements. Release notes.

QFS version 2.2.6

Release 2.2.6

• Release 2.2.6 contains go language bindings, bug fixes, and minor improvements. Release notes.

QFS version 2.2.5

Release 2.2.5

• Release 2.2.5 contains bug fixes and minor improvements. Release notes.

QFS version 2.2.4

Release 2.2.4

• Release 2.2.4 is a bug fix release. Release notes.

Release 2.2.3

• Support for chunk server's node ID that can be used instead of IP/port for determining if chunk server and QFS client are co-located on the same network node and use chunk server to service client requests.
• Bug fixes.

Release notes.

QFS version 2.2.2

Release 2.2.2 is a bug fix release. Release notes.

QFS version 2.2.1

Release 2.2.1 is a bug fix release. Release notes.

QFS version 2.2.0

Release 2.2.0 contains symbolic links support, and minor bug fixes.

QFS version 2.1.3

Release 2.1.3 contains DNS resolver's number of open socket accounting bug fix. This bug might manifest itself in at least one non obvious way: chunk server might continuously fail to create or open chunk files if/when the open sockets counter becomes negative.

Release 2.1.2 contains minor bug fixes, meta and chunk servers watchdog.

Release 2.1.1 contains minor bug fixes.

Non blocking DNS resolver is the new feature in release 2.1.0. The new DNS resolver is intended to improve S3 [compatible] object store IO concurrency and reduce latency.

Meta server replication is the major new feature in release 2.0.0. Meta server replication provides automatic meta server fail over. With meta server replication configured QFS does not have single point of failure.

Release notes are available.

Quantcast File System

Quantcast File System (QFS) is a high-performance, fault-tolerant, distributed file system developed to support MapReduce processing, or other applications reading and writing large files sequentially.

QFS used in Quantcast production cluster.

QFS servers have been tested on 64-bit CentOS 6 extensively and run on Linux variants. The QFS client tools work on OS X and Cygwin as well.

Platform	Build Status
Mac OS X x86_64
Mac OS X arm64
Ubuntu 14.04
Ubuntu 18.04
Ubuntu 20.04
Ubuntu 22.04
Ubuntu 24.04 x86_64
Ubuntu 24.04 arm64
CentOS 6
CentOS 7
CentOS 8
Rocky Linux 9 x86_64
Rocky Linux 9 arm64
Debian 10
Debian 11
Debian 12 x86_64
Debian 12 arm64
Amazon Linux 2023 x86_64
Amazon Linux 2023 arm64

The implementation details and features of QFS are discussed in detail in the project wiki.

Getting QFS

QFS is available through various channels.

• BigTop: QFS packages for rpm and debian based systems are available through the BigTop project.
• Binary Distributions: There are various binary distributions packaged as tarballs available on the Binary Distributions Page.
• Compile from source: QFS can be compiled on Linux variants, Mac OS X, and Cygwin. See the wiki for more information on how to build QFS yourself.

Trying QFS

Once you have aquired QFS through one of the methods above, you can take QFS for a quick test drive. Setting up a single node configuration to familiarize yourself with QFS is very easy.

1. Extract the distribution tarball.

   bash复制

   tar -xzf qfs.tgz && cd qfs

2. Set up a single node QFS instance. This will create a workspace in ~/qfsbase, start two chunk servers and one metaserver.

   bash复制

   examples/sampleservers/sample_setup.py -a install

   bash复制

   Binaries presence checking - OK.
   Setup directories - OK.
   Setup config files - OK.
   Started servers - OK.

3. Add tools binary path to PATH

   bash复制

   PATH=${PWD}/bin/tools:${PATH}

4. Make a temporary directory on the file system

   bash复制

   qfsshell -s localhost -p 20000 -q -- mkdir /qfs/tmp

5. Create a file containing "Hello World", Reed-Solomon encoded, with replication 1.

   bash复制

   echo 'Hello World' | cptoqfs -s localhost -p 20000 -S -k /qfs/tmp/helloworld -d -

6. Cat the file content.

   bash复制

   qfscat -s localhost -p 20000 /qfs/tmp/helloworld

7. Stat the file to see encoding (RS or not), replication level, and mtime.

   bash复制

   qfsshell -s localhost -p 20000 -q -- stat /qfs/tmp/helloworld

8. Copy the file locally to the current directory.

   bash复制

   cpfromqfs -s localhost -p 20000 -k /qfs/tmp/helloworld -d ./helloworld

9. Remove the file from QFS.

   bash复制

   qfsshell -s localhost -p 20000 -q -- rm /qfs/tmp/helloworld

10. Stop the servers.

    bash复制

    examples/sampleservers/sample_setup.py -a stop

11. Uninstall the single node instance.

    bash复制

    examples/sampleservers/sample_setup.py -a uninstall

Benchmarking QFS

A performance comparison between QFS and HDFS 1.0.2 shows QFS is faster both at reading and writing 20 TB of uncompressed data on our test system, a heterogeneous cluster with 6,500 disk drives. See more information regarding this in our wiki.

Contributing to QFS

We welcome contributions to QFS in the form of enhancement requests, patches, additional tests, bug reports, new ideas, and so on. Please submit issues using our JIRA instance and refer to the QFS code contribution policy when contributing code.

Have Questions?

Join the QFS Developer mailing list or search the archives at the Google Group.

Post comments or questions to .

License

QFS is released under the Apache 2.0 license.