kirby Kirby 从 Fastly 中吸收大量日志,并快速计算出各种 Ruby 生态系统统计数据的日均值。H…
Kirby slurps up the firehose of logs from Fastly and calculates daily counts for various Ruby ecosystem statistics, pretty quickly.
For an 80MB gzipped log file containing 915,427 JSON event objects (which is 1.02GB uncompressed):
This is... very good. For comparison, a Python script that used AWS Glue to do something similar took about 30 minutes. My first approach of writing a nom parser-combinator to parse the User Agent field, instead of using a regex, took 18.7 seconds. Processing a gigabyte of almost a million JSON objects into useful histograms in less than 8 seconds just blows my mind. But then I figured out how to use Rayon, and now it can parse 8 gzipped log files in parallel on an 8-core MacBook Pro, and that's super fast.
Then Rust got more optimized and Apple released the M1, and it got still faster. Finally, and I found the profile-guided optimization docs, and it improved even more than I thought was still possible.
Most recently, it also turned out there was a highly contended mutex around the regular expressions and that bought the multi-core version something like 40-60% more speed.
~525 records/second/cpu in Python on Apache Spark in AWS Glue
~14,000 records/second/cpu in Ruby on a 2018 Intel MacBook Pro
~353,000 records/second/cpu in Rust on a 2018 Intel MacBook Pro
~550,000 records/second/cpu in Rust on a 2021 M1 MacBook Pro
~638,000 records/second/cpu in Rust on a 2021 M1 with PGO
~770,000 records/second/cpu in Rust on a 2022 M1 Ultra Studio with PGO
~935,500 records/second/cpu in Rust on a 2025 M4 Max MacBook Pro
~983,500 records/second/cpu in Rust on a 2025 M4 Max with PGO
~1,240,000 records/second/cpu in Rust on a 2024 Ryzen 9 9950X with PGONo. The latest version can parse records really, really fast.
~4,200 records/second in Python with 8 worker instances on AWS Glue
~1,085,000 records/second in Rust with rayon on an 8-core Intel MacBook Pro
~3,195,000 records/second in Rust with rayon on a 10-core M1 MacBook Pro
~3,583,000 records/second in Rust with rayon on M1 with PGO
~12,004,000 records/second in Rust with rayon on a 20-core M1 Ultra with PGO
~13,076,000 records/second in Rust with rayon on a 16-core M4 Max with PGO
~22,559,000 records/second in Rust with rayon on a 32-core Ryzen 9 9950X with PGOIt counts Bundler, RubyGems, and Ruby versions, in hourly buckets, and prints those out as nested JSON to stdout.
Okay, I wrote a blog post with details about creating this library, and a follow up about more optimizations.
暂无开放 Issues,或尚未同步最近议题。