netlog

NetLog

A lightweight, HTTP-centric, log-based (Kafka-style) message queue.

Alpha software

This is still early software and potentially buggy. To peek at the internals start with BigLog.

Roadmap

• low-level log management
• HTTP transport
• scanner based pub/sub
• custom data retention policy
• persistent scanners
• batching
• compression
• good test coverage
• proper documentation
• streaming based pub/sub
• async replication
• kinesis-compatible transport
• gRPC transport

Non-goals

• Match Kafka's performance.
• Distributed system.

Getting started

While posting and fetching single messages is very inefficient, it's the simplest way to get started using nothing but curl commands.

…

One-line-ish pub/sub

# create new topic
curl -XPOST localhost:7200/pubsubdemo

# get scanner ID with jq
export SCANNER=$(curl -s -XPOST "localhost:7200/pubsubdemo/scanner?from=0&persist=true" | jq -r .id)

# subscribe to the topic
while true; do; curl "localhost:7200/pubsubdemo/scan?id=$SCANNER&wait=1h" && echo; done

# IN ANOTHER WINDOW

# publish on the topic
while true; do; read data; curl localhost:7200/pubsubdemo/payload --data-binary $data; done
# write something and hit enter

Contributing

Contributions are more than welcome, check the contributing guidelines. To ask any questions you can write to the netlog-dev mailing list.