Slice and dice logs on the command line
Slice and dice log files on the command line.
Angle-grinder allows you to parse, aggregate, sum, average, min/max, percentile, and sort your data. You can see it, live-updating, in your terminal. Angle grinder is designed for when, for whatever reason, you don't have your data in graphite/honeycomb/kibana/sumologic/splunk/etc. but still want to be able to do sophisticated analytics.
Angle grinder can process well above 1M rows per second (simple pipelines as high as 5M), so it's usable for fairly meaty aggregation. The results will live update in your terminal as data is processed. Angle grinder is a bare bones functional programming language coupled with a pretty terminal UI.
Binaries are available for Linux and OSX. Many more platforms (including Windows) are available if you compile from source. In all of the commands below, the resulting binary will be called agrind. Starting with v0.9.0, agrind can self-update via the --self-update flag. Thanks to the many volunteers who maintain angle-grinder on different package managers & environments!
Brew
brew install angle-grinder
Macports
sudo port selfupdate
sudo port install angle-grinder
pkg install angle-grinder
curl -L https://github.com/rcoh/angle-grinder/releases/download/v0.18.0/agrind-x86_64-unknown-linux-musl.tar.gz \
| tar Ozxf - \
| sudo tee /usr/local/bin/agrind > /dev/null && sudo chmod +x /usr/local/bin/agrind
agrind --self-update
If you have Cargo installed, you can compile & install from source: (Works with Stable Rust >=1.26)
cargo install ag
An angle grinder query is composed of filters followed by a series of operators.
The filters select the lines from the input stream to be transformed by the operators.
Typically, the initial operators will transform the data in some way by parsing fields or JSON from the log line.
The subsequent operators can then aggregate or group the data via operators like sum, average, percentile, etc.
agrind '<filter1> [... <filterN>] | operator1 | operator2 | operator3 | ...'
A simple query that operates on JSON logs and counts the number of logs per level could be:
agrind '* | json | count by log_level'
Field names containing spaces, periods, or quotes must be escaped using ["<FIELD>"]:
agrind '* | json | count by ["date received"], ["grpc.method"]
There are three basic filters:
*: Match all logsfilter-me* (with no quotes) is a case-insensitive match that can include wildcards* matches literal *
, filter-me, or "filter me!".Filters can be combined with AND, OR and NOT
("ERROR" OR WARN*) AND NOT staging | count
Sub-expressions must be grouped in parenthesis. Only lines that match all filters will be passed to the subsequent operators.
Starting with v0.12.0, angle grinder supports aliases, pre-built pipelines do simplify common tasks or formats.
By default, angle-grinder will look in the .agrind-aliases directory in your current working directory and all parent directories.
Alias files look like this:
keyword = "apache"
template = """
parse "* - * [*] \\"* * *\\" * *" as ip, name, timestamp, method, url, protocol, status, contentlength
"""
Your operators are parsed, then expanded into the resulting pipeline. When invalid aliases are present, a warning will be displayed when running angle-grinder.
Note that aliases are currently considered an experimental feature and precise behavior may change in the future.
Examples:
* | apache | count by status
These operators have a 1 to 1 correspondence between input data and output data. 1 row in, 0 or 1 rows out.
json [from other_field]: Extract json-serialized rows into fields for later use. If the row is not valid JSON, then it is dropped. Optionally, from other_field can be
specified. Nested JSON structures are supported out of the box. Simply access nested values with .key[index], for example, .servers[6]. Negative indexing is also supported.
Examples:
* | json
* | parse "INFO *" as js | json from js
Given input like:
{"key": "blah", "nested_key": {"this": "that"}}
* | json | count_distinct(nested_key.this)
logfmt [from other_field]: Extract logfmt-serialized rows into fields for later use. If the row is not valid logfmt, then it is dropped. Optionally, from other_field can be specified. Logfmt is a an output format commonly used by Heroku and Splunk, described at https://www.brandur.org/logfmt.
Examples:
* | logfmt
Given input like:
{"key": "blah", "nested_key": "some=logfmt data=more"}
* | json | logfmt from nested_key | fields some
split[(input_field)] [on separator] [as new_field]: Split the input via the separator (default is ,). Output is an array type. If no input_field or new_field, the contents will be put in the key _split.
Examples:
* | split on " "
Given input like:
INFO web-001 influxd[188053]: 127.0.0.1 "POST /write HTTP/1.0" 204
Output:
[_split=[INFO, web-001, influxd[188053]:, 127.0.0.1, POST /write HTTP/1.0, 204]]
If input_field is used, and there is no new_field specified, then the input_field will be overridden with the split data-structure. For example:
* | parse "* *" as level, csv | split(csv)
Given input like:
INFO darren,hello,50
WARN jonathon,good-bye,100
Will output:
[csv=[darren, hello, 50]] [level=INFO]
[csv=[jonathon, good-bye, 100]] [level=WARN]
Other examples:
* | logfmt | split(raw) on "blah" as tokens | sum(tokens[1])
parse "* pattern * otherpattern *" [from field] as a,b,c [nodrop] [noconvert]: Parse text that matches the pattern into variables.
nodrop is specified. * is equivalent to regular expression .* and is greedy.noconvert will prevent parse from converting parsed fields into structured data and instead preserve them as strings. This can be helpful if you are parsing fields that sometimes have values like 00000.By default, parse operates on the raw text of the message. With from field_name, parse will instead process input from a specific column. Any whitespace in the parse
expression will match any whitespace character in the input text (eg. a literal tab).
Examples:
* | parse "[status_code=*]" as status_code
parse regex "<regex-with-named-captures>" [from field] [nodrop]: Match the
input text against a regular expression and populate the record with the named
captures. Lines that don't match the pattern will be dropped unless nodrop is
specified. By default, parse operates on the raw text of the message. With
from field_name, parse will instead process input from a specific column.
Notes:
\w works as-is).Examples: To parse the phrase "Hello, ...!" and capture the value of the "..." in the name field:
* | parse regex "Hello, (?P<name>\w+)"
fields [only|except|-|+] a, b: Drop fields a, b or include only a, b depending on specified mode.
Examples:
Drop all fields except event and timestamp
* | json | fields + event, timestamp
Drop only the event field
* | fields except event
where <bool-expr>: Drop rows where the condition is not met.
The condition must be an expression that returns a boolean value.
The expression can be as simple as a field name or a comparison (i.e. ==, !=, <=, >=, <, >)
between fields and literal values (i.e. numbers, strings).
The '!' operator can be used to negate the result of a sub-expression.
Note that None == None, so a row where both the left and right sides match a non-existent key will match.
Examples
* | json | where status_code >= 400
* | json | where user_id_a == user_id_b
* | json | where url != "/hostname"
limit #: Limit the number of rows to the given amount. If the number is positive, only the
first N rows are returned. If the number is negative, the last N rows are returned.
Examples
* | limit 10
* | limit -10
<expr> as <name>: The given expression is evaluated and the result is stored
in a field with the given name for the current row. The expression can be
made up of the following:
+, -, *, /: Mathematical operators with the normal precedence rules.
The operators work on numeric values and strings that can automatically be
converted to a number. In addition, these operators work for date-time and
duration values when appropriate. For example, you can take the difference
between two date-times, but cannot add them together.==, != (or <>), <=, >=, <, >: Boolean operators work
on most data types.and, &&, or, ||: Short-circuiting logical operators.<field>: The name of a field in the current row. If the row does not
contain the given field, an error will be reported.The following functions are supported within expressions:
abs(), acos(), asin(), atan(), atan2(),
cbrt(), ceil(), cos(), cosh(), exp(), expm1(), floor(),
hypot(), log(), log10(), log1p(), round(), sin(), sinh(),
sqrt(), tan(), tanh(), toDegrees(),
toRadians()concat(arg0, ..., argN) - Concatenate the arguments into a stringcontains(haystack, needle) - Return true if the haystack contains the needle.length(str) - Returns the number of characters in "str".now() - Returns the current date and time.num(value) - Returns the given value as a number.parseDate(str) - Attempt to parse a date from the given string.parseHex(str) - Attempt to convert a hexadecimal string into an integer.substring(str, startOffset, [endOffset]) - Returns the part of the string
specified by the given starting offset up to the end offset (if specified).toLowerCase(str) - Returns the lowercase version of the string.toUpperCase(str) - Returns the uppercase version of the string.isNull(value) - Returns true if value is null, false otherwise.isEmpty(value) - Returns true if value is null or an empty string, false
otherwise.isBlank(value) - Returns true if value is null, an empty string, or a
whitespace-only string, false otherwise.isNumeric(str) - Returns true if the given string is a number.Examples
Multiply value by 100 to get the percentage
* | json | value * 100 as
No open issues yet, or sync has not completed.