alp

alp

alp is Access Log Profiler

日本語

Installation

Binary distribution

You can pick your download here, and install it as follows:

sudo install  /usr/local/bin/alp

Using your distribution's package system

macOS (Homebrew)

Install alp with Homebrew

• brew install alp

asdf

Install alp with asdf and asdf-alp

asdf plugin-add alp https://github.com/asdf-community/asdf-alp.git
asdf install alp 
asdf global alp 

The difference between v0.4.0 and v1.0.0

See: The difference between v0.4.0 and v1.0.0

Usage

…

ltsv

• Parses a log in LTSV format
• By default, the following labels are parsed:
  • time
    • datetime
  • method
    • HTTP Method
  • uri
    • URI
  • status
    • HTTP Status Code
  • apptime
    • Response time from the upstream server
  • reqtime
    • Request Processing Time (Response time after receiving a request)
• The --xxx-label option can you change the name to any label

…

Log format

Apache

LogFormat "time:%t\tforwardedfor:%{X-Forwarded-For}i\thost:%h\treq:%r\tstatus:%>s\tmethod:%m\turi:%U%q\tsize:%B\treferer:%{Referer}i\tua:%{User-Agent}i\treqtime_microsec:%D\tapptime:%D\tcache:%{X-Cache}o\truntime:%{X-Runtime}o\tvhost:%{Host}i" ltsv

Nginx

…

H2O

access-log:
  format: "time:%t\thost:%h\tua:\"%{User-agent}i\"\tstatus:%s\treq:%r\turi:%U\tapptime:%{duration}x\tsize:%b\tmethod:%m"

json

• Parses newline-delimited JSON access logs. Each non-empty line must contain exactly one JSON object; an object cannot span multiple lines.
• Trailing whitespace is allowed. A second JSON value or any non-whitespace data after the object is rejected instead of being silently ignored.
• By default, the following keys are parsed:
  • time
    • datetime
  • method
    • HTTP Method
  • uri
    • URI
  • status
    • HTTP Status Code
  • response_time
    • Response time from the upstream server
  • request_time
    • Request Processing Time (Response time after receiving a request)
• The --xxx-key option can you change the name to any key

…

Log format

Apache

LogFormat "{\"time\":\"%t\",\"forwardedfor\":\"%{X-Forwarded-For}i\",\"host\":\"%h\",\"req\":\"%r\",\"status\":%>s,\"method\":\"%m\",\"uri\":\"%U%q\",\"body_bytes\":%B,\"referer\":\"%{Referer}i\",\"ua\":\"%{User-Agent}i\",\"reqtime_microsec\":%D,\"response_time\":%D,\"cache\":\"%{X-Cache}o\",\"runtime\":\"%{X-Runtime}o\",\"vhost\":\"%{Host}i\"}" json

Nginx

…

H2O

access-log:
  escape: json
  format: '{"time":"%t","host":"%h","ua":"%{User-agent}i","status":%s,"req":"%r","uri":"%U","response_time":%{duration}x,"body_bytes":%b,"method":"%m"}'

regexp

• Parses the log to match the regular expression
• By default, the following named capture groups are parsed:
  • time
    • datetime
  • method
    • HTTP Method
  • uri
    • URI
  • status
    • HTTP Status Code
  • response_time
    • Response time from the upstream server
  • request_time
    • Request Processing Time (Response time after receiving a request)
• The --xxx-subexp option can you change the name to any named capture groups

…

Log format

Apache

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D" combined_plus

Nginx

    log_format combined_plus '$remote_addr - $remote_user [$time_local] '
                             '"$request" $status $body_bytes_sent '
                             '"$http_referer" "$http_user_agent" $upstream_response_time $request_time';

H2O

access-log:
  format: "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" %{duration}x"

pcap

• Parses the pcap file to extract HTTP request/response packets to analyze the stats
  • Note that the actual response time may time duration from the actual response time because the difference in the timestamp of packet capturing is regarded as the real response time
  • The IP address and TCP port number of the server are required to distinguish between HTTP requests/responses to the server
• Able to specify the IP address of the HTTP server with the --pcap-server-ip option
  • This option can be specified more than once
  • By default, it automatically obtains the IP address from the network interfaces of its own host and uses it
  • However, 127.0.0.1 and ::1 will be the defaults in environments where permissions to retrieve network interface information are restricted.
• Able to specify the TCP port of the HTTP server with the --pcap-server-port option
  • The default server port number is 80.
• Cannot be used with --pos. (not yet supported)

…

diff

• Show the difference between the two profile results
• + means an increasing number of requests and body size, and response time is slower
• - means a decreasing number of requests and body size, and response time is faster

…

Global options

See: Usage samples

• -c, --config
  • The configuration file
  • YAML
• --file=FILE
  • The access log file
• -d, --dump=DUMP
  • File path for creating the profile results to a file
• -l, --load=LOAD
  • File path to read the results of the profile created with the -d, --dump option
  • Can expect it to work fast if you change the --sort and --reverse options for the same profile results
• --sort=count
  • Output the results in sorted order
  • Sort in ascending order
  • max, min, sum, avg
  • max-body, min-body, sum-body, avg-body
  • p90, p95, p99, stddev
  • uri
  • method
  • count
  • The default is count
  • p90, p95, and p99 are modified by the values specified in --percentiles
• -r, --reverse
  • Sort in desecending order
• -q, --query-string
  • URIs up to and including query strings are included in the profile
• --qs-ignore-values
  • Ignore the value of the query string
  • It's not enabled unless use with -q, --query-string
• --decode-uri
  • Decode the URI
• --format=table
  • Print the profile results in a table, Markdown, TSV, CSV and HTML format
  • The default is table format
• --noheaders
  • Print no header when TSV and CSV format
• --show-footers
  • Print the total number of each 1xx ~ 5xx in the footer of the table or Markdown format
• --limit=5000
  • Maximum number of profile results to be printed
  • This setting is to avoid using too much memory
  • The default is 5000 lines
• --location="Local"
  • The timezone of the time specified in the filter condition.
  • Default is localhost timezone
• -o, --output="all"
  • Specify the profile results to be print, separated by commas
  • count,1xx, 2xx, 3xx, 4xx, 5xx, method, uri, min, max, sum, avg, p90, p95, p99, stddev, min_body, max_body, sum_body, avg_body
    • p90, p95, and p99 are modified by the values specified in --percentiles
  • The default is all
• -m, --matching-groups=PATTERN,...
  • Treat URIs that match regular expressions as the same URI
  • Evaluate in the specified order. If matched, no further evaluation is performed.
  • See URI matching groups
• -f, --filters=FILTERS
  • Filters the targets for profile
  • See Filter
• --pos=POSITION_FILE
  • Stores the number of bytes to which the file has been read.
  • If the number of bytes is stored in the POSITION_FILE, the data after that number of bytes will be profiled
  • You can profile without truncating the file
    • Also, it is expected to work fast because it seeks and skips files
• --nosave-pos
  • Data after the number of bytes specified by --pos is profiled, but the number of bytes reads is not stored
• --percentiles
  • Specifies the percentile values to output, separated by commas
  • The default is 90,95,99

URI matching groups

Consider the following cases like /diary/entry/1234 and /diary/entry/5678. If you simply profile URIs with different parameters on the same route, they will be profiled by parameter, but you may want to profile them by the route.

…

…

In such a case, there is an option -m, --matching-groups=PATTERN,.... You can also specify multiple items separated by commas.

Filter

It is a function to include or exclude targets according to the conditions.

Variables

Filter on the following variables:.

• Uri
  • URI
• Method
  • HTTP Method
• Time
  • Datetime string
  • See: https://github.com/tkuchiki/parsetime
• ResponseTime
  • Response time
• BodyBytes
  • Bytes of HTTP Body
• Status
  • HTTP Status Code

Operators

The following operators are available:.

• +, -, *, /, %, **(pow)
• ==, !=, ``, =
• not, !
• and, &&
• or, ||
• matches
  • e.g.
    • Uri matches "PATTERN"
    • not(Uri matches "PATTERN")
• contains
  • e.g.
    • Uri contains "STRING"
    • not(Uri contains "STRING")
• startsWith
  • e.g.
    • Uri startsWith "PREFIX"
    • not(Uri startsWith "PREFIX")
• endsWith
  • e.g.
    • Uri endsWith "SUFFIX"
    • not(Uri endsWith "SUFFIX")
• in
  • e.g.
    • Method in ["GET", "POST"]
    • Method not in ["GET", "POST"]

See: https://github.com/antonmedv/expr/blob/master/docs/Language-Definition.md

Functions

• TimeAgo(duration)
  • now - duration
  • ns, us or µs, ms, s, m, h
  • e.g.
    • Time >= TimeAgo("5m")
• BetweenTime(val, start, end)
  • Like SQL's BETWEEN, returns start <= val && val <= end
  • e.g.
    • BetweenTime(Time, "2019-08-06T00:00:00", "2019-08-06T00:05:00")

Usage samples

See: Usage samples

Donation

Donations are welcome as always!
:heart: Sponsor