profefe

profefe

profefe, a continuous profiling system, collects profiling data from a fleet of running applications and provides API for querying profiling samples for postmortem performance analysis.

Why Continuous Profiling?

"Continuous Profiling and Go" describes the motivation behind profefe:

With the increase in momentum around the term “observability” over the last few years, there is a common misconception amongst the developers, that observability is exclusively about metrics, logs and tracing (a.k.a. “three pillars of observability”) [..] With metrics and tracing, we can see the system on a macro-level. Logs only cover the known parts of the system. Performance profiling is another signal that uncovers the micro-level of a system; continuous profiling allows observing how the components of the application and the infrastructure it runs in, influence the overall system.

How does it work?

See Design Docs documentation.

Quickstart

To build and start profefe collector, run:

$ make
$ ./BUILD/profefe -addr=localhost:10100 -storage-type=badger -badger.dir=/tmp/profefe-data

2019-06-06T00:07:58.499+0200    info    profefe/main.go:86    server is running    {"addr": ":10100"}

The command above starts profefe collector backed by BadgerDB as a storage for profiles. profefe supports other storage types: S3, Google Cloud Storage and ClickHouse.

Run ./BUILD/profefe -help to show the list of all available options.

Example application

profefe ships with a fork of Google Stackdriver Profiler's example application, modified to use profefe agent, that sends profiling data to profefe collector.

To start the example application run the following command in a separate terminal window:

$ go run ./examples/hotapp/main.go

After a brief period, the application will start sending CPU profiles to profefe collector.

send profile: http://localhost:10100/api/0/profiles?service=hotapp-service&labels=version=1.0.0&type=cpu
send profile: http://localhost:10100/api/0/profiles?service=hotapp-service&labels=version=1.0.0&type=cpu
send profile: http://localhost:10100/api/0/profiles?service=hotapp-service&labels=version=1.0.0&type=cpu

With profiling data persisted, query the profiles from the collector using its HTTP API (refer to documentation for collector's HTTP API below). As an example, request all profiling data associated with the given meta-information (service name and a time frame), as a single merged profile:

$ go tool pprof 'http://localhost:10100/api/0/profiles/merge?service=hotapp-service&type=cpu&from=2019-05-30T11:49:00&to=2019-05-30T12:49:00&labels=version=1.0.0'

Fetching profile over HTTP from http://localhost:10100/api/0/profiles...
Saved profile in /Users/varankinv/pprof/pprof.samples.cpu.001.pb.gz
Type: cpu

(pprof) top
Showing nodes accounting for 43080ms, 99.15% of 43450ms total
Dropped 53 nodes (cum &type=[cpu|heap|...]&labels=
body pprof.pb.gz

,
    ···
  }
}

• service — service name (string)
• type — profile type ("cpu", "heap", "block", "mutex", "goroutine", "threadcreate", or "other")
• labels — a set of key-value pairs, e.g. "region=europe-west3,dc=fra,ip=1.2.3.4,version=1.0" (Optional)

Example

$ curl -XPOST \
  "http:///api/0/profiles?service=api-backend&type=cpu&labels=region=europe-west3,dc=fra" \
  --data-binary "@$HOME/pprof/api-backend-cpu.prof"

Store runtime execution traces (experimental)

Go's runtime traces are a special case of profiling data, that can be stored and queried with profefe.

Currently, profefe doesn't support extracting the timestamp of when the trace was created. Client may provide this information via created_at parameter, see below.

POST /api/0/profiles?service=&type=trace&created_at=&labels=
body trace.out

&type=&from=&to=&labels=

    },
    ···
  ]
}

• service — service name
• from, to — a time frame in which profiling data was collected, e.g. "from=2006-01-02T15:04:05"
• type — profile type ("cpu", "heap", "block", "mutex", "goroutine", "threadcreate", "trace", "other") (Optional)
• labels — a set of key-value pairs, e.g. "region=europe-west3,dc=fra,ip=1.2.3.4,version=1.0" (Optional)

Example

$ curl "http:///api/0/profiles?service=api-backend&type=cpu&from=2019-05-01T17:00:00&to=2019-05-25T00:00:00"

Query saved profiling data returning it as a single merged profile

GET /api/0/profiles/merge?service=&type=&from=&to=&labels=

,
    ···
  ]
}

Get profefe server version

GET /api/0/version

,
    "commit": ,
    "build_time": "
  }
}

FAQ

Does continuous profiling affect the performance of the production?

Profiling always comes with some costs. Go collects sampling-based profiling data and for the most applications the real overhead is small enough (refer to "Can I profile my production services" from Go's Diagnostics documentation).

To reduce the costs, users can adjust the frequency of collection rounds, e.g. collect 10 seconds of CPU profiles every 5 minutes.

profefe-agent tries to reduce the overhead further by adding a small jiggling in-between the profiles collection rounds. This distributes the total profiling overhead, making sure that not all instances of application's cluster are being profiled at the same time.

Can I use profefe with non-Go projects?

profefe collects pprof-formatted profiling data. The format is used by Go profiler, but thrid-party profilers for other programming languages support of the format too. For example, google/pprof-nodejs for Node.js, tikv/pprof-rs for Rust, arnaud-lb/php-memory-profiler for PHP, etc.

Integrating those is the subject of building a transport layer between the profiler and profefe.

Further reading

While the topic of continuous profiling in the production is quite unrepresented in the public internet, some research and commercial projects already exist:

• Stackdriver profiler
• Google-Wide Profiling: A Continuous Profiling Infrastructure for Data Centers (paper)
• StackImpact
• conprof
• Opsian - Continuous Profiling for JVM (provides on-premises plan for enterprise customers)
• Liveprof - Continuous Profiling for PHP (RUS)
• FlameScope

profefe is still in its early state. Feedback and contribution are very welcome.

License

MIT