Off Cpu Analysis in Go

In this post I describe how to use Go's tracer to perform off-CPU analysis.

Off-CPU analysis tells you where time is spent while waiting for things(I/O, locks etc.).

Off-CPU analysis complements on-CPU analysis, so that 100% execution time is understood.

Usage

Add profile hooks to your service:

_ "net/http/pprof"

Run your service, which let's say listens on port 7788:

./my_service

Apply some workload, example with hey:

hey -z=10s http://localhost:7788

Collect traces:

curl -o trace.out "http://localhost:7788/debug/pprof/trace?seconds=5"

Generate profile(one of net, sync, syscall, sched based on your needs):

go tool trace -pprof=syscall trace.out > syscall.pprof

View profile with pprof:

go tool pprof -http=: syscall.pprof

or, if you want to use the latest pprof(installed via go install)

pprof -http=: syscall.pprof