The following is the basic timeline for S3. In linux, suspend occurs in 10 internal phases (as shown in the below timeline). Kernel devices can register callbacks in any or all of them and they are executed in order during a suspend/resume. The timeline shows all the registered callbacks and the time taken for completion. It also includes some general task blocks in order to fill out the timeline with contiguous info: e.g. the filesystem sync in prepare and the thaw_processes call in resume_complete.
The zoom buttons allow you to zoom in and out on the timeline for finer detail. The keyboard shortcuts for zoom in/out/reset are +/-/*. You can also click on the timeline contents and drag it left or right. Whenever you hover the mouse over a device, it will highlight all its callbacks in all phases so that you can find them easily. Clicking on a device will produce a device detail pane beneath the timeline.
The device detail pane gives the full name of the device and the total time all of its callbacks took to complete (suspend or resume). Each phase's callback is scaled to its relative length so that you can see where most of the time was spent at a glance. If you'd like to see a tabular list of all the devices and their callback times, click the device detail button on the left. This is also useful in determining which devices have parent child relationships.
If you used the -f option in the test run, the page will include a detailed call trace of all the registered device callbacks. Please note that there is a huge amount of data in a callback trace and your outputs may take time to load on screen. The callgraph data is displayed beneath the detail pane and will filter based on which device you've selected.
The data for these is taken from ftrace. Each item is a collapsible treeview of all the calls that were measured in each device callback. This is the lowest level output which can point a developer to specific functions needing optimization.
The tool also supports some more advanced options. One of which is the dev mode option, which allows you to graph a collection of function calls through the use of kprobes. Dev mode focuses on delay calls: msleep, schedule_timeout, udela. This is useful in determining how much of the wait time is actually hardcoded into the device driver. It will also include a suite of callbacks that are important to the various subsystems in the kernel: e.g. ata and i915. The following is a dev mode output.
You can also run the tool with the -x2 option, which issues two back to back suspend/resumes. This is useful to determine when asynchronous resume processes are fully complete. Many devices spawn async processes which continue after resume complete. It can also be useful to monitor which user processes are executing just prior to and after suspend. The -proc option enables the tool to collect process info in addition to kernel info. The following timeline was run with both -x2 and -proc.
Download & install pm-graph
The tools can most easily be installed via git clone and make install
$> git clone http://github.com/intel/pm-graph.git $> cd pm-graph $> sudo make install $> man sleepgraph