Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 1 | ========================== |
| 2 | Kprobe-based Event Tracing |
| 3 | ========================== |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 4 | |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 5 | :Author: Masami Hiramatsu |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 6 | |
| 7 | Overview |
| 8 | -------- |
Masami Hiramatsu | 77b44d1 | 2009-11-03 19:12:47 -0500 | [diff] [blame] | 9 | These events are similar to tracepoint based events. Instead of Tracepoint, |
| 10 | this is based on kprobes (kprobe and kretprobe). So it can probe wherever |
Naveen N. Rao | c1ac094 | 2017-03-08 22:34:14 +0530 | [diff] [blame] | 11 | kprobes can probe (this means, all functions except those with |
| 12 | __kprobes/nokprobe_inline annotation and those marked NOKPROBE_SYMBOL). |
| 13 | Unlike the Tracepoint based event, this can be added and removed |
Masami Hiramatsu | 77b44d1 | 2009-11-03 19:12:47 -0500 | [diff] [blame] | 14 | dynamically, on the fly. |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 15 | |
Anton Blanchard | 6b0b755 | 2017-02-16 17:00:50 +1100 | [diff] [blame] | 16 | To enable this feature, build your kernel with CONFIG_KPROBE_EVENTS=y. |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 17 | |
Masami Hiramatsu | 77b44d1 | 2009-11-03 19:12:47 -0500 | [diff] [blame] | 18 | Similar to the events tracer, this doesn't need to be activated via |
| 19 | current_tracer. Instead of that, add probe points via |
| 20 | /sys/kernel/debug/tracing/kprobe_events, and enable it via |
Andreas Ziegler | e50891d | 2018-07-16 13:05:57 +0200 | [diff] [blame] | 21 | /sys/kernel/debug/tracing/events/kprobes/<EVENT>/enable. |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 22 | |
| 23 | |
| 24 | Synopsis of kprobe_events |
| 25 | ------------------------- |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 26 | :: |
| 27 | |
Masami Hiramatsu | 6142431 | 2011-06-27 16:26:56 +0900 | [diff] [blame] | 28 | p[:[GRP/]EVENT] [MOD:]SYM[+offs]|MEMADDR [FETCHARGS] : Set a probe |
Alban Crequy | 696ced4 | 2017-04-03 12:36:22 +0200 | [diff] [blame] | 29 | r[MAXACTIVE][:[GRP/]EVENT] [MOD:]SYM[+0] [FETCHARGS] : Set a return probe |
Motohiro KOSAKI | df3ab70 | 2010-01-18 21:35:12 -0500 | [diff] [blame] | 30 | -:[GRP/]EVENT : Clear a probe |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 31 | |
Masami Hiramatsu | f52487e | 2009-09-10 19:53:53 -0400 | [diff] [blame] | 32 | GRP : Group name. If omitted, use "kprobes" for it. |
Masami Hiramatsu | 2fba0c8 | 2009-09-10 19:53:14 -0400 | [diff] [blame] | 33 | EVENT : Event name. If omitted, the event name is generated |
Masami Hiramatsu | 6142431 | 2011-06-27 16:26:56 +0900 | [diff] [blame] | 34 | based on SYM+offs or MEMADDR. |
| 35 | MOD : Module name which has given SYM. |
| 36 | SYM[+offs] : Symbol+offset where the probe is inserted. |
Masami Hiramatsu | 2fba0c8 | 2009-09-10 19:53:14 -0400 | [diff] [blame] | 37 | MEMADDR : Address where the probe is inserted. |
Alban Crequy | 696ced4 | 2017-04-03 12:36:22 +0200 | [diff] [blame] | 38 | MAXACTIVE : Maximum number of instances of the specified function that |
| 39 | can be probed simultaneously, or 0 for the default value |
| 40 | as defined in Documentation/kprobes.txt section 1.3.1. |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 41 | |
Masami Hiramatsu | 2fba0c8 | 2009-09-10 19:53:14 -0400 | [diff] [blame] | 42 | FETCHARGS : Arguments. Each probe can have up to 128 args. |
Masami Hiramatsu | 2e06ff6 | 2009-10-07 18:27:59 -0400 | [diff] [blame] | 43 | %REG : Fetch register REG |
| 44 | @ADDR : Fetch memory at ADDR (ADDR should be in kernel) |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 45 | @SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol) |
Masami Hiramatsu | 2e06ff6 | 2009-10-07 18:27:59 -0400 | [diff] [blame] | 46 | $stackN : Fetch Nth entry of stack (N >= 0) |
| 47 | $stack : Fetch stack address. |
Masami Hiramatsu | 1464010 | 2010-01-05 17:46:48 -0500 | [diff] [blame] | 48 | $retval : Fetch return value.(*) |
Omar Sandoval | 35abb67 | 2016-06-08 18:38:02 -0700 | [diff] [blame] | 49 | $comm : Fetch current task comm. |
Masami Hiramatsu | 1464010 | 2010-01-05 17:46:48 -0500 | [diff] [blame] | 50 | +|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(**) |
Masami Hiramatsu | 93ccae7 | 2010-04-12 13:17:08 -0400 | [diff] [blame] | 51 | NAME=FETCHARG : Set NAME as the argument name of FETCHARG. |
| 52 | FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types |
Masami Hiramatsu | 17ce3dc | 2016-08-18 17:57:50 +0900 | [diff] [blame] | 53 | (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types |
| 54 | (x8/x16/x32/x64), "string" and bitfield are supported. |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 55 | |
Masami Hiramatsu | 1464010 | 2010-01-05 17:46:48 -0500 | [diff] [blame] | 56 | (*) only for return probe. |
| 57 | (**) this is useful for fetching a field of data structures. |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 58 | |
Masami Hiramatsu | 1ff511e | 2011-02-04 21:52:05 +0900 | [diff] [blame] | 59 | Types |
| 60 | ----- |
| 61 | Several types are supported for fetch-args. Kprobe tracer will access memory |
| 62 | by given type. Prefix 's' and 'u' means those types are signed and unsigned |
Masami Hiramatsu | bdca79c2 | 2016-08-18 17:59:21 +0900 | [diff] [blame] | 63 | respectively. 'x' prefix implies it is unsigned. Traced arguments are shown |
| 64 | in decimal ('s' and 'u') or hexadecimal ('x'). Without type casting, 'x32' |
| 65 | or 'x64' is used depends on the architecture (e.g. x86-32 uses x32, and |
| 66 | x86-64 uses x64). |
Masami Hiramatsu | 1ff511e | 2011-02-04 21:52:05 +0900 | [diff] [blame] | 67 | String type is a special type, which fetches a "null-terminated" string from |
| 68 | kernel space. This means it will fail and store NULL if the string container |
| 69 | has been paged out. |
| 70 | Bitfield is another special type, which takes 3 parameters, bit-width, bit- |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 71 | offset, and container-size (usually 32). The syntax is:: |
Masami Hiramatsu | 1ff511e | 2011-02-04 21:52:05 +0900 | [diff] [blame] | 72 | |
| 73 | b<bit-width>@<bit-offset>/<container-size> |
| 74 | |
Omar Sandoval | 35abb67 | 2016-06-08 18:38:02 -0700 | [diff] [blame] | 75 | For $comm, the default type is "string"; any other type is invalid. |
| 76 | |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 77 | |
| 78 | Per-Probe Event Filtering |
| 79 | ------------------------- |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 80 | Per-probe event filtering feature allows you to set different filter on each |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 81 | probe and gives you what arguments will be shown in trace buffer. If an event |
Masami Hiramatsu | 77b44d1 | 2009-11-03 19:12:47 -0500 | [diff] [blame] | 82 | name is specified right after 'p:' or 'r:' in kprobe_events, it adds an event |
| 83 | under tracing/events/kprobes/<EVENT>, at the directory you can see 'id', |
Andreas Ziegler | 31130c8 | 2018-07-16 13:05:58 +0200 | [diff] [blame] | 84 | 'enable', 'format', 'filter' and 'trigger'. |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 85 | |
Andreas Ziegler | e50891d | 2018-07-16 13:05:57 +0200 | [diff] [blame] | 86 | enable: |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 87 | You can enable/disable the probe by writing 1 or 0 on it. |
| 88 | |
| 89 | format: |
Masami Hiramatsu | eca0d91 | 2009-09-10 19:53:38 -0400 | [diff] [blame] | 90 | This shows the format of this probe event. |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 91 | |
| 92 | filter: |
Masami Hiramatsu | eca0d91 | 2009-09-10 19:53:38 -0400 | [diff] [blame] | 93 | You can write filtering rules of this event. |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 94 | |
Masami Hiramatsu | e08d1c6 | 2009-09-10 19:53:30 -0400 | [diff] [blame] | 95 | id: |
| 96 | This shows the id of this probe event. |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 97 | |
Andreas Ziegler | 31130c8 | 2018-07-16 13:05:58 +0200 | [diff] [blame] | 98 | trigger: |
| 99 | This allows to install trigger commands which are executed when the event is |
| 100 | hit (for details, see Documentation/trace/events.rst, section 6). |
Masami Hiramatsu | 77b44d1 | 2009-11-03 19:12:47 -0500 | [diff] [blame] | 101 | |
Masami Hiramatsu | cd7e7bd | 2009-08-13 16:35:42 -0400 | [diff] [blame] | 102 | Event Profiling |
| 103 | --------------- |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 104 | You can check the total number of probe hits and probe miss-hits via |
Masami Hiramatsu | cd7e7bd | 2009-08-13 16:35:42 -0400 | [diff] [blame] | 105 | /sys/kernel/debug/tracing/kprobe_profile. |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 106 | The first column is event name, the second is the number of probe hits, |
Masami Hiramatsu | cd7e7bd | 2009-08-13 16:35:42 -0400 | [diff] [blame] | 107 | the third is the number of probe miss-hits. |
| 108 | |
| 109 | |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 110 | Usage examples |
| 111 | -------------- |
| 112 | To add a probe as a new event, write a new definition to kprobe_events |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 113 | as below:: |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 114 | |
Motohiro KOSAKI | 580d9e0 | 2010-01-18 21:35:05 -0500 | [diff] [blame] | 115 | echo 'p:myprobe do_sys_open dfd=%ax filename=%dx flags=%cx mode=+4($stack)' > /sys/kernel/debug/tracing/kprobe_events |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 116 | |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 117 | This sets a kprobe on the top of do_sys_open() function with recording |
Masami Hiramatsu | 1464010 | 2010-01-05 17:46:48 -0500 | [diff] [blame] | 118 | 1st to 4th arguments as "myprobe" event. Note, which register/stack entry is |
| 119 | assigned to each function argument depends on arch-specific ABI. If you unsure |
| 120 | the ABI, please try to use probe subcommand of perf-tools (you can find it |
| 121 | under tools/perf/). |
| 122 | As this example shows, users can choose more familiar names for each arguments. |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 123 | :: |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 124 | |
Motohiro KOSAKI | 580d9e0 | 2010-01-18 21:35:05 -0500 | [diff] [blame] | 125 | echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 126 | |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 127 | This sets a kretprobe on the return point of do_sys_open() function with |
Masami Hiramatsu | 99329c4 | 2009-10-07 18:27:48 -0400 | [diff] [blame] | 128 | recording return value as "myretprobe" event. |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 129 | You can see the format of these events via |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 130 | /sys/kernel/debug/tracing/events/kprobes/<EVENT>/format. |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 131 | :: |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 132 | |
| 133 | cat /sys/kernel/debug/tracing/events/kprobes/myprobe/format |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 134 | name: myprobe |
| 135 | ID: 780 |
| 136 | format: |
| 137 | field:unsigned short common_type; offset:0; size:2; signed:0; |
| 138 | field:unsigned char common_flags; offset:2; size:1; signed:0; |
| 139 | field:unsigned char common_preempt_count; offset:3; size:1;signed:0; |
| 140 | field:int common_pid; offset:4; size:4; signed:1; |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 141 | |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 142 | field:unsigned long __probe_ip; offset:12; size:4; signed:0; |
| 143 | field:int __probe_nargs; offset:16; size:4; signed:1; |
| 144 | field:unsigned long dfd; offset:20; size:4; signed:0; |
| 145 | field:unsigned long filename; offset:24; size:4; signed:0; |
| 146 | field:unsigned long flags; offset:28; size:4; signed:0; |
| 147 | field:unsigned long mode; offset:32; size:4; signed:0; |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 148 | |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 149 | |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 150 | print fmt: "(%lx) dfd=%lx filename=%lx flags=%lx mode=%lx", REC->__probe_ip, |
| 151 | REC->dfd, REC->filename, REC->flags, REC->mode |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 152 | |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 153 | You can see that the event has 4 arguments as in the expressions you specified. |
| 154 | :: |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 155 | |
| 156 | echo > /sys/kernel/debug/tracing/kprobe_events |
| 157 | |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 158 | This clears all probe points. |
Masami Hiramatsu | 5a0d905 | 2009-09-14 16:49:37 -0400 | [diff] [blame] | 159 | |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 160 | Or, |
| 161 | :: |
Motohiro KOSAKI | df3ab70 | 2010-01-18 21:35:12 -0500 | [diff] [blame] | 162 | |
| 163 | echo -:myprobe >> kprobe_events |
| 164 | |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 165 | This clears probe points selectively. |
Motohiro KOSAKI | df3ab70 | 2010-01-18 21:35:12 -0500 | [diff] [blame] | 166 | |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 167 | Right after definition, each event is disabled by default. For tracing these |
Masami Hiramatsu | 5a0d905 | 2009-09-14 16:49:37 -0400 | [diff] [blame] | 168 | events, you need to enable it. |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 169 | :: |
Masami Hiramatsu | 5a0d905 | 2009-09-14 16:49:37 -0400 | [diff] [blame] | 170 | |
| 171 | echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable |
| 172 | echo 1 > /sys/kernel/debug/tracing/events/kprobes/myretprobe/enable |
| 173 | |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 174 | And you can see the traced information via /sys/kernel/debug/tracing/trace. |
| 175 | :: |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 176 | |
| 177 | cat /sys/kernel/debug/tracing/trace |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 178 | # tracer: nop |
| 179 | # |
| 180 | # TASK-PID CPU# TIMESTAMP FUNCTION |
| 181 | # | | | | | |
| 182 | <...>-1447 [001] 1038282.286875: myprobe: (do_sys_open+0x0/0xd6) dfd=3 filename=7fffd1ec4440 flags=8000 mode=0 |
| 183 | <...>-1447 [001] 1038282.286878: myretprobe: (sys_openat+0xc/0xe <- do_sys_open) $retval=fffffffffffffffe |
| 184 | <...>-1447 [001] 1038282.286885: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=40413c flags=8000 mode=1b6 |
| 185 | <...>-1447 [001] 1038282.286915: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3 |
| 186 | <...>-1447 [001] 1038282.286969: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=4041c6 flags=98800 mode=10 |
| 187 | <...>-1447 [001] 1038282.286976: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3 |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 188 | |
| 189 | |
Changbin Du | 263ee77 | 2018-02-17 13:39:39 +0800 | [diff] [blame] | 190 | Each line shows when the kernel hits an event, and <- SYMBOL means kernel |
Masami Hiramatsu | d8ec918 | 2009-08-19 21:13:57 +0200 | [diff] [blame] | 191 | returns from SYMBOL(e.g. "sys_open+0x1b/0x1d <- do_sys_open" means kernel |
| 192 | returns from do_sys_open to sys_open+0x1b). |
| 193 | |