blob: 8bfc75c908061710ec3e406c7497000b08a80050 [file] [log] [blame]
Changbin Du263ee772018-02-17 13:39:39 +08001==========================
2Kprobe-based Event Tracing
3==========================
Masami Hiramatsud8ec9182009-08-19 21:13:57 +02004
Changbin Du263ee772018-02-17 13:39:39 +08005:Author: Masami Hiramatsu
Masami Hiramatsud8ec9182009-08-19 21:13:57 +02006
7Overview
8--------
Masami Hiramatsu77b44d12009-11-03 19:12:47 -05009These events are similar to tracepoint based events. Instead of Tracepoint,
10this is based on kprobes (kprobe and kretprobe). So it can probe wherever
Naveen N. Raoc1ac0942017-03-08 22:34:14 +053011kprobes can probe (this means, all functions except those with
12__kprobes/nokprobe_inline annotation and those marked NOKPROBE_SYMBOL).
13Unlike the Tracepoint based event, this can be added and removed
Masami Hiramatsu77b44d12009-11-03 19:12:47 -050014dynamically, on the fly.
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020015
Anton Blanchard6b0b7552017-02-16 17:00:50 +110016To enable this feature, build your kernel with CONFIG_KPROBE_EVENTS=y.
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020017
Masami Hiramatsu77b44d12009-11-03 19:12:47 -050018Similar to the events tracer, this doesn't need to be activated via
19current_tracer. Instead of that, add probe points via
20/sys/kernel/debug/tracing/kprobe_events, and enable it via
Andreas Zieglere50891d2018-07-16 13:05:57 +020021/sys/kernel/debug/tracing/events/kprobes/<EVENT>/enable.
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020022
23
24Synopsis of kprobe_events
25-------------------------
Changbin Du263ee772018-02-17 13:39:39 +080026::
27
Masami Hiramatsu61424312011-06-27 16:26:56 +090028 p[:[GRP/]EVENT] [MOD:]SYM[+offs]|MEMADDR [FETCHARGS] : Set a probe
Alban Crequy696ced42017-04-03 12:36:22 +020029 r[MAXACTIVE][:[GRP/]EVENT] [MOD:]SYM[+0] [FETCHARGS] : Set a return probe
Motohiro KOSAKIdf3ab702010-01-18 21:35:12 -050030 -:[GRP/]EVENT : Clear a probe
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020031
Masami Hiramatsuf52487e2009-09-10 19:53:53 -040032 GRP : Group name. If omitted, use "kprobes" for it.
Masami Hiramatsu2fba0c82009-09-10 19:53:14 -040033 EVENT : Event name. If omitted, the event name is generated
Masami Hiramatsu61424312011-06-27 16:26:56 +090034 based on SYM+offs or MEMADDR.
35 MOD : Module name which has given SYM.
36 SYM[+offs] : Symbol+offset where the probe is inserted.
Masami Hiramatsu2fba0c82009-09-10 19:53:14 -040037 MEMADDR : Address where the probe is inserted.
Alban Crequy696ced42017-04-03 12:36:22 +020038 MAXACTIVE : Maximum number of instances of the specified function that
39 can be probed simultaneously, or 0 for the default value
40 as defined in Documentation/kprobes.txt section 1.3.1.
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020041
Masami Hiramatsu2fba0c82009-09-10 19:53:14 -040042 FETCHARGS : Arguments. Each probe can have up to 128 args.
Masami Hiramatsu2e06ff62009-10-07 18:27:59 -040043 %REG : Fetch register REG
44 @ADDR : Fetch memory at ADDR (ADDR should be in kernel)
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020045 @SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol)
Masami Hiramatsu2e06ff62009-10-07 18:27:59 -040046 $stackN : Fetch Nth entry of stack (N >= 0)
47 $stack : Fetch stack address.
Masami Hiramatsu14640102010-01-05 17:46:48 -050048 $retval : Fetch return value.(*)
Omar Sandoval35abb672016-06-08 18:38:02 -070049 $comm : Fetch current task comm.
Masami Hiramatsu14640102010-01-05 17:46:48 -050050 +|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(**)
Masami Hiramatsu93ccae72010-04-12 13:17:08 -040051 NAME=FETCHARG : Set NAME as the argument name of FETCHARG.
52 FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types
Masami Hiramatsu17ce3dc2016-08-18 17:57:50 +090053 (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types
54 (x8/x16/x32/x64), "string" and bitfield are supported.
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020055
Masami Hiramatsu14640102010-01-05 17:46:48 -050056 (*) only for return probe.
57 (**) this is useful for fetching a field of data structures.
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020058
Masami Hiramatsu1ff511e2011-02-04 21:52:05 +090059Types
60-----
61Several types are supported for fetch-args. Kprobe tracer will access memory
62by given type. Prefix 's' and 'u' means those types are signed and unsigned
Masami Hiramatsubdca79c22016-08-18 17:59:21 +090063respectively. 'x' prefix implies it is unsigned. Traced arguments are shown
64in decimal ('s' and 'u') or hexadecimal ('x'). Without type casting, 'x32'
65or 'x64' is used depends on the architecture (e.g. x86-32 uses x32, and
66x86-64 uses x64).
Masami Hiramatsu1ff511e2011-02-04 21:52:05 +090067String type is a special type, which fetches a "null-terminated" string from
68kernel space. This means it will fail and store NULL if the string container
69has been paged out.
70Bitfield is another special type, which takes 3 parameters, bit-width, bit-
Changbin Du263ee772018-02-17 13:39:39 +080071offset, and container-size (usually 32). The syntax is::
Masami Hiramatsu1ff511e2011-02-04 21:52:05 +090072
73 b<bit-width>@<bit-offset>/<container-size>
74
Omar Sandoval35abb672016-06-08 18:38:02 -070075For $comm, the default type is "string"; any other type is invalid.
76
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020077
78Per-Probe Event Filtering
79-------------------------
Changbin Du263ee772018-02-17 13:39:39 +080080Per-probe event filtering feature allows you to set different filter on each
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020081probe and gives you what arguments will be shown in trace buffer. If an event
Masami Hiramatsu77b44d12009-11-03 19:12:47 -050082name is specified right after 'p:' or 'r:' in kprobe_events, it adds an event
83under tracing/events/kprobes/<EVENT>, at the directory you can see 'id',
Andreas Ziegler31130c82018-07-16 13:05:58 +020084'enable', 'format', 'filter' and 'trigger'.
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020085
Andreas Zieglere50891d2018-07-16 13:05:57 +020086enable:
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020087 You can enable/disable the probe by writing 1 or 0 on it.
88
89format:
Masami Hiramatsueca0d912009-09-10 19:53:38 -040090 This shows the format of this probe event.
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020091
92filter:
Masami Hiramatsueca0d912009-09-10 19:53:38 -040093 You can write filtering rules of this event.
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020094
Masami Hiramatsue08d1c62009-09-10 19:53:30 -040095id:
96 This shows the id of this probe event.
Masami Hiramatsud8ec9182009-08-19 21:13:57 +020097
Andreas Ziegler31130c82018-07-16 13:05:58 +020098trigger:
99 This allows to install trigger commands which are executed when the event is
100 hit (for details, see Documentation/trace/events.rst, section 6).
Masami Hiramatsu77b44d12009-11-03 19:12:47 -0500101
Masami Hiramatsucd7e7bd2009-08-13 16:35:42 -0400102Event Profiling
103---------------
Changbin Du263ee772018-02-17 13:39:39 +0800104You can check the total number of probe hits and probe miss-hits via
Masami Hiramatsucd7e7bd2009-08-13 16:35:42 -0400105/sys/kernel/debug/tracing/kprobe_profile.
Changbin Du263ee772018-02-17 13:39:39 +0800106The first column is event name, the second is the number of probe hits,
Masami Hiramatsucd7e7bd2009-08-13 16:35:42 -0400107the third is the number of probe miss-hits.
108
109
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200110Usage examples
111--------------
112To add a probe as a new event, write a new definition to kprobe_events
Changbin Du263ee772018-02-17 13:39:39 +0800113as below::
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200114
Motohiro KOSAKI580d9e02010-01-18 21:35:05 -0500115 echo 'p:myprobe do_sys_open dfd=%ax filename=%dx flags=%cx mode=+4($stack)' > /sys/kernel/debug/tracing/kprobe_events
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200116
Changbin Du263ee772018-02-17 13:39:39 +0800117This sets a kprobe on the top of do_sys_open() function with recording
Masami Hiramatsu14640102010-01-05 17:46:48 -05001181st to 4th arguments as "myprobe" event. Note, which register/stack entry is
119assigned to each function argument depends on arch-specific ABI. If you unsure
120the ABI, please try to use probe subcommand of perf-tools (you can find it
121under tools/perf/).
122As this example shows, users can choose more familiar names for each arguments.
Changbin Du263ee772018-02-17 13:39:39 +0800123::
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200124
Motohiro KOSAKI580d9e02010-01-18 21:35:05 -0500125 echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200126
Changbin Du263ee772018-02-17 13:39:39 +0800127This sets a kretprobe on the return point of do_sys_open() function with
Masami Hiramatsu99329c42009-10-07 18:27:48 -0400128recording return value as "myretprobe" event.
Changbin Du263ee772018-02-17 13:39:39 +0800129You can see the format of these events via
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200130/sys/kernel/debug/tracing/events/kprobes/<EVENT>/format.
Changbin Du263ee772018-02-17 13:39:39 +0800131::
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200132
133 cat /sys/kernel/debug/tracing/events/kprobes/myprobe/format
Changbin Du263ee772018-02-17 13:39:39 +0800134 name: myprobe
135 ID: 780
136 format:
137 field:unsigned short common_type; offset:0; size:2; signed:0;
138 field:unsigned char common_flags; offset:2; size:1; signed:0;
139 field:unsigned char common_preempt_count; offset:3; size:1;signed:0;
140 field:int common_pid; offset:4; size:4; signed:1;
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200141
Changbin Du263ee772018-02-17 13:39:39 +0800142 field:unsigned long __probe_ip; offset:12; size:4; signed:0;
143 field:int __probe_nargs; offset:16; size:4; signed:1;
144 field:unsigned long dfd; offset:20; size:4; signed:0;
145 field:unsigned long filename; offset:24; size:4; signed:0;
146 field:unsigned long flags; offset:28; size:4; signed:0;
147 field:unsigned long mode; offset:32; size:4; signed:0;
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200148
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200149
Changbin Du263ee772018-02-17 13:39:39 +0800150 print fmt: "(%lx) dfd=%lx filename=%lx flags=%lx mode=%lx", REC->__probe_ip,
151 REC->dfd, REC->filename, REC->flags, REC->mode
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200152
Changbin Du263ee772018-02-17 13:39:39 +0800153You can see that the event has 4 arguments as in the expressions you specified.
154::
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200155
156 echo > /sys/kernel/debug/tracing/kprobe_events
157
Changbin Du263ee772018-02-17 13:39:39 +0800158This clears all probe points.
Masami Hiramatsu5a0d9052009-09-14 16:49:37 -0400159
Changbin Du263ee772018-02-17 13:39:39 +0800160Or,
161::
Motohiro KOSAKIdf3ab702010-01-18 21:35:12 -0500162
163 echo -:myprobe >> kprobe_events
164
Changbin Du263ee772018-02-17 13:39:39 +0800165This clears probe points selectively.
Motohiro KOSAKIdf3ab702010-01-18 21:35:12 -0500166
Changbin Du263ee772018-02-17 13:39:39 +0800167Right after definition, each event is disabled by default. For tracing these
Masami Hiramatsu5a0d9052009-09-14 16:49:37 -0400168events, you need to enable it.
Changbin Du263ee772018-02-17 13:39:39 +0800169::
Masami Hiramatsu5a0d9052009-09-14 16:49:37 -0400170
171 echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
172 echo 1 > /sys/kernel/debug/tracing/events/kprobes/myretprobe/enable
173
Changbin Du263ee772018-02-17 13:39:39 +0800174And you can see the traced information via /sys/kernel/debug/tracing/trace.
175::
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200176
177 cat /sys/kernel/debug/tracing/trace
Changbin Du263ee772018-02-17 13:39:39 +0800178 # tracer: nop
179 #
180 # TASK-PID CPU# TIMESTAMP FUNCTION
181 # | | | | |
182 <...>-1447 [001] 1038282.286875: myprobe: (do_sys_open+0x0/0xd6) dfd=3 filename=7fffd1ec4440 flags=8000 mode=0
183 <...>-1447 [001] 1038282.286878: myretprobe: (sys_openat+0xc/0xe <- do_sys_open) $retval=fffffffffffffffe
184 <...>-1447 [001] 1038282.286885: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=40413c flags=8000 mode=1b6
185 <...>-1447 [001] 1038282.286915: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3
186 <...>-1447 [001] 1038282.286969: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=4041c6 flags=98800 mode=10
187 <...>-1447 [001] 1038282.286976: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200188
189
Changbin Du263ee772018-02-17 13:39:39 +0800190Each line shows when the kernel hits an event, and <- SYMBOL means kernel
Masami Hiramatsud8ec9182009-08-19 21:13:57 +0200191returns from SYMBOL(e.g. "sys_open+0x1b/0x1d <- do_sys_open" means kernel
192returns from do_sys_open to sys_open+0x1b).
193