| Keeping data small | |
| When many applets are compiled into busybox, all rw data and | |
| bss for each applet are concatenated. Including those from libc, | |
| if static busybox is built. When busybox is started, _all_ this data | |
| is allocated, not just that one part for selected applet. | |
| What "allocated" exactly means, depends on arch. | |
| On NOMMU it's probably bites the most, actually using real | |
| RAM for rwdata and bss. On i386, bss is lazily allocated | |
| by COWed zero pages. Not sure about rwdata - also COW? | |
| In order to keep busybox NOMMU and small-mem systems friendly | |
| we should avoid large global data in our applets, and should | |
| minimize usage of libc functions which implicitly use | |
| such structures. | |
| Small experiment to measure "parasitic" bbox memory consumption: | |
| here we start 1000 "busybox sleep 10" in parallel. | |
| busybox binary is practically allyesconfig static one, | |
| built against uclibc. Run on x86-64 machine with 64-bit kernel: | |
| bash-3.2# nmeter '%t %c %m %p %[pn]' | |
| 23:17:28 .......... 168M 0 147 | |
| 23:17:29 .......... 168M 0 147 | |
| 23:17:30 U......... 168M 1 147 | |
| 23:17:31 SU........ 181M 244 391 | |
| 23:17:32 SSSSUUU... 223M 757 1147 | |
| 23:17:33 UUU....... 223M 0 1147 | |
| 23:17:34 U......... 223M 1 1147 | |
| 23:17:35 .......... 223M 0 1147 | |
| 23:17:36 .......... 223M 0 1147 | |
| 23:17:37 S......... 223M 0 1147 | |
| 23:17:38 .......... 223M 1 1147 | |
| 23:17:39 .......... 223M 0 1147 | |
| 23:17:40 .......... 223M 0 1147 | |
| 23:17:41 .......... 210M 0 906 | |
| 23:17:42 .......... 168M 1 147 | |
| 23:17:43 .......... 168M 0 147 | |
| This requires 55M of memory. Thus 1 trivial busybox applet | |
| takes 55k of memory on 64-bit x86 kernel. | |
| On 32-bit kernel we need ~26k per applet. | |
| Script: | |
| i=1000; while test $i != 0; do | |
| echo -n . | |
| busybox sleep 30 & | |
| i=$((i - 1)) | |
| done | |
| echo | |
| wait | |
| (Data from NOMMU arches are sought. Provide 'size busybox' output too) | |
| Example 1 | |
| One example how to reduce global data usage is in | |
| archival/libarchive/decompress_gunzip.c: | |
| /* This is somewhat complex-looking arrangement, but it allows | |
| * to place decompressor state either in bss or in | |
| * malloc'ed space simply by changing #defines below. | |
| * Sizes on i386: | |
| * text data bss dec hex | |
| * 5256 0 108 5364 14f4 - bss | |
| * 4915 0 0 4915 1333 - malloc | |
| */ | |
| #define STATE_IN_BSS 0 | |
| #define STATE_IN_MALLOC 1 | |
| (see the rest of the file to get the idea) | |
| This example completely eliminates globals in that module. | |
| Required memory is allocated in unpack_gz_stream() [its main module] | |
| and then passed down to all subroutines which need to access 'globals' | |
| as a parameter. | |
| Example 2 | |
| In case you don't want to pass this additional parameter everywhere, | |
| take a look at archival/gzip.c. Here all global data is replaced by | |
| single global pointer (ptr_to_globals) to allocated storage. | |
| In order to not duplicate ptr_to_globals in every applet, you can | |
| reuse single common one. It is defined in libbb/ptr_to_globals.c | |
| as struct globals *const ptr_to_globals, but the struct globals is | |
| NOT defined in libbb.h. You first define your own struct: | |
| struct globals { int a; char buf[1000]; }; | |
| and then declare that ptr_to_globals is a pointer to it: | |
| #define G (*ptr_to_globals) | |
| ptr_to_globals is declared as constant pointer. | |
| This helps gcc understand that it won't change, resulting in noticeably | |
| smaller code. In order to assign it, use SET_PTR_TO_GLOBALS macro: | |
| SET_PTR_TO_GLOBALS(xzalloc(sizeof(G))); | |
| Typically it is done in <applet>_main(). Another variation is | |
| to use stack: | |
| int <applet>_main(...) | |
| { | |
| #undef G | |
| struct globals G; | |
| memset(&G, 0, sizeof(G)); | |
| SET_PTR_TO_GLOBALS(&G); | |
| Now you can reference "globals" by G.a, G.buf and so on, in any function. | |
| bb_common_bufsiz1 | |
| There is one big common buffer in bss - bb_common_bufsiz1. It is a much | |
| earlier mechanism to reduce bss usage. Each applet can use it for | |
| its needs. Library functions are prohibited from using it. | |
| 'G.' trick can be done using bb_common_bufsiz1 instead of malloced buffer: | |
| #define G (*(struct globals*)&bb_common_bufsiz1) | |
| Be careful, though, and use it only if globals fit into bb_common_bufsiz1. | |
| Since bb_common_bufsiz1 is BUFSIZ + 1 bytes long and BUFSIZ can change | |
| from one libc to another, you have to add compile-time check for it: | |
| if (sizeof(struct globals) > sizeof(bb_common_bufsiz1)) | |
| BUG_<applet>_globals_too_big(); | |
| Drawbacks | |
| You have to initialize it by hand. xzalloc() can be helpful in clearing | |
| allocated storage to 0, but anything more must be done by hand. | |
| All global variables are prefixed by 'G.' now. If this makes code | |
| less readable, use #defines: | |
| #define dev_fd (G.dev_fd) | |
| #define sector (G.sector) | |
| Finding non-shared duplicated strings | |
| strings busybox | sort | uniq -c | sort -nr | |
| gcc's data alignment problem | |
| The following attribute added in vi.c: | |
| static int tabstop; | |
| static struct termios term_orig __attribute__ ((aligned (4))); | |
| static struct termios term_vi __attribute__ ((aligned (4))); | |
| reduces bss size by 32 bytes, because gcc sometimes aligns structures to | |
| ridiculously large values. asm output diff for above example: | |
| tabstop: | |
| .zero 4 | |
| .section .bss.term_orig,"aw",@nobits | |
| - .align 32 | |
| + .align 4 | |
| .type term_orig, @object | |
| .size term_orig, 60 | |
| term_orig: | |
| .zero 60 | |
| .section .bss.term_vi,"aw",@nobits | |
| - .align 32 | |
| + .align 4 | |
| .type term_vi, @object | |
| .size term_vi, 60 | |
| gcc doesn't seem to have options for altering this behaviour. | |
| gcc 3.4.3 and 4.1.1 tested: | |
| char c = 1; | |
| // gcc aligns to 32 bytes if sizeof(struct) >= 32 | |
| struct { | |
| int a,b,c,d; | |
| int i1,i2,i3; | |
| } s28 = { 1 }; // struct will be aligned to 4 bytes | |
| struct { | |
| int a,b,c,d; | |
| int i1,i2,i3,i4; | |
| } s32 = { 1 }; // struct will be aligned to 32 bytes | |
| // same for arrays | |
| char vc31[31] = { 1 }; // unaligned | |
| char vc32[32] = { 1 }; // aligned to 32 bytes | |
| -fpack-struct=1 reduces alignment of s28 to 1 (but probably | |
| will break layout of many libc structs) but s32 and vc32 | |
| are still aligned to 32 bytes. | |
| I will try to cook up a patch to add a gcc option for disabling it. | |
| Meanwhile, this is where it can be disabled in gcc source: | |
| gcc/config/i386/i386.c | |
| int | |
| ix86_data_alignment (tree type, int align) | |
| { | |
| #if 0 | |
| if (AGGREGATE_TYPE_P (type) | |
| && TYPE_SIZE (type) | |
| && TREE_CODE (TYPE_SIZE (type)) == INTEGER_CST | |
| && (TREE_INT_CST_LOW (TYPE_SIZE (type)) >= 256 | |
| || TREE_INT_CST_HIGH (TYPE_SIZE (type))) && align < 256) | |
| return 256; | |
| #endif | |
| Result (non-static busybox built against glibc): | |
| # size /usr/srcdevel/bbox/fix/busybox.t0/busybox busybox | |
| text data bss dec hex filename | |
| 634416 2736 23856 661008 a1610 busybox | |
| 632580 2672 22944 658196 a0b14 busybox_noalign | |
| Keeping code small | |
| Use scripts/bloat-o-meter to check whether introduced changes | |
| didn't generate unnecessary bloat. This script needs unstripped binaries | |
| to generate a detailed report. To automate this, just use | |
| "make bloatcheck". It requires busybox_old binary to be present, | |
| use "make baseline" to generate it from unmodified source, or | |
| copy busybox_unstripped to busybox_old before modifying sources | |
| and rebuilding. | |
| Set CONFIG_EXTRA_CFLAGS="-fno-inline-functions-called-once", | |
| produce "make bloatcheck", see the biggest auto-inlined functions. | |
| Now, set CONFIG_EXTRA_CFLAGS back to "", but add NOINLINE | |
| to some of these functions. In 1.16.x timeframe, the results were | |
| (annotated "make bloatcheck" output): | |
| function old new delta | |
| expand_vars_to_list - 1712 +1712 win | |
| lzo1x_optimize - 1429 +1429 win | |
| arith_apply - 1326 +1326 win | |
| read_interfaces - 1163 +1163 loss, leave w/o NOINLINE | |
| logdir_open - 1148 +1148 win | |
| check_deps - 1148 +1148 loss | |
| rewrite - 1039 +1039 win | |
| run_pipe 358 1396 +1038 win | |
| write_status_file - 1029 +1029 almost the same, leave w/o NOINLINE | |
| dump_identity - 987 +987 win | |
| mainQSort3 - 921 +921 win | |
| parse_one_line - 916 +916 loss | |
| summarize - 897 +897 almost the same | |
| do_shm - 884 +884 win | |
| cpio_o - 863 +863 win | |
| subCommand - 841 +841 loss | |
| receive - 834 +834 loss | |
| 855 bytes saved in total. | |
| scripts/mkdiff_obj_bloat may be useful to automate this process: run | |
| "scripts/mkdiff_obj_bloat NORMALLY_BUILT_TREE FORCED_NOINLINE_TREE" | |
| and select modules which shrank. |