nexus:~# ttutils set -p MEM_SWAPOUT -t loop -c 10 [crash point set] name:MEM_SWAPOUT type:loop count:10 nexus:~# ./dump_tools_analysis/helper/brk TRIG_LOOP oom-killer: gfp_mask=0x201d2, order=0 Mem-info: DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 used:2 cpu 0 cold: low 0, high 2, batch 1 used:1 cpu 1 hot: low 2, high 6, batch 1 used:0 cpu 1 cold: low 0, high 2, batch 1 used:0 cpu 2 hot: low 2, high 6, batch 1 used:2 cpu 2 cold: low 0, high 2, batch 1 used:1 cpu 3 hot: low 2, high 6, batch 1 used:2 cpu 3 cold: low 0, high 2, batch 1 used:1 Normal per-cpu: cpu 0 hot: low 62, high 186, batch 31 used:126 cpu 0 cold: low 0, high 62, batch 31 used:43 cpu 1 hot: low 62, high 186, batch 31 used:63 cpu 1 cold: low 0, high 62, batch 31 used:30 cpu 2 hot: low 62, high 186, batch 31 used:93 cpu 2 cold: low 0, high 62, batch 31 used:48 cpu 3 hot: low 62, high 186, batch 31 used:92 cpu 3 cold: low 0, high 62, batch 31 used:40 HighMem per-cpu: cpu 0 hot: low 30, high 90, batch 15 used:44 cpu 0 cold: low 0, high 30, batch 15 used:17 cpu 1 hot: low 30, high 90, batch 15 used:48 cpu 1 cold: low 0, high 30, batch 15 used:18 cpu 2 hot: low 30, high 90, batch 15 used:43 cpu 2 cold: low 0, high 30, batch 15 used:26 cpu 3 hot: low 30, high 90, batch 15 used:44 cpu 3 cold: low 0, high 30, batch 15 used:23 Free pages: 11936kB (120kB HighMem) Active:121419 inactive:121133 dirty:0 writeback:0 unstable:0 free:2984 slab:1815 mapped:242270 pagetables:565 DMA free:4092kB min:68kB low:84kB high:100kB active:4560kB inactive:4112kB present:16384kB pages_scanned:21630 all_unreclaimable? no lowmem_reserve[]: 0 880 1006 Normal free:7724kB min:3756kB low:4692kB high:5632kB active:417872kB inactive:417720kB present:901120kB pages_scanned:1325196 all_unreclaimable? no lowmem_reserve[]: 0 0 1015 HighMem free:120kB min:128kB low:160kB high:192kB active:63244kB inactive:62700kB present:129920kB pages_scanned:123328 all_unreclaimable? no lowmem_reserve[]: 0 0 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4092kB Normal: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7724kB HighMem: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 120kB Swap cache: add 559001, delete 558987, find 176/311, race 1+0 Free swap = 0kB Total swap = 1076312kB Free swap: 0kB 261856 pages of RAM 32480 pages of HIGHMEM 11366 reserved pages 345 pages shared 14 pages swap cached 0 pages dirty 0 pages writeback 242270 pages mapped 1815 pages slab 565 pages pagetables Out of Memory: Killed process 2241 (brk). oom-killer: gfp_mask=0xd2, order=0 Mem-info: DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 used:2 cpu 0 cold: low 0, high 2, batch 1 used:1 cpu 1 hot: low 2, high 6, batch 1 used:0 cpu 1 cold: low 0, high 2, batch 1 used:0 cpu 2 hot: low 2, high 6, batch 1 used:2 cpu 2 cold: low 0, high 2, batch 1 used:1 cpu 3 hot: low 2, high 6, batch 1 used:2 cpu 3 cold: low 0, high 2, batch 1 used:1 Normal per-cpu: cpu 0 hot: low 62, high 186, batch 31 used:126 cpu 0 cold: low 0, high 62, batch 31 used:43 cpu 1 hot: low 62, high 186, batch 31 used:63 cpu 1 cold: low 0, high 62, batch 31 used:30 cpu 2 hot: low 62, high 186, batch 31 used:93 cpu 2 cold: low 0, high 62, batch 31 used:48 cpu 3 hot: low 62, high 186, batch 31 used:92 cpu 3 cold: low 0, high 62, batch 31 used:40 HighMem per-cpu: cpu 0 hot: low 30, high 90, batch 15 used:44 cpu 0 cold: low 0, high 30, batch 15 used:17 cpu 1 hot: low 30, high 90, batch 15 used:48 cpu 1 cold: low 0, high 30, batch 15 used:18 cpu 2 hot: low 30, high 90, batch 15 used:43 cpu 2 cold: low 0, high 30, batch 15 used:27 cpu 3 hot: low 30, high 90, batch 15 used:44 cpu 3 cold: low 0, high 30, batch 15 used:23 Free pages: 11936kB (120kB HighMem) Active:121386 inactive:121165 dirty:0 writeback:0 unstable:0 free:2984 slab:1815 mapped:242269 pagetables:565 DMA free:4092kB min:68kB low:84kB high:100kB active:4656kB inactive:4016kB present:16384kB pages_scanned:27022 all_unreclaimable? no lowmem_reserve[]: 0 880 1006 Normal free:7724kB min:3756kB low:4692kB high:5632kB active:417852kB inactive:417740kB present:901120kB pages_scanned:1822719 all_unreclaimable? no lowmem_reserve[]: 0 0 1015 HighMem free:120kB min:128kB low:160kB high:192kB active:63128kB inactive:62812kB present:129920kB pages_scanned:218799 all_unreclaimable? no lowmem_reserve[]: 0 0 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4092kB Normal: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7724kB HighMem: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 120kB Swap cache: add 559001, delete 558987, find 176/311, race 1+0 Free swap = 0kB Total swap = 1076312kB Free swap: 0kB 261856 pages of RAM 32480 pages of HIGHMEM 11366 reserved pages 460 pages shared 14 pages swap cached 0 pages dirty 0 pages writeback oom-killer: gfp_mask=0x201d2, order=0 Mem-info: DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 used:2 cpu 0 cold: low 0, high 2, batch 1 used:1 cpu 1 hot: low 2, high 6, batch 1 used:0 cpu 1 cold: low 0, high 2, batch 1 used:0 cpu 2 hot: low 2, high 6, batch 1 used:2 cpu 2 cold: low 0, high 2, batch 1 used:1 cpu 3 hot: low 2, high 6, batch 1 used:2 cpu 3 cold: low 0, high 2, batch 1 used:1 Normal per-cpu: cpu 0 hot: low 62, high 186, batch 31 used:126 cpu 0 cold: low 0, high 62, batch 31 used:43 cpu 1 hot: low 62, high 186, batch 31 used:63 cpu 1 cold: low 0, high 62, batch 31 used:30 cpu 2 hot: low 62, high 186, batch 31 used:93 cpu 2 cold: low 0, high 62, batch 31 used:48 cpu 3 hot: low 62, high 186, batch 31 used:92 cpu 3 cold: low 0, high 62, batch 31 used:40 HighMem per-cpu: cpu 0 hot: low 30, high 90, batch 15 used:44 cpu 0 cold: low 0, high 30, batch 15 used:17 cpu 1 hot: low 30, high 90, batch 15 used:48 cpu 1 cold: low 0, high 30, batch 15 used:18 cpu 2 hot: low 30, high 90, batch 15 used:43 cpu 2 cold: low 0, high 30, batch 15 used:27 cpu 3 hot: low 30, high 90, batch 15 used:44 cpu 3 cold: low 0, high 30, batch 15 used:23 Free pages: 11936kB (120kB HighMem) Active:121282 inactive:121205 dirty:0 writeback:0 unstable:0 free:2984 slab:1815 mapped:242269 pagetables:565 DMA free:4092kB min:68kB low:84kB high:100kB active:4412kB inactive:4260kB present:16384kB pages_scanned:33623 all_unreclaimable? no lowmem_reserve[]: 0 880 1006 Normal free:7724kB min:3756kB low:4692kB high:5632kB active:417820kB inactive:417644kB present:901120kB pages_scanned:2623607 all_unreclaimable? no lowmem_reserve[]: 0 0 1015 HighMem free:120kB min:128kB low:160kB high:192kB active:62896kB inactive:62916kB present:129920kB pages_scanned:324562 all_unreclaimable? no lowmem_reserve[]: 0 0 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4092kB Normal: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7724kB HighMem: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 120kB Swap cache: add 559001, delete 558987, find 176/311, race 1+0 Free swap = 0kB Total swap = 1076312kB Free swap: 0kB 261856 pages of RAM 32480 pages of HIGHMEM 11366 reserved pages 404 pages shared 14 pages swap cached 0 pages dirty 0 pages writeback 242269 pages mapped 1815 pages slab 565 pages pagetables oom-killer: gfp_mask=0xd2, order=0 Mem-info: DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 used:2 cpu 0 cold: low 0, high 2, batch 1 used:1 cpu 1 hot: low 2, high 6, batch 1 used:0 cpu 1 cold: low 0, high 2, batch 1 used:0 cpu 2 hot: low 2, high 6, batch 1 used:2 cpu 2 cold: low 0, high 2, batch 1 used:1 cpu 3 hot: low 2, high 6, batch 1 used:2 cpu 3 cold: low 0, high 2, batch 1 used:1 Normal per-cpu: cpu 0 hot: low 62, high 186, batch 31 used:132 cpu 0 cold: low 0, high 62, batch 31 used:43 cpu 1 hot: low 62, high 186, batch 31 used:63 cpu 1 cold: low 0, high 62, batch 31 used:30 cpu 2 hot: low 62, high 186, batch 31 used:93 cpu 2 cold: low 0, high 62, batch 31 used:48 cpu 3 hot: low 62, high 186, batch 31 used:92 cpu 3 cold: low 0, high 62, batch 31 used:40 HighMem per-cpu: cpu 0 hot: low 30, high 90, batch 15 used:44 cpu 0 cold: low 0, high 30, batch 15 used:17 cpu 1 hot: low 30, high 90, batch 15 used:48 cpu 1 cold: low 0, high 30, batch 15 used:18 cpu 2 hot: low 30, high 90, batch 15 used:43 cpu 2 cold: low 0, high 30, batch 15 used:27 cpu 3 hot: low 30, high 90, batch 15 used:44 cpu 3 cold: low 0, high 30, batch 15 used:23 Free pages: 11936kB (120kB HighMem) Active:121030 inactive:121489 dirty:0 writeback:0 unstable:0 free:2984 slab:1809 mapped:242269 pagetables:565 DMA free:4092kB min:68kB low:84kB high:100kB active:4220kB inactive:4452kB present:16384kB pages_scanned:39283 all_unreclaimable? no lowmem_reserve[]: 0 880 1006 Normal free:7724kB min:3756kB low:4692kB high:5632kB active:416928kB inactive:418536kB present:901120kB pages_scanned:3118840 all_unreclaimable? no lowmem_reserve[]: 0 0 1015 HighMem free:120kB min:128kB low:160kB high:192kB active:62972kB inactive:62968kB present:129920kB pages_scanned:405757 all_unreclaimable? no lowmem_reserve[]: 0 0 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4092kB Normal: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7724kB HighMem: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 120kB Swap cache: add 559001, delete 558987, find 176/311, race 1+0 Free swap = 0kB Total swap = 1076312kB Free swap: 0kB 261856 pages of RAM 32480 pages of HIGHMEM 11366 reserved pages 409 pages shared 14 pages swap cached 0 pages dirty 0 pages writeback 242269 pages mapped 1809 pages slab 565 pages pagetables oom-killer: gfp_mask=0x201d2, order=0 Mem-info: DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 used:2 cpu 0 cold: low 0, high 2, batch 1 used:1 cpu 1 hot: low 2, high 6, batch 1 used:0 cpu 1 cold: low 0, high 2, batch 1 used:0 cpu 2 hot: low 2, high 6, batch 1 used:2 cpu 2 cold: low 0, high 2, batch 1 used:1 cpu 3 hot: low 2, high 6, batch 1 used:2 cpu 3 cold: low 0, high 2, batch 1 used:1 Normal per-cpu: cpu 0 hot: low 62, high 186, batch 31 used:132 cpu 0 cold: low 0, high 62, batch 31 used:43 cpu 1 hot: low 62, high 186, batch 31 used:63 cpu 1 cold: low 0, high 62, batch 31 used:30 cpu 2 hot: low 62, high 186, batch 31 used:93 cpu 2 cold: low 0, high 62, batch 31 used:48 cpu 3 hot: low 62, high 186, batch 31 used:92 cpu 3 cold: low 0, high 62, batch 31 used:40 HighMem per-cpu: cpu 0 hot: low 30, high 90, batch 15 used:44 cpu 0 cold: low 0, high 30, batch 15 used:17 cpu 1 hot: low 30, high 90, batch 15 used:48 cpu 1 cold: low 0, high 30, batch 15 used:18 cpu 2 hot: low 30, high 90, batch 15 used:43 cpu 2 cold: low 0, high 30, batch 15 used:27 cpu 3 hot: low 30, high 90, batch 15 used:44 cpu 3 cold: low 0, high 30, batch 15 used:23 Free pages: 11936kB (120kB HighMem) Active:121240 inactive:121311 dirty:0 writeback:0 unstable:0 free:2984 slab:1809 mapped:242269 pagetables:565 DMA free:4092kB min:68kB low:84kB high:100kB active:4576kB inactive:4096kB present:16384kB pages_scanned:41568 all_unreclaimable? no lowmem_reserve[]: 0 880 1006 Normal free:7724kB min:3756kB low:4692kB high:5632kB active:417384kB inactive:418208kB present:901120kB pages_scanned:3198618 all_unreclaimable? no lowmem_reserve[]: 0 0 1015 HighMem free:120kB min:128kB low:160kB high:192kB active:63000kB inactive:62940kB present:129920kB pages_scanned:406014 all_unreclaimable? no lowmem_reserve[]: 0 0 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4092kB Normal: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7724kB HighMem: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 120kB Swap cache: add 559001, delete 558987, find 176/311, race 1+0 Free swap = 0kB Total swap = 1076312kB Free swap: 0kB 261856 pages of RAM 32480 pages of HIGHMEM 11366 reserved pages 344 pages shared 14 pages swap cached 0 pages dirty 0 pages writeback 242269 pages mapped 1809 pages slab 565 pages pagetables oom-killer: gfp_mask=0x201d2, order=0 Mem-info: DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 used:2 cpu 0 cold: low 0, high 2, batch 1 used:1 cpu 1 hot: low 2, high 6, batch 1 used:0 cpu 1 cold: low 0, high 2, batch 1 used:0 cpu 2 hot: low 2, high 6, batch 1 used:2 cpu 2 cold: low 0, high 2, batch 1 used:1 cpu 3 hot: low 2, high 6, batch 1 used:2 cpu 3 cold: low 0, high 2, batch 1 used:1 Normal per-cpu: cpu 0 hot: low 62, high 186, batch 31 used:132 cpu 0 cold: low 0, high 62, batch 31 used:43 cpu 1 hot: low 62, high 186, batch 31 used:63 cpu 1 cold: low 0, high 62, batch 31 used:30 cpu 2 hot: low 62, high 186, batch 31 used:93 cpu 2 cold: low 0, high 62, batch 31 used:48 cpu 3 hot: low 62, high 186, batch 31 used:92 cpu 3 cold: low 0, high 62, batch 31 used:40 HighMem per-cpu: cpu 0 hot: low 30, high 90, batch 15 used:44 cpu 0 cold: low 0, high 30, batch 15 used:17 cpu 1 hot: low 30, high 90, batch 15 used:48 cpu 1 cold: low 0, high 30, batch 15 used:18 cpu 2 hot: low 30, high 90, batch 15 used:43 cpu 2 cold: low 0, high 30, batch 15 used:27 cpu 3 hot: low 30, high 90, batch 15 used:44 cpu 3 cold: low 0, high 30, batch 15 used:23 Free pages: 11936kB (120kB HighMem) Active:121371 inactive:121148 dirty:0 writeback:0 unstable:0 free:2984 slab:1809 mapped:242269 pagetables:565 DMA free:4092kB min:68kB low:84kB high:100kB active:4656kB inactive:4016kB present:16384kB pages_scanned:41680 all_unreclaimable? no lowmem_reserve[]: 0 880 1006 Normal free:7724kB min:3756kB low:4692kB high:5632kB active:417740kB inactive:417724kB present:901120kB pages_scanned:3205888 all_unreclaimable? no lowmem_reserve[]: 0 0 1015 HighMem free:120kB min:128kB low:160kB high:192kB active:63088kB inactive:62852kB present:129920kB pages_scanned:407180 all_unreclaimable? no lowmem_reserve[]: 0 0 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4092kB Normal: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7724kB HighMem: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 120kB Swap cache: add 559001, delete 558987, find 176/311, race 1+0 Free swap = 0kB Total swap = 1076312kB Free swap: 0kB 261856 pages of RAM 32480 pages of HIGHMEM 11366 reserved pages 344 pages shared 14 pages swap cached 0 pages dirty 0 pages writeback 242269 pages mapped 1809 pages slab 565 pages pagetables 242269 pages mapped 1815 pages slab 565 pages pagetables Out of Memory: Killed process 2240 (memdrain). Broadcast message from root (console) (Fri Aug 26 14:45:21 2005): The system is going down for reboot NOW!