尽管我们将近一半的内存用于FS缓存,但我们已经遇到了OOM杀手.我们每分钟记录一次内存统计数据(如上图所示),但似乎有足够的可用性.
... Mem: 15339640k total,15268304k used,71336k free,3152k buffers Swap: 0k total,0k used,0k free,6608384k cached Mem: 15339640k total,14855280k used,484360k free,13748k buffers Swap: 0k total,6481852k cached [OOM killer: postgres killed] Mem: 15339640k total,8212200k used,7127440k free,32776k buffers Swap: 0k total,2394444k cached ...
来自syslog的OOM详细信息：
... Jun 10 05:45:25 db kernel: [11209156.840462] wal-e invoked oom-killer: gfp_mask=0x201da,order=0,oom_adj=0,oom_score_adj=0 Jun 10 05:45:25 db kernel: [11209156.840469] wal-e cpuset=/ mems_allowed=0 Jun 10 05:45:25 db kernel: [11209156.840474] Pid: 7963,comm: wal-e Not tainted 3.2.0-43-virtual #68-Ubuntu Jun 10 05:45:25 db kernel: [11209156.840477] Call Trace: Jun 10 05:45:25 db kernel: [11209156.840498] [<ffffffff81119711>] dump_header+0x91/0xe0 Jun 10 05:45:25 db kernel: [11209156.840502] [<ffffffff81119a95>] oom_kill_process+0x85/0xb0 Jun 10 05:45:25 db kernel: [11209156.840506] [<ffffffff81119e3a>] out_of_memory+0xfa/0x220 Jun 10 05:45:25 db kernel: [11209156.840511] [<ffffffff8111f823>] __alloc_pages_nodemask+0x8c3/0x8e0 Jun 10 05:45:25 db kernel: [11209156.840520] [<ffffffff81216e00>] ? noalloc_get_block_write+0x30/0x30 Jun 10 05:45:25 db kernel: [11209156.840528] [<ffffffff811566c6>] alloc_pages_current+0xb6/0x120 Jun 10 05:45:25 db kernel: [11209156.840534] [<ffffffff81116637>] __page_cache_alloc+0xb7/0xd0 Jun 10 05:45:25 db kernel: [11209156.840539] [<ffffffff81118602>] filemap_fault+0x212/0x3c0 Jun 10 05:45:25 db kernel: [11209156.840553] [<ffffffff81138c32>] __do_fault+0x72/0x550 Jun 10 05:45:25 db kernel: [11209156.840557] [<ffffffff8113c2ea>] handle_pte_fault+0xfa/0x200 Jun 10 05:45:25 db kernel: [11209156.840562] [<ffffffff8100638e>] ? xen_pmd_val+0xe/0x10 Jun 10 05:45:25 db kernel: [11209156.840567] [<ffffffff81005309>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e Jun 10 05:45:25 db kernel: [11209156.840571] [<ffffffff8113d559>] handle_mm_fault+0x269/0x370 Jun 10 05:45:25 db kernel: [11209156.840576] [<ffffffff8100a56d>] ? xen_force_evtchn_callback+0xd/0x10 Jun 10 05:45:25 db kernel: [11209156.840581] [<ffffffff8100ad42>] ? check_events+0x12/0x20 Jun 10 05:45:25 db kernel: [11209156.840589] [<ffffffff8165b3cb>] do_page_fault+0x14b/0x520 Jun 10 05:45:25 db kernel: [11209156.840594] [<ffffffff81160d64>] ? kmem_cache_free+0x104/0x110 Jun 10 05:45:25 db kernel: [11209156.840600] [<ffffffff811ba2c8>] ? ep_remove+0xa8/0xc0 Jun 10 05:45:25 db kernel: [11209156.840604] [<ffffffff811bb133>] ? sys_epoll_ctl+0xb3/0x3d0 Jun 10 05:45:25 db kernel: [11209156.840614] [<ffffffff81658035>] page_fault+0x25/0x30 Jun 10 05:45:25 db kernel: [11209156.840617] Mem-Info: Jun 10 05:45:25 db kernel: [11209156.840618] Node 0 DMA per-cpu: Jun 10 05:45:25 db kernel: [11209156.840622] cpu 0: hi: 0,btch: 1 usd: 0 Jun 10 05:45:25 db kernel: [11209156.840624] cpu 1: hi: 0,btch: 1 usd: 0 Jun 10 05:45:25 db kernel: [11209156.840627] cpu 2: hi: 0,btch: 1 usd: 0 Jun 10 05:45:25 db kernel: [11209156.840629] cpu 3: hi: 0,btch: 1 usd: 0 Jun 10 05:45:25 db kernel: [11209156.840631] Node 0 DMA32 per-cpu: Jun 10 05:45:25 db kernel: [11209156.840634] cpu 0: hi: 186,btch: 31 usd: 30 Jun 10 05:45:25 db kernel: [11209156.840637] cpu 1: hi: 186,btch: 31 usd: 47 Jun 10 05:45:25 db kernel: [11209156.840639] cpu 2: hi: 186,btch: 31 usd: 15 Jun 10 05:45:25 db kernel: [11209156.840641] cpu 3: hi: 186,btch: 31 usd: 2 Jun 10 05:45:25 db kernel: [11209156.840643] Node 0 Normal per-cpu: Jun 10 05:45:25 db kernel: [11209156.840646] cpu 0: hi: 186,btch: 31 usd: 0 Jun 10 05:45:25 db kernel: [11209156.840648] cpu 1: hi: 186,btch: 31 usd: 14 Jun 10 05:45:25 db kernel: [11209156.840650] cpu 2: hi: 186,btch: 31 usd: 0 Jun 10 05:45:25 db kernel: [11209156.840653] cpu 3: hi: 186,btch: 31 usd: 1 Jun 10 05:45:25 db kernel: [11209156.840658] active_anon:3616567 inactive_anon:4798 isolated_anon:0 Jun 10 05:45:25 db kernel: [11209156.840660] active_file:98 inactive_file:168 isolated_file:20 Jun 10 05:45:25 db kernel: [11209156.840661] unevictable:1597 dirty:73 writeback:0 unstable:0 Jun 10 05:45:25 db kernel: [11209156.840662] free:16921 slab_reclaimable:17631 slab_unreclaimable:7534 Jun 10 05:45:25 db kernel: [11209156.840663] mapped:1614529 shmem:1613928 pagetables:124012 bounce:0 Jun 10 05:45:25 db kernel: [11209156.840666] Node 0 DMA free:7888kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7632kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jun 10 05:45:25 db kernel: [11209156.840681] lowmem_reserve[]: 0 4016 15112 15112 Jun 10 05:45:25 db kernel: [11209156.840686] Node 0 DMA32 free:48368kB min:4176kB low:5220kB high:6264kB active_anon:3776804kB inactive_anon:28kB active_file:0kB inactive_file:20kB unevictable:932kB isolated(anon):0kB isolated(file):0kB present:4112640kB mlocked:932kB dirty:0kB writeback:0kB mapped:1458536kB shmem:1458632kB slab_reclaimable:17604kB slab_unreclaimable:8088kB kernel_stack:1872kB pagetables:190616kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:437 all_unreclaimable? yes Jun 10 05:45:25 db kernel: [11209156.840698] lowmem_reserve[]: 0 0 11095 11095 Jun 10 05:45:25 db kernel: [11209156.840703] Node 0 Normal free:11428kB min:11548kB low:14432kB high:17320kB active_anon:10689464kB inactive_anon:19164kB active_file:528kB inactive_file:652kB unevictable:5456kB isolated(anon):0kB isolated(file):80kB present:11362176kB mlocked:5456kB dirty:292kB writeback:0kB mapped:4999580kB shmem:4997080kB slab_reclaimable:52920kB slab_unreclaimable:22048kB kernel_stack:2584kB pagetables:305432kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1974 all_unreclaimable? yes Jun 10 05:45:25 db kernel: [11209156.840715] lowmem_reserve[]: 0 0 0 0 Jun 10 05:45:25 db kernel: [11209156.840720] Node 0 DMA: 2*4kB 3*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 2*2048kB 0*4096kB = 7888kB Jun 10 05:45:25 db kernel: [11209156.840752] Node 0 DMA32: 5813*4kB 2636*8kB 114*16kB 15*32kB 5*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 48372kB Jun 10 05:45:25 db kernel: [11209156.840776] Node 0 Normal: 1888*4kB 10*8kB 46*16kB 4*32kB 3*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 11760kB Jun 10 05:45:25 db kernel: [11209156.840788] 1615243 total pagecache pages Jun 10 05:45:25 db kernel: [11209156.840790] 0 pages in swap cache Jun 10 05:45:25 db kernel: [11209156.840801] Swap cache stats: add 0,delete 0,find 0/0 Jun 10 05:45:25 db kernel: [11209156.840803] Free swap = 0kB Jun 10 05:45:25 db kernel: [11209156.840805] Total swap = 0kB Jun 10 05:45:25 db kernel: [11209156.909794] 3934192 pages RAM Jun 10 05:45:25 db kernel: [11209156.909804] 99282 pages reserved Jun 10 05:45:25 db kernel: [11209156.909809] 18899146 pages shared Jun 10 05:45:25 db kernel: [11209156.909811] 2198511 pages non-shared Jun 10 05:45:25 db kernel: [11209156.909817] [ pid ] uid tgid total_vm RSS cpu oom_adj oom_score_adj name Jun 10 05:45:25 db kernel: [11209156.909835] [ 332] 0 332 4308 109 1 0 0 upstart-udev-br Jun 10 05:45:25 db kernel: [11209156.909845] [ 346] 0 346 5384 271 2 -17 -1000 udevd Jun 10 05:45:25 db kernel: [11209156.909851] [ 408] 0 408 5364 174 2 -17 -1000 udevd ... Jun 10 05:45:25 db kernel: [11209156.910703] [ 7963] 111 7963 17456 2966 0 0 0 wal-e Jun 10 05:45:25 db kernel: [11209156.910707] [ 7968] 111 7968 1639372 2351 3 0 0 postgres Jun 10 05:45:25 db kernel: [11209156.910711] [ 7969] 111 7969 1639371 1934 2 0 0 postgres Jun 10 05:45:25 db kernel: [11209156.910716] Out of memory: Kill process 12443 (postgres) score 418 or sacrifice child Jun 10 05:45:25 db kernel: [11209156.910733] Killed process 12443 (postgres) total-vm:6555152kB,anon-RSS:4600kB,file-RSS:6396572kB Jun 10 05:45:30 db kernel: [11209159.293083] postgres invoked oom-killer: gfp_mask=0x201da,oom_score_adj=0 Jun 10 05:45:31 db kernel: [11209159.293091] postgres cpuset=/ mems_allowed=0 Jun 10 05:45:31 db kernel: [11209159.293095] Pid: 6508,comm: postgres Not tainted 3.2.0-43-virtual #68-Ubuntu Jun 10 05:45:31 db kernel: [11209159.293098] Call Trace: Jun 10 05:45:31 db kernel: [11209159.293111] [<ffffffff81119711>] dump_header+0x91/0xe0 Jun 10 05:45:31 db kernel: [11209159.293115] [<ffffffff81119a95>] oom_kill_process+0x85/0xb0 Jun 10 05:45:31 db kernel: [11209159.293119] [<ffffffff81119e3a>] out_of_memory+0xfa/0x220 ...
我们可以尝试将这些分辨率提高到大约每秒一次,但是在这里有没有任何理由可以使用OOM？ (我们已经看到了http://bl0rg.krunch.be/oom-frag.html,但是我们正在使用更大的绝对内存量,其中大部分是内核的FS缓存.)
shared_buffers = 6GB effective_cache_size = 8GB
看起来你(我在症状非常相似的情况下)确实已经耗尽内存并且被缓存的数字搞糊涂了.
显然有Linux not freeing large disk cache when memory demand goes up的情况
特别是(我真的不明白为什么),可以在“缓存”(页面缓存)下报告postgres的shared_buffers.在你的情况下,缓存在顶部的6481852k匹配OOM杀手的日志中的这一行：
Jun 10 05:45:25 db kernel: [11209156.840788] 1615243 total pagecache pages
(1615243 * 4KB~ = 6481852k) – 意味着在调用OOM-killer之前,页面缓存确实没有丢弃.
然而,文件支持的页面很少(我假设active_file：98 inactive_file：168类似于/ proc / meminfo的Active(file)/Inactive(file)),所以它不是我们所知道和喜欢的可丢弃页面.
https://www.depesz.com/2012/06/09/how-much-ram-is-postgresql-using/上的帖子演示了一个示例会话,其中关闭postgres会导致“缓存”减少shared_buffers的大小(滚动到“大部分都来自磁盘缓存 – 正如预期的那样,因为它用于shared_buffers.”) – 不幸的是,它没有表明postgres的版本,也没有用于实验的内核.
我正在使用3.13.0-67 x86_64和PG 9.3.在9.3中,他们从使用Sys V共享内存(shmget)切换到anonymous mmap(...R+W,MAP_SHARED|MAP_ANONYMOUS|MAP_HASSEMAPHORE...)+fork()(在9.4中,这可以通过dynamic_shared_memory_type进行配置).但我找不到任何解释为什么这些mmap()应该出现在“缓存”中,为什么,只有https://access.redhat.com/solutions/406773说“缓存：页面缓存中的内存(磁盘缓存和共享内存)”
鉴于有many kinds of shared memory我既开明也有困惑……
