我有一个运行名为ABC的应用程序的多用户CentOS 6.1数据库系统.服务器是64位,8GB RAM和6个vcpu(在VMWare ESXi 4上).我们经常转储到dmesg和系统日志,详细说明内核页面分配失败.
在这种情况下,这些消息意味着什么？我们如何解决这个问题？
Feb 5 08:10:52 Fruity kernel: ABC: page allocation failure. order:1,mode:0x20 Feb 5 08:10:52 Fruity kernel: Pid: 23588,comm: ABC Not tainted 2.6.32-131.17.1.el6.x86_64 #1 Feb 5 08:10:52 Fruity kernel: Call Trace: Feb 5 08:10:52 Fruity kernel: <IRQ> [<ffffffff8112016e>] ? __alloc_pages_nodemask+0x71e/0x8b0 Feb 5 08:10:52 Fruity kernel: [<ffffffff81159a52>] ? kmem_getpages+0x62/0x170 Feb 5 08:10:52 Fruity kernel: [<ffffffff8115a66a>] ? fallback_alloc+0x1ba/0x270 Feb 5 08:10:52 Fruity kernel: [<ffffffff8115a0bf>] ? cache_grow+0x2cf/0x320 Feb 5 08:10:52 Fruity kernel: [<ffffffff8115a3e9>] ? ____cache_alloc_node+0x99/0x160 Feb 5 08:10:52 Fruity kernel: [<ffffffff8115b1ab>] ? kmem_cache_alloc+0x11b/0x190 Feb 5 08:10:52 Fruity kernel: [<ffffffff81411ba8>] ? sk_prot_alloc+0x48/0x1a0 Feb 5 08:10:52 Fruity kernel: [<ffffffff81411e12>] ? sk_clone+0x22/0x2c0 Feb 5 08:10:52 Fruity kernel: [<ffffffff8145caf6>] ? inet_csk_clone+0x16/0xd0 Feb 5 08:10:52 Fruity kernel: [<ffffffff81475be3>] ? tcp_create_openreq_child+0x23/0x450 Feb 5 08:10:52 Fruity kernel: [<ffffffff814735cd>] ? tcp_v4_syn_recv_sock+0x4d/0x2a0 Feb 5 08:10:52 Fruity kernel: [<ffffffff814759a1>] ? tcp_check_req+0x201/0x420 Feb 5 08:10:52 Fruity kernel: [<ffffffff8146b4b6>] ? tcp_rcv_state_process+0x116/0xa30 Feb 5 08:10:52 Fruity kernel: [<ffffffff8105055a>] ? enqueue_entity+0x13a/0x340 Feb 5 08:10:52 Fruity kernel: [<ffffffff81472feb>] ? tcp_v4_do_rcv+0x35b/0x430 Feb 5 08:10:52 Fruity kernel: [<ffffffff81474760>] ? tcp_v4_rcv+0x4e0/0x860 Feb 5 08:10:52 Fruity kernel: [<ffffffff8105dc32>] ? default_wake_function+0x12/0x20 Feb 5 08:10:52 Fruity kernel: [<ffffffff8145247d>] ? ip_local_deliver_finish+0xdd/0x2d0 Feb 5 08:10:52 Fruity kernel: [<ffffffff81452708>] ? ip_local_deliver+0x98/0xa0 Feb 5 08:10:52 Fruity kernel: [<ffffffff81451bcd>] ? ip_rcv_finish+0x12d/0x440 Feb 5 08:10:52 Fruity kernel: [<ffffffff8104fc08>] ? update_curr+0xf8/0x1e0 Feb 5 08:10:52 Fruity kernel: [<ffffffff81452155>] ? ip_rcv+0x275/0x350 Feb 5 08:10:52 Fruity kernel: [<ffffffff8141dccb>] ? __netif_receive_skb+0x39b/0x6b0 Feb 5 08:10:52 Fruity kernel: [<ffffffff810db997>] ? cpu_quiet_msk+0x77/0x130 Feb 5 08:10:52 Fruity kernel: [<ffffffff8141e07a>] ? process_backlog+0x9a/0x100 Feb 5 08:10:52 Fruity kernel: [<ffffffff81422533>] ? net_rx_action+0x103/0x2f0 Feb 5 08:10:52 Fruity kernel: [<ffffffff8106f6e1>] ? __do_softirq+0xc1/0x1d0 Feb 5 08:10:52 Fruity kernel: [<ffffffff8100c2cc>] ? call_softirq+0x1c/0x30 Feb 5 08:10:52 Fruity kernel: [<ffffffff8100c2cc>] ? call_softirq+0x1c/0x30 Feb 5 08:10:52 Fruity kernel: <EOI> [<ffffffff8100df05>] ? do_softirq+0x65/0xa0 Feb 5 08:10:52 Fruity kernel: [<ffffffff81070028>] ? local_bh_enable_ip+0x98/0xa0 Feb 5 08:10:52 Fruity kernel: [<ffffffff814dd92b>] ? _spin_unlock_bh+0x1b/0x20 Feb 5 08:10:52 Fruity kernel: [<ffffffff8140f46e>] ? release_sock+0xce/0xe0 Feb 5 08:10:52 Fruity kernel: [<ffffffff81483953>] ? inet_stream_connect+0x183/0x2c0 Feb 5 08:10:52 Fruity kernel: [<ffffffff8108e180>] ? autoremove_wake_function+0x0/0x40 Feb 5 08:10:52 Fruity kernel: [<ffffffff8140d007>] ? sys_connect+0xd7/0xf0 Feb 5 08:10:52 Fruity kernel: [<ffffffff8145f652>] ? compat_tcp_setsockopt+0x22/0x30 Feb 5 08:10:52 Fruity kernel: [<ffffffff8140eb9c>] ? compat_sock_common_setsockopt+0x1c/0x30 Feb 5 08:10:52 Fruity kernel: [<ffffffff81437d05>] ? compat_sys_setsockopt+0x85/0x220 Feb 5 08:10:52 Fruity kernel: [<ffffffff81184828>] ? sys_fcntl+0x118/0x530 Feb 5 08:10:52 Fruity kernel: [<ffffffff8143805e>] ? compat_sys_socketcall+0x1be/0x200 Feb 5 08:10:52 Fruity kernel: [<ffffffff810478b0>] ? sysenter_dispatch+0x7/0x2e Feb 5 08:10:52 Fruity kernel: Mem-Info: Feb 5 08:10:52 Fruity kernel: Node 0 DMA per-cpu: Feb 5 08:10:52 Fruity kernel: cpu 0: hi: 0,btch: 1 usd: 0 Feb 5 08:10:52 Fruity kernel: cpu 1: hi: 0,btch: 1 usd: 0 Feb 5 08:10:52 Fruity kernel: cpu 2: hi: 0,btch: 1 usd: 0 Feb 5 08:10:52 Fruity kernel: cpu 3: hi: 0,btch: 1 usd: 0 Feb 5 08:10:52 Fruity kernel: cpu 4: hi: 0,btch: 1 usd: 0 Feb 5 08:10:52 Fruity kernel: cpu 5: hi: 0,btch: 1 usd: 0 Feb 5 08:10:52 Fruity kernel: Node 0 DMA32 per-cpu: Feb 5 08:10:52 Fruity kernel: cpu 0: hi: 186,btch: 31 usd: 167 Feb 5 08:10:52 Fruity kernel: cpu 1: hi: 186,btch: 31 usd: 44 Feb 5 08:10:52 Fruity kernel: cpu 2: hi: 186,btch: 31 usd: 59 Feb 5 08:10:52 Fruity kernel: cpu 3: hi: 186,btch: 31 usd: 46 Feb 5 08:10:52 Fruity kernel: cpu 4: hi: 186,btch: 31 usd: 157 Feb 5 08:10:52 Fruity kernel: cpu 5: hi: 186,btch: 31 usd: 45 Feb 5 08:10:52 Fruity kernel: Node 0 Normal per-cpu: Feb 5 08:10:52 Fruity kernel: cpu 0: hi: 186,btch: 31 usd: 182 Feb 5 08:10:52 Fruity kernel: cpu 1: hi: 186,btch: 31 usd: 15 Feb 5 08:10:52 Fruity kernel: cpu 3: hi: 186,btch: 31 usd: 88 Feb 5 08:10:52 Fruity kernel: cpu 4: hi: 186,btch: 31 usd: 181 Feb 5 08:10:52 Fruity kernel: cpu 5: hi: 186,btch: 31 usd: 33 Feb 5 08:10:52 Fruity kernel: active_anon:79381 inactive_anon:21406 isolated_anon:0 Feb 5 08:10:52 Fruity kernel: active_file:395766 inactive_file:1432708 isolated_file:0 Feb 5 08:10:52 Fruity kernel: unevictable:0 dirty:297 writeback:0 unstable:0 Feb 5 08:10:52 Fruity kernel: free:31126 slab_reclaimable:25909 slab_unreclaimable:44714 Feb 5 08:10:52 Fruity kernel: mapped:3908 shmem:103 pagetables:4196 bounce:0 Feb 5 08:10:52 Fruity kernel: Node 0 DMA free:15680kB min:124kB low:152kB high:184kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15284kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Feb 5 08:10:52 Fruity kernel: lowmem_reserve[]: 0 3000 8050 8050 Feb 5 08:10:52 Fruity kernel: Node 0 DMA32 free:56332kB min:25140kB low:31424kB high:37708kB active_anon:36800kB inactive_anon:33152kB active_file:631228kB inactive_file:2126792kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:360kB writeback:0kB mapped:4500kB shmem:4kB slab_reclaimable:42108kB slab_unreclaimable:4760kB kernel_stack:256kB pagetables:1228kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Feb 5 08:10:52 Fruity kernel: lowmem_reserve[]: 0 0 5050 5050 Feb 5 08:10:52 Fruity kernel: Node 0 Normal free:52492kB min:42316kB low:52892kB high:63472kB active_anon:280724kB inactive_anon:52472kB active_file:951836kB inactive_file:3603784kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5171200kB mlocked:0kB dirty:828kB writeback:0kB mapped:11132kB shmem:408kB slab_reclaimable:61528kB slab_unreclaimable:174096kB kernel_stack:3112kB pagetables:15556kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Feb 5 08:10:52 Fruity kernel: lowmem_reserve[]: 0 0 0 0 Feb 5 08:10:52 Fruity kernel: Node 0 DMA: 4*4kB 2*8kB 2*16kB 0*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15680kB Feb 5 08:10:52 Fruity kernel: Node 0 DMA32: 12892*4kB 79*8kB 30*16kB 10*32kB 4*64kB 19*128kB 3*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 56456kB Feb 5 08:10:52 Fruity kernel: Node 0 Normal: 12558*4kB 35*8kB 1*16kB 1*32kB 4*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 52864kB Feb 5 08:10:52 Fruity kernel: 1834295 total pagecache pages Feb 5 08:10:52 Fruity kernel: 5823 pages in swap cache Feb 5 08:10:52 Fruity kernel: Swap cache stats: add 112073,delete 106250,find 283106960/283124994 Feb 5 08:10:52 Fruity kernel: Free swap = 8352448kB Feb 5 08:10:52 Fruity kernel: Total swap = 8388600kB Feb 5 08:10:52 Fruity kernel: 2097136 pages RAM Feb 5 08:10:52 Fruity kernel: 48740 pages reserved Feb 5 08:10:52 Fruity kernel: 73879 pages shared Feb 5 08:10:52 Fruity kernel: 1940523 pages non-shared
编辑：即使下面提出了一些更改,这仍然会发生.当前跟踪看起来像：
Feb 29 04:45:33 Fruity kernel: swapper: page allocation failure. order:1,mode:0x20 Feb 29 04:45:33 Fruity kernel: Pid: 0,comm: swapper Not tainted 2.6.32-131.17.1.el6.x86_64 #1 Feb 29 04:45:33 Fruity kernel: Call Trace: Feb 29 04:45:33 Fruity kernel: <IRQ> [<ffffffff8112016e>] ? __alloc_pages_nodemask+0x71e/0x8b0 Feb 29 04:45:33 Fruity kernel: [<ffffffff81159a52>] ? kmem_getpages+0x62/0x170 Feb 29 04:45:33 Fruity kernel: [<ffffffff8115a66a>] ? fallback_alloc+0x1ba/0x270 Feb 29 04:45:33 Fruity kernel: [<ffffffff8115a0bf>] ? cache_grow+0x2cf/0x320 Feb 29 04:45:33 Fruity kernel: [<ffffffff8115a3e9>] ? ____cache_alloc_node+0x99/0x160 Feb 29 04:45:33 Fruity kernel: [<ffffffff8115b1ab>] ? kmem_cache_alloc+0x11b/0x190 Feb 29 04:45:33 Fruity kernel: [<ffffffff81411ba8>] ? sk_prot_alloc+0x48/0x1a0 Feb 29 04:45:33 Fruity kernel: [<ffffffff81411e12>] ? sk_clone+0x22/0x2c0 Feb 29 04:45:33 Fruity kernel: [<ffffffff8145caf6>] ? inet_csk_clone+0x16/0xd0 Feb 29 04:45:33 Fruity kernel: [<ffffffff81475be3>] ? tcp_create_openreq_child+0x23/0x450 Feb 29 04:45:33 Fruity kernel: [<ffffffff814735cd>] ? tcp_v4_syn_recv_sock+0x4d/0x2a0 Feb 29 04:45:33 Fruity kernel: [<ffffffff814759a1>] ? tcp_check_req+0x201/0x420 Feb 29 04:45:33 Fruity kernel: [<ffffffff8146b4b6>] ? tcp_rcv_state_process+0x116/0xa30 Feb 29 04:45:33 Fruity kernel: [<ffffffff81472feb>] ? tcp_v4_do_rcv+0x35b/0x430 Feb 29 04:45:33 Fruity kernel: [<ffffffff81413a1b>] ? consume_skb+0x3b/0x80 Feb 29 04:45:33 Fruity kernel: [<ffffffff81474760>] ? tcp_v4_rcv+0x4e0/0x860 Feb 29 04:45:33 Fruity kernel: [<ffffffff8145247d>] ? ip_local_deliver_finish+0xdd/0x2d0 Feb 29 04:45:33 Fruity kernel: [<ffffffff81452708>] ? ip_local_deliver+0x98/0xa0 Feb 29 04:45:33 Fruity kernel: [<ffffffff81451bcd>] ? ip_rcv_finish+0x12d/0x440 Feb 29 04:45:33 Fruity kernel: [<ffffffff81452155>] ? ip_rcv+0x275/0x350 Feb 29 04:45:33 Fruity kernel: [<ffffffff8141dccb>] ? __netif_receive_skb+0x39b/0x6b0 Feb 29 04:45:33 Fruity kernel: [<ffffffff810a41a4>] ? __smp_call_function_single+0x64/0xe0 Feb 29 04:45:33 Fruity kernel: [<ffffffff8141ffd8>] ? netif_receive_skb+0x58/0x60 Feb 29 04:45:33 Fruity kernel: [<ffffffffa0131853>] ? vmxnet3_poll+0x403/0x9f0 [vmxnet3] Feb 29 04:45:33 Fruity kernel: [<ffffffffa0036c40>] ? pvscsi_process_completion_ring+0xe0/0x350 [vmw_pvscsi] Feb 29 04:45:33 Fruity kernel: [<ffffffff81422533>] ? net_rx_action+0x103/0x2f0 Feb 29 04:45:33 Fruity kernel: [<ffffffff8106f6e1>] ? __do_softirq+0xc1/0x1d0 Feb 29 04:45:33 Fruity kernel: [<ffffffff810d6930>] ? handle_IRQ_event+0x60/0x170 Feb 29 04:45:33 Fruity kernel: [<ffffffff8100c2cc>] ? call_softirq+0x1c/0x30 Feb 29 04:45:33 Fruity kernel: [<ffffffff8100df05>] ? do_softirq+0x65/0xa0 Feb 29 04:45:33 Fruity kernel: [<ffffffff8106f4c5>] ? irq_exit+0x85/0x90 Feb 29 04:45:33 Fruity kernel: [<ffffffff814e3195>] ? do_IRQ+0x75/0xf0 Feb 29 04:45:33 Fruity kernel: [<ffffffff8100bad3>] ? ret_from_intr+0x0/0x11 Feb 29 04:45:33 Fruity kernel: <EOI> [<ffffffff8103628b>] ? native_safe_halt+0xb/0x10 Feb 29 04:45:33 Fruity kernel: [<ffffffff810142ed>] ? default_idle+0x4d/0xb0 Feb 29 04:45:33 Fruity kernel: [<ffffffff81009e86>] ? cpu_idle+0xb6/0x110 Feb 29 04:45:33 Fruity kernel: [<ffffffff814c33da>] ? rest_init+0x7a/0x80 Feb 29 04:45:33 Fruity kernel: [<ffffffff81c1df28>] ? start_kernel+0x41d/0x429 Feb 29 04:45:33 Fruity kernel: [<ffffffff81c1d33a>] ? x86_64_start_reservations+0x125/0x129 Feb 29 04:45:33 Fruity kernel: [<ffffffff81c1d438>] ? x86_64_start_kernel+0xfa/0x109 Feb 29 04:45:33 Fruity kernel: Mem-Info: Feb 29 04:45:33 Fruity kernel: Node 0 DMA per-cpu: Feb 29 04:45:33 Fruity kernel: cpu 0: hi: 0,btch: 1 usd: 0 Feb 29 04:45:33 Fruity kernel: cpu 1: hi: 0,btch: 1 usd: 0 Feb 29 04:45:33 Fruity kernel: cpu 2: hi: 0,btch: 1 usd: 0 Feb 29 04:45:33 Fruity kernel: cpu 3: hi: 0,btch: 1 usd: 0 Feb 29 04:45:33 Fruity kernel: cpu 4: hi: 0,btch: 1 usd: 0 Feb 29 04:45:33 Fruity kernel: cpu 5: hi: 0,btch: 1 usd: 0 Feb 29 04:45:33 Fruity kernel: Node 0 DMA32 per-cpu: Feb 29 04:45:33 Fruity kernel: cpu 0: hi: 186,btch: 31 usd: 46 Feb 29 04:45:33 Fruity kernel: cpu 1: hi: 186,btch: 31 usd: 1 Feb 29 04:45:33 Fruity kernel: cpu 2: hi: 186,btch: 31 usd: 23 Feb 29 04:45:33 Fruity kernel: cpu 3: hi: 186,btch: 31 usd: 10 Feb 29 04:45:33 Fruity kernel: cpu 4: hi: 186,btch: 31 usd: 38 Feb 29 04:45:33 Fruity kernel: cpu 5: hi: 186,btch: 31 usd: 2 Feb 29 04:45:33 Fruity kernel: Node 0 Normal per-cpu: Feb 29 04:45:33 Fruity kernel: cpu 0: hi: 186,btch: 31 usd: 65 Feb 29 04:45:33 Fruity kernel: cpu 1: hi: 186,btch: 31 usd: 0 Feb 29 04:45:33 Fruity kernel: cpu 2: hi: 186,btch: 31 usd: 14 Feb 29 04:45:33 Fruity kernel: cpu 3: hi: 186,btch: 31 usd: 2 Feb 29 04:45:33 Fruity kernel: cpu 4: hi: 186,btch: 31 usd: 29 Feb 29 04:45:33 Fruity kernel: cpu 5: hi: 186,btch: 31 usd: 50 Feb 29 04:45:33 Fruity kernel: active_anon:118532 inactive_anon:29343 isolated_anon:0 Feb 29 04:45:33 Fruity kernel: active_file:870242 inactive_file:899801 isolated_file:0 Feb 29 04:45:33 Fruity kernel: unevictable:0 dirty:5135 writeback:0 unstable:0 Feb 29 04:45:33 Fruity kernel: free:33179 slab_reclaimable:34315 slab_unreclaimable:45350 Feb 29 04:45:33 Fruity kernel: mapped:3464 shmem:133 pagetables:4997 bounce:0 Feb 29 04:45:33 Fruity kernel: Node 0 DMA free:15680kB min:124kB low:152kB high:184kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15284kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Feb 29 04:45:33 Fruity kernel: lowmem_reserve[]: 0 3000 8050 8050 Feb 29 04:45:33 Fruity kernel: Node 0 DMA32 free:64200kB min:25140kB low:31424kB high:37708kB active_anon:59816kB inactive_anon:47980kB active_file:1319196kB inactive_file:1374832kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:7680kB writeback:0kB mapped:3004kB shmem:40kB slab_reclaimable:62060kB slab_unreclaimable:5368kB kernel_stack:160kB pagetables:900kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Feb 29 04:45:33 Fruity kernel: lowmem_reserve[]: 0 0 5050 5050 Feb 29 04:45:33 Fruity kernel: Node 0 Normal free:52836kB min:42316kB low:52892kB high:63472kB active_anon:414312kB inactive_anon:69392kB active_file:2161772kB inactive_file:2224372kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5171200kB mlocked:0kB dirty:12860kB writeback:0kB mapped:10852kB shmem:492kB slab_reclaimable:75200kB slab_unreclaimable:176032kB kernel_stack:3384kB pagetables:19088kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Feb 29 04:45:33 Fruity kernel: lowmem_reserve[]: 0 0 0 0 Feb 29 04:45:33 Fruity kernel: Node 0 DMA: 4*4kB 2*8kB 2*16kB 0*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15680kB Feb 29 04:45:33 Fruity kernel: Node 0 DMA32: 15988*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 63952kB Feb 29 04:45:33 Fruity kernel: Node 0 Normal: 13209*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 52836kB Feb 29 04:45:33 Fruity kernel: 1776603 total pagecache pages Feb 29 04:45:33 Fruity kernel: 6398 pages in swap cache Feb 29 04:45:33 Fruity kernel: Swap cache stats: add 163231,delete 156833,find 403959091/403986630 Feb 29 04:45:33 Fruity kernel: Free swap = 8339552kB Feb 29 04:45:33 Fruity kernel: Total swap = 8388600kB Feb 29 04:45:33 Fruity kernel: 2097136 pages RAM Feb 29 04:45:33 Fruity kernel: 48740 pages reserved Feb 29 04:45:33 Fruity kernel: 198220 pages shared Feb 29 04:45:33 Fruity kernel: 1833933 pages non-shared
在这里大声思考,但您是否考虑过使用sysctl增加vm.min_free_kbytes值？
原文链接：https://www.f2er.com/centos/373667.html就像是：
sysctl vm.min_free_kbytes=16384
(ps – 不是100％确定它在centos上的假设,更可能在/ proc / sys / vm / min_free_kbytes下找到)
