这个消息对某人来说很熟悉吗?
zfs-clu-01a rpcmod: [ID 851375 kern.warning] WARNING: svc_cots_kdup no slots free zfs-clu-01a last message repeated 75220 times
我们在ZFS上使用NFS时遇到了很多麻烦.
实际上我们不能通过NFS复制大文件(> 500mo),我们有性能
的问题.
NFS client using NFSv3 over TCP **--NFS Server Configuration :** Concurrent NFSD Servers: 4096 NFSD queue length: 512 Concurrent LOCKD Servers: 4096 LOCKD queue length 512 **-- System configuration** **1/ Disable ZIL** Settings -> Preferences -> System Sys_zfs_nocacheflush Yes (default: No) Warning: it is unsafe without UPS **2/Disable Nagel algoritm** Settings -> Preferences -> Network Net_tcp_naglim_def 1 (default: 4095) **4/ disable sync** Data Management -> Shares -> Folder xxx Sync disable (default: Standard)
如果需要,我可以提供更多信息.
——– UPDATE 01 ————-
我有一个新的有趣的事情:
这个问题只有或者最常见于“centos”系统(6.5版本)
与Debian(上一版本)相同的是在同一时间段内在此文件副本上运行良好
以下是更多信息:
OS: Nexenta Ver. 31-5-0 Load Average 1.25,1.26,1.21 NFSv3 Server (configuration in my last email)
1)NFS客户端
Clients Number: 15 with very high access demand and then we have more than 20 others low access Centos 6 and Debian 7 NFS Client v3 Mout options : nfs (rw,noatime,nodiratime,vers=3,rsize=16384,wsize=16384,proto=tcp,mountproto=tcp,nolock) Nics: 1x 1g Dedicated NFS vlan
2)zpool列表
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT syspool 464G 181G 283G 39% 1.00x ONLINE - tank1 6.52T 2.38T 4.13T 36% 1.00x ONLINE -
3)zpool状态
pool: syspool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM syspool ONLINE 0 0 0 c0t0d0s0 ONLINE 0 0 0 errors: No known data errors pool: tank1 state: ONLINE scan: resilvered 190G in 81h49m with 0 errors on Sun Jan 26 15:35:10 2014 config: NAME STATE READ WRITE CKSUM tank1 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c6t5000C5005F0C34A7d0 ONLINE 0 0 0 c6t5000C5005F1261CBd0 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 c6t5000C5005F531E2Fd0 ONLINE 0 0 0 c6t5000C5005FD1677Fd0 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 c6t5000C5005FD18BF7d0 ONLINE 0 0 0 c6t5000C5005FD1907Bd0 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 c6t50000394B820EA30d0 ONLINE 0 0 0 c6t5000C5005FD1D133d0 ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 c6t5000C5005FD73F97d0 ONLINE 0 0 0 c6t5000C5005FD7518Bd0 ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 c6t5000C5005FD7C2A3d0 ONLINE 0 0 0 c6t5000C5005FDC123Bd0 ONLINE 0 0 0 mirror-6 ONLINE 0 0 0 c6t5000C5005FDE1D13d0 ONLINE 0 0 0 c6t5000C5005FDEAFDFd0 ONLINE 0 0 0 mirror-7 ONLINE 0 0 0 c6t5000C5005FDEB127d0 ONLINE 0 0 0 c6t5000C5005FDEB513d0 ONLINE 0 0 0 mirror-8 ONLINE 0 0 0 c6t5000C5005FDF88CBd0 ONLINE 0 0 0 c6t5000C5005FDF9C4Fd0 ONLINE 0 0 0 mirror-9 ONLINE 0 0 0 c6t5000C50067802A17d0 ONLINE 0 0 0 c6t5000C500678037A7d0 ONLINE 0 0 0 mirror-10 ONLINE 0 0 0 c6t5000C5006780D6F7d0 ONLINE 0 0 0 c6t5000C5006780DEB3d0 ONLINE 0 0 0 mirror-11 ONLINE 0 0 0 c6t5000C5006780E90Fd0 ONLINE 0 0 0 c6t5000C5006780E99Bd0 ONLINE 0 0 0 cache c0t1d0 ONLINE 0 0 0 c0t2d0 ONLINE 0 0 0 spares c6t5000C5006BF8C91Fd0 AVAIL errors: No known data errors
4)hddisco
=c6t5000C5006BF8C91Fd0 driver sd driver_instance 37 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS08 serial S0M19QGA guid 5000c5006bf8c91f device_id id1,sd@n5000c5006bf8c91f is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5006bf8c91d P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5006bf8c91e P end =c6t5000C5005F531E2Fd0 driver sd driver_instance 26 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M04DCN guid 5000c5005f531e2f device_id id1,sd@n5000c5005f531e2f is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005f531e2d P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005f531e2e P end =c6t5000C5005FDEAFDFd0 driver sd driver_instance 27 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M061KF guid 5000c5005fdeafdf device_id id1,sd@n5000c5005fdeafdf is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fdeafdd P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fdeafde P end =c6t5000C5006780E90Fd0 driver sd driver_instance 29 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M06GV9 guid 5000c5006780e90f device_id id1,sd@n5000c5006780e90f is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5006780e90d P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5006780e90e P end =c6t5000C5005FD1677Fd0 driver sd driver_instance 28 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M05WCR guid 5000c5005fd1677f device_id id1,sd@n5000c5005fd1677f is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd1677d P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd1677e P end =c6t5000C5005FDF9C4Fd0 driver sd driver_instance 30 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M03SZR guid 5000c5005fdf9c4f device_id id1,sd@n5000c5005fdf9c4f is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fdf9c4d P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fdf9c4e P end =c6t5000C5005FD7518Bd0 driver sd driver_instance 25 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M06JCM guid 5000c5005fd7518b device_id id1,sd@n5000c5005fd7518b is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd75189 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd7518a P end =c6t5000C5005F1261CBd0 driver sd driver_instance 24 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M03LZY guid 5000c5005f1261cb device_id id1,sd@n5000c5005f1261cb is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005f1261c9 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005f1261ca P end =c6t5000C5006780E99Bd0 driver sd driver_instance 20 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M06GV8 guid 5000c5006780e99b device_id id1,sd@n5000c5006780e99b is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5006780e999 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5006780e99a P end =c6t5000C5005FD1907Bd0 driver sd driver_instance 23 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M0657D guid 5000c5005fd1907b device_id id1,sd@n5000c5005fd1907b is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd19079 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd1907a P end =c6t5000C5005FDF88CBd0 driver sd driver_instance 22 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M04CQ7 guid 5000c5005fdf88cb device_id id1,sd@n5000c5005fdf88cb is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fdf88c9 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fdf88ca P end =c6t5000C5005FDC123Bd0 driver sd driver_instance 21 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M05WMD guid 5000c5005fdc123b device_id id1,sd@n5000c5005fdc123b is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fdc1239 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fdc123a P end =c6t5000C5006780D6F7d0 driver sd driver_instance 19 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M05TVK guid 5000c5006780d6f7 device_id id1,sd@n5000c5006780d6f7 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5006780d6f5 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5006780d6f6 P end =c6t5000C5005FDEB127d0 driver sd driver_instance 13 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M061JX guid 5000c5005fdeb127 device_id id1,sd@n5000c5005fdeb127 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fdeb125 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fdeb126 P end =c6t5000C5005FD73F97d0 driver sd driver_instance 16 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M06J86 guid 5000c5005fd73f97 device_id id1,sd@n5000c5005fd73f97 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd73f95 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd73f96 P end =c6t5000C5005F0C34A7d0 driver sd driver_instance 14 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M0405D guid 5000c5005f0c34a7 device_id id1,sd@n5000c5005f0c34a7 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005f0c34a5 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005f0c34a6 P end =c6t5000C500678037A7d0 driver sd driver_instance 15 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M03FNP guid 5000c500678037a7 device_id id1,sd@n5000c500678037a7 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c500678037a5 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c500678037a6 P end =c6t5000C5005FD18BF7d0 driver sd driver_instance 17 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M0659G guid 5000c5005fd18bf7 device_id id1,sd@n5000c5005fd18bf7 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd18bf5 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd18bf6 P end =c6t5000C50067802A17d0 driver sd driver_instance 18 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M03WJV guid 5000c50067802a17 device_id id1,sd@n5000c50067802a17 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c50067802a15 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c50067802a16 P end =c6t50000394B820EA30d0 driver sd driver_instance 32 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor TOSHIBA product AL13SEB600 revision DE06 serial 43L0A02HFRD3 guid 50000394b820ea30 device_id id1,sd@n50000394b820ea30 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w50000394b820ea32 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w50000394b820ea33 P end =c6t5000C5005FDE1D13d0 driver sd driver_instance 11 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M060LE guid 5000c5005fde1d13 device_id id1,sd@n5000c5005fde1d13 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fde1d11 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fde1d12 P end =c6t5000C5005FDEB513d0 driver sd driver_instance 6 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M061HL guid 5000c5005fdeb513 device_id id1,sd@n5000c5005fdeb513 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fdeb511 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fdeb512 P end =c6t5000C5005FD7C2A3d0 driver sd driver_instance 9 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M06HG0 guid 5000c5005fd7c2a3 device_id id1,sd@n5000c5005fd7c2a3 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd7c2a1 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd7c2a2 P end =c6t5000C5005FD1D133d0 driver sd driver_instance 7 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M064EN guid 5000c5005fd1d133 device_id id1,sd@n5000c5005fd1d133 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd1d131 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd1d132 P end =c6t5000C5006780DEB3d0 driver sd driver_instance 10 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M060TL guid 5000c5006780deb3 device_id id1,sd@n5000c5006780deb3 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5006780deb1 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5006780deb2 P end =c0t2d0 driver sd driver_instance 34 device_type disk removable no media_present yes size 511503761408 size_str 476.4GB caching_supported no vendor DELL product PERC H710 revision 3.13 serial 0022d14c0cc2ee9e1900ac67e7a0a38c guid 6b8ca3a0e767ac00199eeec20c4cd122 device_id id1,sd@n6b8ca3a0e767ac00199eeec20c4cd122 is_ssd no mpxio_enabled no lun_no 0 parent mr_sas parent_instance 0 =c0t1d0 driver sd driver_instance 33 device_type disk removable no media_present yes size 511503761408 size_str 476.4GB caching_supported no vendor DELL product PERC H710 revision 3.13 serial 0027a4060badee9e1900ac67e7a0a38c guid 6b8ca3a0e767ac00199eeead0b06a427 device_id id1,sd@n6b8ca3a0e767ac00199eeead0b06a427 is_ssd no mpxio_enabled no lun_no 0 parent mr_sas parent_instance 0 =c0t0d0 driver sd driver_instance 5 device_type disk removable no media_present yes size 499558383616 size_str 465.3GB caching_supported no vendor DELL product PERC H710 revision 3.13 serial 003f0f9308bb2c521900ac67e7a0a38c guid 6b8ca3a0e767ac0019522cbb08930f3f device_id id1,sd@n6b8ca3a0e767ac0019522cbb08930f3f is_ssd no mpxio_enabled no lun_no 0 parent mr_sas parent_instance 0
@R_404_323@
Hmms,看起来像系统(Nexenta)正在获得太多的NFS连接,或者你遇到了一个bug.您可以尝试提升maxdupreqs,这可能会掩盖问题,但让我们先尝试一下:
echo 'rpcmod`maxdupreqs/W0t4096' | mdb -kw echo 'rpcmod`cotsmaxdupreqs/W0t4096' | mdb -kw
要检查它的当前值是什么用途:
echo 'rpcmod`maxdupreqs::print' | mdb -k echo 'rpcmod`cotsmaxdupreqs::print' | mdb -k
但也许它会有所帮助.