NFS Performance Testing/Tuning Report and Recommendations¶

NFS Performance Testing/Tuning Report¶

First phase NFS performance investigation¶

Based on my first stage of NFS performance investigation, I have finished 2 parts of investigation:

1. Compare with bob’s system and my system.
   • Conclusion: Storage hardware is one reason of bad NFS I/O performance
2. Perform some benchmark test use bonnie++/iostat, DD and rsync
   • Files I/O is the bottleneck of NFS
   • Small size file I/O performance is too bad.

First phase report: http://www.fossology.org/attachments/download/1533/nfs_performance_investigation_phase1.ppt

Second phase NFS performance investigation¶

Based on Bob’s comments and Mark’s great papers for NFS performance tuning, I do more second stage of NFS performance investigation:

Change the mount method we use for now, use single NFS server instead of multiple cross mount¶

Test Result

• Test file: 285MB
• Cluster NFS: unpack time: 620s | db time: 84s | repo time: 232s | cpu time: 100s
• Single NFS: unpack time: 540s | db time: 66s | repo time: 173s | cpu time: 101s

Conclusion: Single NFS server have better performance than cross mount, about 13% performance improve

Based on http://nfs.sourceforge.net/ and 1 paper Mark send me, some points and tests about tuning NFS¶

Before my tuning, I test some result as my default configuration of single NFS: * Test file: 285MB * DD test: 105MB/s * Unpack from UI: 540s

Tuning the NFS client

• Tuning the network, NFS can’t be any faster than the network
  • Use netperf get my network environment highest throughout is 112MB/s
  • Don’t do more tuning
• Use NFSv3 (default)
  • Use TCP (default)
• Use the maximum transfer size (default)
  • Do DD test to change small and large transfer size
  • DD test: 91MB/s
• Don’t use soft
  • In my base test system, set this as soft
  • Change to hard, DD test: 105MB/s, UI test: 541s; no more changes
• Use intr
  • In my base test system, no add
  • Add intr, DD test: 105MB/s, no more changes
• Use the maximum MTU
  • Don’t do it, need change netcard and switch configuration
• No more Mount Options
  • In my base test system, no more mount options
• Parallelism (default)

Tuning the NFS server

• Tune the Storage Hardware
• Tune the Block Layer
• Tune the Filesystem
• Tune the VM
• Tune the PCI cards
• Tune the Network
• Use async
  • In my base test system: use async
  • Change to sync: DD test: 74MB/s
• Use no_subtree_check
  • In my base test system: use no_subtree_check
  • Change to subtree_check: 102MB/s
• Use More Server Threads
  • In my base test sytem: default is 8
  • Change to small and to large (4, 16, 32, 128), DD test no more changes

Third phase NFS performance investigation¶

Test Environment¶

Hardware:

• bl480c-1.test (CPU: 2xQuad-Core Intel Xeon, 3.00GHz MEM: 12x2GB) web/db/scheduler
• bl460c-9.test (CPU: 2xMulti-Core Intel Xeon, 2.33GHZ MEM: 4x1GB) agent
• bl460c-13.test (CPU: 1xMulti-Core Intel Xeon, 3.00GHZ MEM: 4x1GB) agent
• bl460c-15.test (CPU: 1xMulti-Core Intel Xeon, 3.00GHZ MEM: 4x1GB) agent
• bl460c-10.test (CPU: 2xMulti-Core Intel Xeon, 2.33GHZ MEM: 4x1GB) nfs server

Test Data:

• SLES-11-SP1-DVD-x86_64-GMC3-DVD1.iso 2.9G
• SLES-11-SP1-DVD-x86_64-GMC3-DVD2.iso 4.3G
• SLES-11-SP1-DVD-x86_64-GMC3-DVD3.iso 2.7G

Test Case:

• Current corss mount system: Using UI upload 3 isos as close as together, only queue nomos job.
• 1 dedicated NFS server use single nfs mount: Using UI upload 3 isos as close as together, only queue nomos job.
• 1 dedicated NFS server use single nfs mount, increase nfs server memory from 4G to 8G: Using UI upload 3 isos as close as together, didn't finished all nomos job.

Test Results¶

NFS configuration Recommendations¶

So based on above investigation and testing, I have the following recommendations for multiple FOSSology setup:

• If you have a amazing server with large memory, i recommend use single NFS server with large memory to instead of cross mount NFS, also suggest the NFS server don't run other things db/web/scheduler/agent.
  Otherwise cross mount configuration NFS will be our prefer.
• About NFS client mount options
  • Modern Linux clients mostly have good defaults, most mount options you specify will make things worse
  • Suggest options: rw,intr
  • Tuning your network
• About NFS server
  • Suggest export options: async,no_subtree_check
  • Use More Server Threads
  • Your system Storage Hardware/Block Layer/Filesystem/VM/PCI cards/Network more effect on the NFS performance, if you need more improvement, suggest focus on these parts