SOLVED Slow NFS after CPU + Mainboard change

[Herr_Merlin]

Hi all,

we are running a freenas as NFS datastore for our LAB vmware cluster.
Past hardware:
Some Supermicro X9 with 2x Xeon E5 Quadcore V0 3,4Ghz HT disabled

Current Hardware:
Some Supermicro X9 with 2x Xeon E5 Tencore V2 2,8Ghz HT disabled

Hardware which did not change:
256GB DDR3 1600Mhz
3x HP H240 SAS12G HBA (1 HBA holds all disk of the first pool, the other of the second and the third has all ZIL + L2ARC SSDs)
10G Intel NIC

2 zpools are in place:
First pool:
25x1,8TB 2.5" SAS 10k configured as 12x mirror of 2 disk with 1 spare.
2x HGST SAS12G ZeusIOPS 800GB as mirrored ZIL
1x Samsung SAS 12G PM 480GB as L2ARC
Dataset and pool size 16K, dedup on, compression lz4, sync always, atime off

Second pool:
14x 4TB SAS 7,2k configured as 6x mirror of 2 disk with 2 spares.
2x HGST SAS12 ZeusIOPS 800GB as mirrored ZIL
1xSamsung SAS 12G PM 480GB as L2ARC
Dataset and pool size 16k, dedup off, compression lz4, sync always, atime off


Performance prior to the hardware change:
First pool:
~ 280MB/s write
~ 1100MB/s read

Second pool:
~ 95MB/s write
~ 320MB/s read

Performance after the hardware change:
First Pool:
~ 25MB/s write
~ 80MB/s read

Second pool:
~20MB/s write
~40MB/s read

We had issues with the dual quad core configuration as with dedup and heavy load NFS would time out as dedup used 100% of the CPU in a peak.
To work around this issue we replaced the hardware with 2x 10Core as mentioned above. Due to the old PSU with the old board we just went for another Supermicro Server with board, CPU and newer more efficient PSU, configured BIOS the same, added the HBAs disk etc. and booted the system.
Reconfigured the NFS service to be allowed to use a max of 18 cores and that's it.

I am quite confused what might be the cause of this issue.

- Pools are not degraded
- SMART for all disk is fine
- network test shows 10GbE transfers
- system has way more power now..

 

[Herr_Merlin]

for further testing I disabled jumbo frames.. only tested the first pool so far:
~280MB/s write
but read is down to ~400MB/s

anyone any ideas at all?

 

[VioletDragon]

for further testing I disabled jumbo frames.. only tested the first pool so far:
~280MB/s write
but read is down to ~400MB/s

anyone any ideas at all?

Jumbo Frames needs to be enabled on the Managed Switch or if link only needs to be enabled each side. This problem on 10gig or 1gig? any Link aggregated setup?

 

[Herr_Merlin]

Jumbo frames were on on switch, VMware and FreeNAS.
I disabled on all sides.
We are talking dual 10G configured just as link fail over.

 

[VioletDragon]

Jumbo frames were on on switch, VMware and FreeNAS.
I disabled on all sides.
We are talking dual 10G configured just as link fail over.

Ok Jumbo can be good or bad, cause bad performance in some cases. I guess you have Link Aggregation then? test on a Single Link without any Link aggregation.

 

[Herr_Merlin]

Yep but no lcap.
I don't get. Why prior to the hardware upgrade it ran fine and now not anymore..
But single link vs the fail over lagg shows no difference

 

[VioletDragon]

Yep but no lcap.
I don't get. Why prior to the hardware upgrade it ran fine and now not anymore..
But single link vs the fail over lagg shows no difference

Hardware change can cause all sorts of strange activty. One thing to look at is the link speed of the card. run sudo lspci -vv and make sure that the cards are working at correct link speeds, different motherboards can have different PCI-E Lane speed.

 

[Spearfoot]

Hi all,

we are running a freenas as NFS datastore for our LAB vmware cluster.
Past hardware:
Some Supermicro X9 with 2x Xeon E5 Quadcore V0 3,4Ghz HT disabled

Current Hardware:
Some Supermicro X9 with 2x Xeon E5 Tencore V2 2,8Ghz HT disabled

Hardware which did not change:
256GB DDR3 1600Mhz
3x HP H240 SAS12G HBA (1 HBA holds all disk of the first pool, the other of the second and the third has all ZIL + L2ARC SSDs)
10G Intel NIC

2 zpools are in place:
First pool:
25x1,8TB 2.5" SAS 10k configured as 12x mirror of 2 disk with 1 spare.
2x HGST SAS12G ZeusIOPS 800GB as mirrored ZIL
1x Samsung SAS 12G PM 480GB as L2ARC
Dataset and pool size 16K, dedup on, compression lz4, sync always, atime off

Second pool:
14x 4TB SAS 7,2k configured as 6x mirror of 2 disk with 2 spares.
2x HGST SAS12 ZeusIOPS 800GB as mirrored ZIL
1xSamsung SAS 12G PM 480GB as L2ARC
Dataset and pool size 16k, dedup off, compression lz4, sync always, atime off


Performance prior to the hardware change:
First pool:
~ 280MB/s write
~ 1100MB/s read

Second pool:
~ 95MB/s write
~ 320MB/s read

Performance after the hardware change:
First Pool:
~ 25MB/s write
~ 80MB/s read

Second pool:
~20MB/s write
~40MB/s read

We had issues with the dual quad core configuration as with dedup and heavy load NFS would time out as dedup used 100% of the CPU in a peak.
To work around this issue we replaced the hardware with 2x 10Core as mentioned above. Due to the old PSU with the old board we just went for another Supermicro Server with board, CPU and newer more efficient PSU, configured BIOS the same, added the HBAs disk etc. and booted the system.
Reconfigured the NFS service to be allowed to use a max of 18 cores and that's it.

I am quite confused what might be the cause of this issue.

- Pools are not degraded
- SMART for all disk is fine
- network test shows 10GbE transfers
- system has way more power now..

I own a SuperMicro X9DRi-LN4F system (see 'my systems' below) -- it is very sensitive to the PCIe slot settings and won't boot at all if these aren't configured correctly.

I assume you have installed the latest BIOS on your systems? If not, I would do that first.

Then I would double-check the PCIe settings and make sure they match the generation and width of the boards you have installed.

 

[Herr_Merlin]

Latest Bios applied and those settings set. No change
Read still is slower as prior

 

[Spearfoot]

Latest Bios applied and those settings set. No change
Read still is slower as prior

My settings probably won't work for your server -- the slots have to be configured for your particular cards.

Or perhaps I misunderstood you?

 

[Herr_Merlin]

I have configured mine according to slot layout and usage.
Slots with controllers run at x8 gen3.
Latest Bios installed etc.

 

[Samuel Tai]

Have you confirmed your H240s are running in HBA mode?

 

[Herr_Merlin]

yes they are. Those are HBA by nature but you can flash them with the HP Storage toolkit to do some sort of strange software RAID.. anyways they are flashed to HBA.
From the freenas OS point of view only the CPUs, Mainboard (Chipset is the same on old and new) and NICs changed. To more power. The positive effects of the higher CPU power I can already notice. The NFS share does not time out anymore under heavy load (currently just running ~30 VMs from the pool with the 1,8TB SAS drives)

 

[Herr_Merlin]

Fixed but don't know how.
migrated all VM's from the pool to another SAN and back. Since than it's running as desired.