I have two SuperMicro servers with nearly identical motherboards, dual cpus, 128GB ram and LSI HBAs.
The larger chassis runs freenas on the bare metal, Intel X550-T2, has 3 raidz2 vdevs of 6 x 6TB drives. It also setup a 4 ssd mirrored pool and a NVMe single drive pool for testing.
The smaller chassis runs freenas in a hyper-v client, Chelsio T520-BT and has the HBA with 6 x 8TB drive passed through. They are setup in a raidz2 configuration as well.
My testing desktop has a 10Gbe nic, NVME drive, 32GB RAM, 9900k and runs Windows 10 Pro 1903.
In testing, I copy a couple large video files between the desktop and server. In all tests, it's the same files together, to the same share and both on a fresh reboot and after a few hours running.
The Intel X550 server will start off slower on reads around 200-300 Mb/sec and then get to 600-700Mb/sec. The writes max out around 500Mb/sec as well. This is across the three different storage types (HDD pool, SSD pool and NVME pool). I do notice that the disks are not read from on subsequent reads, meaning that it should be in memory.
The Chelsio T520-BT server is much different. It will start around 500 Mb/sec read and then ramp up to 1Gb/sec. The writes seems to max out around 500-600Mb/sec. This is just against the single HDD pool using raidz2.
Is the Chelsio card just that much better? Would it be worth trying to swap the cards between servers? I was lucky to find the card on ebay for $90 USD and would be open to getting a second, but they aren't easy to find.
The larger chassis runs freenas on the bare metal, Intel X550-T2, has 3 raidz2 vdevs of 6 x 6TB drives. It also setup a 4 ssd mirrored pool and a NVMe single drive pool for testing.
The smaller chassis runs freenas in a hyper-v client, Chelsio T520-BT and has the HBA with 6 x 8TB drive passed through. They are setup in a raidz2 configuration as well.
My testing desktop has a 10Gbe nic, NVME drive, 32GB RAM, 9900k and runs Windows 10 Pro 1903.
In testing, I copy a couple large video files between the desktop and server. In all tests, it's the same files together, to the same share and both on a fresh reboot and after a few hours running.
The Intel X550 server will start off slower on reads around 200-300 Mb/sec and then get to 600-700Mb/sec. The writes max out around 500Mb/sec as well. This is across the three different storage types (HDD pool, SSD pool and NVME pool). I do notice that the disks are not read from on subsequent reads, meaning that it should be in memory.
The Chelsio T520-BT server is much different. It will start around 500 Mb/sec read and then ramp up to 1Gb/sec. The writes seems to max out around 500-600Mb/sec. This is just against the single HDD pool using raidz2.
Is the Chelsio card just that much better? Would it be worth trying to swap the cards between servers? I was lucky to find the card on ebay for $90 USD and would be open to getting a second, but they aren't easy to find.
