Chelsio T520-CR vs Chelsio T520-SO-CR

Scampicfx

Contributor
Joined
Jul 4, 2016
Messages
125
Hey guys,

I know that "SO" means "Server Offload", but what exactly are the differences between
- Chelsio T520-CR
- Chelsio T520-SO-CR?

There is a huge price difference between those two network adapters at our local dealer. In my opinion "Server Offload" sounds like an extra feature which imho whould mean that this network adapter is more expensive. However, this one is a lot cheaper than the T520-CR. Am I making any wrong assumptions? ;)

The reason for asking this point is, that recently I ordered a T520-CR via eBay. However, after installing this network adapter our system freezes during BIOS "Detecting onboard AHCI Devices". We use mainboard Supermicro X10SRH-CF and it freezzes at the Supermicro Logo. When looking into IPMI during this freezed Boot screen, we notice that the "PCH temperature sensor" displays "not present". This is kind of strange, because as soon as we unplug this network adapter, everything works smoothly again and PCH temperature sensor displays temperature correctly.

Personally, I hope that the network adapter we bought is kind of damaged. Therefore we want to buy an original new sealed network adapter and test it again.
It's very frustrating right now since our new FreeNAS server is just about to get started, but now remains shut down due to this network adapter issue :/

Our dealer offers both network adapters (Chelsio T520-CR and T520-SO-CR) and I'm a bit unsure which one to choose!

Thanks so much!

EDIT: There is an official statement from Chelsio:

T520-so will offload (tcp/rdma/iscsi) a handful of connections where the t520 will offload 32K connections. Offloaded connections see a 2x to 3x improvement in performance.

When I get correct and after stuyding the product sheets, this means:
T520-SO-CR offloads only 32 connections
T520-CR offloads 32.000 connections and is the more performant network adapter.

Cheers
Scampicfx
 
Last edited:
Joined
Feb 2, 2016
Messages
574
Search and ye shall receive...

Chelsio T520-SO-CR vs T520-CR

Everything I've read seems to indicate the performance increase isn't worth the $500 delta.

Cheers,
Matt
 

Brian Stretch

Dabbler
Joined
May 2, 2017
Messages
16
FWIW, I noticed these on eBay this morning:
http://www.ebay.com/itm/T520-CR-Chelsio-T520-CR-PCIe3-0x8-2-10GbE-SFP-NIC/252683834517
Which is sort of tempting, just because and assuming they're genuine, but for my use case 32 connections is more than enough. And you're almost to the cost of a T6225-SO-CR for ~$300 if you can find one. 25Gb optics are much more expensive than 10Gb though, ~$175 each?, and I don't know anyone who's tried the T6225 with FreeNAS yet. They are on the FreeBSD compatibility list.
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
Speaking about offload, FreeNAS 9.10 by default does not use any of those offloads that differentiate "SO" and full cards. TCP offload (TOE) can be enabled on 9.10 by adding "toe" to the interface options. iSCSI offload may get supported later in 11.x, but comparing to TOE it gives measurable benefits only when IOPS reaches hundreds of thousands, and throughout reaches multiple dozens of gigabits.
 

Scampicfx

Contributor
Joined
Jul 4, 2016
Messages
125
Thanks for your postings!

Meanwhile we're still stuck with the Supermicro Mainboard freezing during BIOS.

picture.jpg


Exactly at this screen, the system remains unresponsive...

In the last line you notice "Invoking Boot Menu". I pressed multiple times F11. But nothing more happens. (it is the same when pressing DEL -> then the message "Entering Setup" appears, but actually the system hangs!).

Since I still had the impression, that the network adapter is damaged, I ordered an additional network adapter.
I tested both Chelsio T520-CR and Chelsio T520-SO-CR in Supermicro X10SRH-CF.

I tried to install the network adapters into different PCIE slots.
I tried to disable the PCIE slot the network adapter is installed,.
I tried both T520-CR and T520-SO-CR.
I tried BIOS Boot DUAL, LEGACY and UEFI
I checked that "Network Stack" in PCIE BIOS Configuration is "Enabled".

However, none of the mentioned points did work!

BIOS Version 2.0a is installed...

What am I doing wrong?

EDIT: I found an YouTube Video that is showing exactly the same problem I have. Notice the flickering of the screen when it comes to the freeze! That's the same issue I have.


I found it here: https://forums.servethehome.com/index.php?threads/supermicro-board-stuck-on-logo-b2.6807/
 
Last edited:

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
For the first, I am always disabling that splash screen on all my boards to see all possible technical information. For the second, you may try to remove the NIC, go to the BIOS settings and disable BIOS for the respective PCIe slot (or all slots which you are not using for booting). I think sometimes problems like that may happen if you have too many adapters with own BIOS installed.
 

Scampicfx

Contributor
Joined
Jul 4, 2016
Messages
125
Dear mav,

thanks for your feedback.

I was in contact with Supermicro. They supplied a new BIOS version for this mainboard. Now, everything is working normal :)

Thanks!
 

Brian Stretch

Dabbler
Joined
May 2, 2017
Messages
16
I discovered another reason to favor the T520-SO-CR over the T520-CR: 9W vs. 12W power consumption. If you put a -CR in a low airflow desktop case it'll get extremely hot. Glad I borrowed one from work to test before buying. Similar story with the T6225 series, 10W vs. 13W. FreeNAS 11 does support the T6225 series.

EDIT: Some T520's have a dinky heatsink, some a big one. The T520-SO-CR I have has a dinky one. I suspect newer T520 builds got the big heatsink and all T6225's did but can't say for certain, and that any card with the dinky heatsink should get a fair amount of airflow.
http://productspec.ul.com/document.php?id=NWGQ.E311704
 
Last edited:

Holt Andrei Tiberiu

Contributor
Joined
Jan 13, 2016
Messages
129
T520-SO-CR has a smaller build and heatsink, it also has no inboard memory, it is the cheapest of the T5 series, I would not touch it for heavy-duty
work.

T520-CR is the mainstream adapter, onboard memory, and o lot of features, definitely go for this one, especially if you go ISCSI way.

T520-OCP-SO is the high-end adapter, it has ASIC for network related traffic, all, it is sold for system builders, I did not find a card with this chipset.
In other words, go for T520-CR, it will make Intel and Broadcom cards look bad, for performance point of view.






I discovered another reason to favor the T520-SO-CR over the T520-CR: 9W vs. 12W power consumption. If you put a -CR in a low airflow desktop case it'll get extremely hot. Glad I borrowed one from work to test before buying. Similar story with the T6225 series, 10W vs. 13W. FreeNAS 11 does support the T6225 series.

EDIT: Some T520's have a dinky heatsink, some a big one. The T520-SO-CR I have has a dinky one. I suspect newer T520 builds got the big heatsink and all T6225's did but can't say for certain, and that any card with the dinky heatsink should get a fair amount of airflow.
http://productspec.ul.com/document.php?id=NWGQ.E311704
 

Octopuss

Patron
Joined
Jan 4, 2019
Messages
461
Isn't T520-SO-CR more than enough for home environment though?

I've been looking at Ebay from time to time, but I've never seen any of the newer revisions (which supposedly aren't as hot).
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
Isn't T520-SO-CR more than enough for home environment though?
Yes, should be enough. Full cards have benefits of TCP and iSCSI offload, but we don't even enable those features by default, since in most environments performance benefits do not oгеweight risk of using rarely used code. But all the requited code is built into FreeNAS, so somebody who wish to experiment is welcome to do it.
 

Holt Andrei Tiberiu

Contributor
Joined
Jan 13, 2016
Messages
129
For home yes.
I was referring to production, heavy IO use.
I will get my T520-CR cards in 2-3 weeks and will post some benchmarks.
Currently, I use N320 from chelsio, but I am limited to 4 GBS/PORT. I cannot pass 1250 MB/sec in iSCSI.
In the servers I user Intel 520 Series, same, PCI-EX 2.0
N320 cards are PCI-EX 2.0, and I am limited by the 8 x 500 megabit of those cards. T520 are PCI-EX 3.0, they have 8 x 984 MB/Sec bandwidth.


1579201859805.png


Yes, should be enough. Full cards have benefits of TCP and iSCSI offload, but we don't even enable those features by default, since in most environments performance benefits do not oгеweight risk of using rarely used code. But all the requited code is built into FreeNAS, so somebody who wish to experiment is welcome to do it.
 
Joined
Dec 29, 2014
Messages
1,135
Full cards have benefits of TCP and iSCSI offload
I have the following NIC card in both my FreeNAS units.
Code:
t5nex0: <Chelsio T580-LP-CR> mem 0xfb300000-0xfb37ffff,0xfa000000-0xfaffffff,0xf
bc04000-0xfbc05fff irq 64 at device 0.4 numa-domain 1 on pci14
cxl0: <port 0> numa-domain 1 on t5nex0
cxl0: Ethernet address: 00:07:43:30:f2:70
cxl0: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
cxl1: <port 1> numa-domain 1 on t5nex0
cxl1: Ethernet address: 00:07:43:30:f2:78
cxl1: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)

I don't know why, but enabling TOE on these cards caused NFS to have intermittent connectivity. My environment is mostly a lab, so I don't think it really matters. Still, I was troubled by not being able to use that feature. It manifested as 'NFS server not responding' messages, and then it would starting responding again after a while.
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
I don't know why, but enabling TOE on these cards caused NFS to have intermittent connectivity.
You may retry this on 11.3. New OS version may bring new fixes. There is no reason why it should not work, but each time I tried it before I hit some sort of issue. But there is a hope it will work one day. And there is a Chelsio FreeBSD committer to report the issues to.
 

dcol

Dabbler
Joined
May 1, 2020
Messages
28
I know this thread is a year old, but I have something to add for those suspecting issues with their Chelsio T520-CR.
This card can run excessively hot affecting performance, so I put a 50mm fan on top of the heatsink and now the card runs cool and gives solid performance. I used Fractal Silent R3 50mm. Easy to install, the mounting holes line up.
 

Gcon

Explorer
Joined
Aug 1, 2015
Messages
59
I know this thread is a year old, but I have something to add for those suspecting issues with their Chelsio T520-CR.
This card can run excessively hot affecting performance, so I put a 50mm fan on top of the heatsink and now the card runs cool and gives solid performance. I used Fractal Silent R3 50mm. Easy to install, the mounting holes line up.

necro'ing this thread some more. Do you have a low-airflow home NAS case, or something with good airflow like a rackmount server?

I'm looking at the Chelsio cards for a TrueNAS box after an intel X710 card started giving me grief. I have an X520-DA2 in the box at the moment, but it is only PCIe 2.0, and I want the best (yet affordable) PCIe 3.0 card I can find, that isn't the Intel X710. ServeTheHome recommend Chelsio then Intel, so investigating the Chelsio gear. Thanks.
 

bfarnam

Cadet
Joined
Nov 3, 2021
Messages
8
These cards do run HOT.
In my TrueNAS setup, I use the active cooling version of the T420-CR:
Chelsio T420-CR with Active (FAN) Cooling
110-1146-40 C0 (Copper Heatsink)
PCB FAB 100-1120-00 Rev C
110-1146-40 E0 (Natural Aluminum Heatsink)
PCB FAB 100-1120-00 Rev C
 

dcol

Dabbler
Joined
May 1, 2020
Messages
28
necro'ing this thread some more. Do you have a low-airflow home NAS case, or something with good airflow like a rackmount server?

I'm looking at the Chelsio cards for a TrueNAS box after an intel X710 card started giving me grief. I have an X520-DA2 in the box at the moment, but it is only PCIe 2.0, and I want the best (yet affordable) PCIe 3.0 card I can find, that isn't the Intel X710. ServeTheHome recommend Chelsio then Intel, so investigating the Chelsio gear. Thanks.
More bandwidth with the X710 due to PCIe 3.0. I run Intel X710 in LAGG mode. Haven't had an issue yet.
 
Top