10G Chelsio T3 not recognized on a Supermicro 2758F

angoll

Cadet
Joined
Mar 5, 2016
Messages
5
Hi all,

First thanks in advance, I looked in the forums and I found people with similar setup working, but I'm not able to find why is not working.

I bought a Chelsio N320 from ebay to connect my Windows with my Nas via a 10Gb link. But I'm unable to get the Chelsio cart to be recognized by the motherboard.

From the label: CC2-N320E-SR (B) this card should be T3 and supported by the driver cxgb that is already loaded in the kernel, I also added the tunable ilf_cxgb_load = "YES" to the loader, but without luck.
  • ifconfig doesn't show the interface
  • pciconf -lv only show the Intel Atom.

I download a ISO from Chelsio (Chelsio Boot Software) and the No card where detected. I tried 2 cards, also I plugged the cards to a Windows Machine and the windows seems to recognize the cards without a problem.

I'm starting to think that maybe I have to modify the BIOS Setup, I tried diferent options but I wasnt able to make it to work. Also, not sure if I have to do it, but I enabled the I2C1/I2C2 jumpers (PCI-E Slot SMB Enable in the manual).

History of the board, I had the board fix and for the C2000 bug (watchdogs) and supermicro tested the board on Feb 2018.
Firmware Revision: 02.23 Build Time: 11/20/2015
BIOS Version: 2.1 Build Time: 01/18/2018

Do you know how can I make it work?
If you need more info please let me know.

Thanks
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
I plugged the cards to a Windows Machine and the windows seems to recognize the cards without a problem.
Sounds like the motherboard for sure.
It may help anyone trying to help to know the mobo model number and the
Version of FreeNAS you are running.
My recommendation is to fire up a secure shell and run some dimidecode
commands to check your PCIe slots.
dmidecode -t 9
Please post the output here and we might be able to help.
By default, the PCIe slots should be enabled in the bios and should
be listed but let's see what the above command shows as a first step.
 

angoll

Cadet
Joined
Mar 5, 2016
Messages
5
Hi

Many thanks BigDave for your reply.

I'm using FreeNas version 11.1-U4

I run a dmidecode -t 0,1,2,9 , first more info about the motherboard and at the end the results -type 9

Code:
# dmidecode 3.1
Scanning /dev/mem for entry point.
SMBIOS 2.8 present.
53 structures occupying 2186 bytes.
Table at 0x7F4D8000.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
	Vendor: American Megatrends Inc.
	Version: 2.1
	Release Date: 01/18/2018
	Address: 0xF0000
	Runtime Size: 64 kB
	ROM Size: 6144 kB
	Characteristics:
		PCI is supported
		BIOS is upgradeable
		BIOS shadowing is allowed
		Boot from CD is supported
		Selectable boot is supported
		BIOS ROM is socketed
		EDD is supported
		5.25"/1.2 MB floppy services are supported (int 13h)
		3.5"/720 kB floppy services are supported (int 13h)
		3.5"/2.88 MB floppy services are supported (int 13h)
		Print screen service is supported (int 5h)
		8042 keyboard services are supported (int 9h)
		Serial services are supported (int 14h)
		Printer services are supported (int 17h)
		ACPI is supported
		USB legacy is supported
		BIOS boot specification is supported
		Targeted content distribution is supported
		UEFI is supported
	BIOS Revision: 5.6

Handle 0x0001, DMI type 1, 27 bytes
System Information
	Manufacturer: Supermicro
	Product Name: A1SAi
	Version: 123456789
	Serial Number: ................
	UUID: Not Settable
	Wake-up Type: Power Switch
	SKU Number: 081315D9
	Family: SMC X10

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
	Manufacturer: Supermicro
	Product Name: A1SRi-2758F
	Version: 1.00
	Serial Number: ......................
	Asset Tag: To be filled by O.E.M.
	Features:
		Board is a hosting board
		Board is replaceable
	Location In Chassis: To be filled by O.E.M.
	Chassis Handle: 0x0003
	Type: Motherboard
	Contained Object Handles: 0

Handle 0x001C, DMI type 9, 17 bytes
System Slot Information
	Designation: Slot1 PCI-E 2.0 X8
	Type: x8 PCI Express
	Current Usage: In Use
	Length: Long
	ID: 0
	Characteristics:
		3.3 V is provided
		Opening is shared
		PME signal is supported
	Bus Address: 0000:00:03.0


Thanks
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
Can I assume correctly that that the output you posted is with one of the Chelsio cards installed in the PCIe slot, because it shows "IN USE".
Things to try;
Set your bios back to default settings.
Remove any Tunables you have added to Freenas and reboot.
Place jumpers back in original positions.
Install (in the PCIe slot) another card that you have, that works with FreeNAS.
Contact customer support.
 

angoll

Cadet
Joined
Mar 5, 2016
Messages
5
Thanks BigDave for the reply, yes the results was with the card in, and with the Bios set to the default values.

So after a day of try and error, I was able to make the card be recognized by FreeNas.


I reset the bios again, removed the jumpers and to eliminate the tunables/freenas I use a ISO file from the manufacter to upgrade the firmware, to check if the card was detected. After tring a few combinations I changed the value:

PCIe Slot 1 Bifurcation from Auto to X4X4

I managed to get the card recognized, booted freenas and check the dmesg, from there the card was with an old firmware, not suported by the driver, I upgraded the firmware to the last version.

I tried to set bios value back to auto but with out luck.

Code:
cxgbc0: <Chelsio T320, 2 ports> mem 0xdf181000-0xdf181fff,0xdf180000-0xdf180fff irq 23 at device 0.0 on pci5
cxgbc0: PCIe x4 Link, expect reduced performance
cxgbc0: using MSI-X interrupts (9 vectors)
cxgb0: <Port 0 10GBASE-R> on cxgbc0
cxgb0: Using defaults for TSO: 65518/35/2048
cxgb0: Ethernet address: 00:07:43:0c:xx:xx
cxgb1: <Port 1 10GBASE-R> on cxgbc0
cxgb1: Using defaults for TSO: 65518/35/2048
cxgb1: Ethernet address: 00:07:43:0c:xx:xx
cxgbc0: Firmware Version 7.11.0
cxgb0: link state changed to DOWN
cxgb0: link state changed to UP



Iperf3 Results doesn't make sense to me:

I did some iperf3 testing, but I'm not sure if the numbers are correct, also don't fully understand the PCI Bus.

Can some point me to the right direction as how to check this numbers?

My full 10G setup:
  • One card on the FreeNas server, as showed above
  • One on a Windows machine,
    I executed get-netadapterhardwareinfo and I can see that the PcieLinkWith of the Ethernet 3 and Etherned 4 (both are from the Chelsio card) each have a value of 2.5 GT/s

Iperf3 tests:

  • windows (client) --> freenas (server)
    iperf3 -c 192.168.10.5
    iperf3 --server -B 192.168.10.5
Code:
[  4] local 192.168.10.3 port 50795 connected to 192.168.10.5 port 5201
[ ID] Interval		   Transfer	 Bandwidth
[  4]   0.00-1.00   sec   743 MBytes  6.23 Gbits/sec
[  4]   1.00-2.00   sec   780 MBytes  6.54 Gbits/sec
[  4]   2.00-3.00   sec   784 MBytes  6.57 Gbits/sec
[  4]   3.00-4.00   sec   783 MBytes  6.57 Gbits/sec
[  4]   4.00-5.00   sec   783 MBytes  6.57 Gbits/sec
[  4]   5.00-6.00   sec   783 MBytes  6.56 Gbits/sec
[  4]   6.00-7.00   sec   782 MBytes  6.56 Gbits/sec
[  4]   7.00-8.00   sec   781 MBytes  6.55 Gbits/sec
[  4]   8.00-9.00   sec   780 MBytes  6.54 Gbits/sec
[  4]   9.00-10.00  sec   783 MBytes  6.57 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bandwidth
[  4]   0.00-10.00  sec  7.60 GBytes  6.53 Gbits/sec				  receiver

  • windows (client) --> freenas (server) Reverse
    iperf3 -c 192.168.10.5 -R
    iperf3 --server -B 192.168.10.5
Code:
Reverse mode, remote host 192.168.10.5 is sending
[  4] local 192.168.10.3 port 50849 connected to 192.168.10.5 port 5201
[ ID] Interval		   Transfer	 Bandwidth
[  4]   0.00-1.00   sec   721 MBytes  6.04 Gbits/sec
[  4]   1.00-2.00   sec   738 MBytes  6.19 Gbits/sec
[  4]   2.00-3.00   sec   728 MBytes  6.11 Gbits/sec
[  4]   3.00-4.00   sec   750 MBytes  6.29 Gbits/sec
[  4]   4.00-5.00   sec   753 MBytes  6.32 Gbits/sec
[  4]   5.00-6.00   sec   733 MBytes  6.15 Gbits/sec
[  4]   6.00-7.00   sec   756 MBytes  6.34 Gbits/sec
[  4]   7.00-8.00   sec   758 MBytes  6.36 Gbits/sec
[  4]   8.00-9.00   sec   759 MBytes  6.36 Gbits/sec
[  4]   9.00-10.00  sec   755 MBytes  6.33 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bandwidth	   Retr
[  4]   0.00-10.00  sec  7.28 GBytes  6.25 Gbits/sec	0			 sender
[  4]   0.00-10.00  sec  7.28 GBytes  6.25 Gbits/sec				  receiver


  • Freenas (client) --> windows (server)
    iperf3 -c 192.168.10.3
    iperf3 --server -B 192.168.10.3
Code:
[  5] local 192.168.10.3 port 5201 connected to 192.168.10.5 port 46244
[ ID] Interval		   Transfer	 Bandwidth
[  5]   0.00-1.00   sec   263 MBytes  2.21 Gbits/sec
[  5]   1.00-2.00   sec   387 MBytes  3.25 Gbits/sec
[  5]   2.00-3.00   sec   389 MBytes  3.26 Gbits/sec
[  5]   3.00-4.00   sec   384 MBytes  3.22 Gbits/sec
[  5]   4.00-5.00   sec   342 MBytes  2.87 Gbits/sec
[  5]   5.00-6.00   sec   380 MBytes  3.18 Gbits/sec
[  5]   6.00-7.00   sec   381 MBytes  3.20 Gbits/sec
[  5]   7.00-8.00   sec   388 MBytes  3.25 Gbits/sec
[  5]   8.00-9.00   sec   387 MBytes  3.24 Gbits/sec
[  5]   9.00-10.00  sec   387 MBytes  3.25 Gbits/sec
[  5]  10.00-10.31  sec   121 MBytes  3.27 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bandwidth
[  5]   0.00-10.31  sec  3.72 GBytes  3.10 Gbits/sec				  receiver


I also run iperf3 -c 192.168.10.3 -P 4 to run in parallel an I get
Code:
[SUM]   0.00-10.32  sec  7.19 GBytes  6.18 bits/sec	   sender
[SUM]   0.00-10.32  sec  7.19 GBytes  6.18 Gbits/sec	 receiver



Using the SMB I'm getting around 500 MB/s read and write speeds to a Sata III SSD, I didn't touch the MTU size as share started not working if I increase it.

Does this numbers make sense?

PS: sorry if I changed a bit the topic of this post, I think the networking part are related to the hardware setup/problems. If I should open a diferent thread I will.

Thanks
 

Jpmur

Cadet
Joined
May 7, 2020
Messages
1
I do have exactly the same problem with a N320E and the ASRI-2758F mobo from supermicro.

I tried a cheap sata card earlier in the same pciE. So I know that the slot works. The N320E according to the specs is pci1.1 and the PCI from the mobo is 2.0. So no worries there...

A thing who came suspicious is and I saw that the N320E consumption is 15w according to the specs.

Knowing that the board is a very low powered system with a small atom of about 20w. Knowing that the board have its own power stages that you just have to put 12v on the board. Would it be possible that the N320E are too much power hungry for the pciE?

I will try what you did to make it work but that may be why the bios refused to start the card. And that's why we need to update it?

Will see for myself!
 

James Gardiner

Dabbler
Joined
Jul 14, 2017
Messages
19
Hi Guys, adding to this, I am using this card. using iperf, I get 9.3Gbits/s in sending to the FreeNAS, but only 1.7Gbit/sec pulling from the FreeNAS server.
I have some nvme M.2 SSD in it and I can get 2100MB/s local, but only about 170MB/s over NFS share. and considering the results of the iperf3, that sounds about right.

Is there anything I can do to utilise this card or should I dump it and go to another?

Thanks..
 
Top