Quick 10gbe upgrade question

AstroZombie

Explorer
Joined
Jul 20, 2014
Messages
76
So i'm patiently awaiting delivery of my Netgear GS110MX for this and just thought i'd check that I should be ok.

My plan is to have a 10gbe link between my server and my windows machine for large file copies. I know the windows box is a simple pci-e adaptor and bam problem solved.

From my research for freenas in theory if i pick up a SolarFlare SFN5122F and a SPF+ module it should be just plug and play and picked up in driver? (obviously with the correct IP assignments etc being done)
 

MikeyG

Patron
Joined
Dec 8, 2017
Messages
442
That Netgear switch appears to have only copper ports. You are planning to connect the SFP+ NIC to it? I believe the SolarFlare NICs are supported. If so it would be automatic. Best check the 10gbe thread in the forums though. I think people usually go with Intel or Chelsio.
 
Joined
Jul 2, 2019
Messages
648
That Netgear switch appears to have only copper ports. You are planning to connect the SFP+ NIC to it? I believe the SolarFlare NICs are supported. If so it would be automatic. Best check the 10gbe thread in the forums though. I think people usually go with Intel or Chelsio.
I was curious and the GS110MX has 2 Copper 10GigE ports. You could get a copper SPF+ module like this one on eBay: https://ebay.us/DqMh3f

The 10GigE SFP+ modules seem mighty expensive, though...

------
Edit: I just noticed the picture on the top of the GS110MX page shows 2 switches with SFP+ ports. They seem to be XS505M or XS508M models....
 
Joined
Jul 2, 2019
Messages
648
That Netgear switch appears to have only copper ports. You are planning to connect the SFP+ NIC to it? I believe the SolarFlare NICs are supported. If so it would be automatic. Best check the 10gbe thread in the forums though. I think people usually go with Intel or Chelsio.
I agree with @mgittelman - check the forums on which 10GigE NIC is best supported by FreeNAS TrueNAS Core (e.g., FreeBSD).
 

AstroZombie

Explorer
Joined
Jul 20, 2014
Messages
76
I read this resource for a good couple of pages and changed my choice to a Chelsio N320E. (Motherboard is an ASROCK X470D4U)

(10gtek spf+ module: https://www.amazon.co.uk/gp/product/B01M8O3MAL/ref=ppx_yo_dt_b_asin_title_o03_s00?ie=UTF8&psc=1)

I have just installed it in my server which can see the card however the link stays down and i'm getting no carrier when I check with ifconfig.

I have a CAT 6A cable going from the chelsio but no dice on the switch end in terms of activity.

Verified it's not the cables or port by switching in my 10gbe link on my PC.

Any ideas?
 
Last edited:

MikeyG

Patron
Joined
Dec 8, 2017
Messages
442
Do you have any way to test that card with a direct attach cable to another sfp+ port? Adding in the 10gtek feels like asking for trouble to me even though it should work. I assume the cables are relatively short? Those adapters don't put out a ton of power.
 

AstroZombie

Explorer
Joined
Jul 20, 2014
Messages
76
Do you have any way to test that card with a direct attach cable to another sfp+ port? Adding in the 10gtek feels like asking for trouble to me even though it should work. I assume the cables are relatively short? Those adapters don't put out a ton of power.
I have a 1gbe SFP adapter laying around from my ER-X.

I done some trouble shooting last night, I managed to bring the link up over a 1gbe port on the same switch for a little bit.

I also discovered something seems to be up with the switch. I use a ASUS XG-C100C in my windows PC that has been working fine in one of the 10gbe ports the moment I plugged it into the other I got activity but my PC proceeded to be unable to access sites at random and speed tests failed when trying to upload or received kilobits.

Pinging common sites from cmd also had timeouts.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
I would have just gone with the simple solution Intel x540. The chelsio card you choose is really old and it is also common for those rj45 conversion transceivers to just not work. They pull so much power compared to the sfp+ ones. They could just be out of spec for the card.
 
Last edited:

AstroZombie

Explorer
Joined
Jul 20, 2014
Messages
76
I would have just gone soon the simple solution Intel x540. The chelsio card you choose is really old and I is also common for those rj45 conversion transceivers to just not work. They pull so much power compared to the sfp+ ones. They could just be out of spec for the card.
You may have a point but the second issue I've discovered with the switch is concerning as it is repeatable.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
even though it should work.

And you think that, why, exactly?

I'm pretty sure Chelsio generally requires a vendor-locked optic. You also need to bear in mind that the SFP+ and card need to be able to communicate together, the SFP+ module is *smart*, and the card may very well be going "copper? Whatchu talkin' 'bout, Willis?"

The chelsio card you choose is really old

That's true as well. There've been a few folks showing up with the Chelsio 320's and various odd problems, many of which seem to be heat related, because the cards run warm.

and I is also common for those rj45 conversion transceivers to just not work. They pull so much power compared to the sfp+ ones. They could just be out of spec for the card.

Well, you basically have two routes to success here. Vendor-supplied 10GbE copper SFP+'s could theoretically draw as much power as they like because they can be coordinated with the switch to work. Just because the SFP+ spec specifies a 1.5W-per-SFP+ limit doesn't mean you couldn't engineer for more than that, but you have to remember that the big reason SFP+ exists is so that there can be high-density switching in the data center with 48 or 96 ports on a linecard. The other technique is to note that most of the energy is consumed in achieving distance, so what you see on the copper SFP+'s is a severe reduction in how far you can run the copper.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
What are my options here?

If you want to retain the copper switch, probably find an Intel X540 copper card. These were severely problematic years ago, but the last several folks thru here that I've talked to seem to be doing okay with them.

But the copper switch tends to be a limiting, expensive option in my opinion, which is why I push for hobbyists to shoot for SFP+ with vendor optics and SC fiber. Mikrotik has an excellent selection of small inexpensive 10G switches available.

For example, the CRS326-24G-2S has two SFP+ and 24 gigabit ports for around the same price as that Netgear you got, plus it's not a dumb unmanaged brick, but actually a decent (low end but still quite decent) managed switch with VLAN's and all the basic smart features available. It can also be used with RouterOS if you want it to be your home router (not something I suggest, but a great capability regardless).

If you really only needed to connect your PC and NAS while also being connected to a normal 1G network, Mikrotik makes the CRS305-1G-4S-IN for less than $150 which can act as an add-on to your network without actually replacing whatever switchgear you have. The downside is that the uplink to your network would be 1G.

This is the frustrating thing, it's harder to remediate stuff after the fact.
 

AstroZombie

Explorer
Joined
Jul 20, 2014
Messages
76
If you want to retain the copper switch, probably find an Intel X540 copper card. These were severely problematic years ago, but the last several folks thru here that I've talked to seem to be doing okay with them.

But the copper switch tends to be a limiting, expensive option in my opinion, which is why I push for hobbyists to shoot for SFP+ with vendor optics and SC fiber. Mikrotik has an excellent selection of small inexpensive 10G switches available.

For example, the CRS326-24G-2S has two SFP+ and 24 gigabit ports for around the same price as that Netgear you got, plus it's not a dumb unmanaged brick, but actually a decent (low end but still quite decent) managed switch with VLAN's and all the basic smart features available. It can also be used with RouterOS if you want it to be your home router (not something I suggest, but a great capability regardless).

If you really only needed to connect your PC and NAS while also being connected to a normal 1G network, Mikrotik makes the CRS305-1G-4S-IN for less than $150 which can act as an add-on to your network without actually replacing whatever switchgear you have. The downside is that the uplink to your network would be 1G.

This is the frustrating thing, it's harder to remediate stuff after the fact.

I think i'll go with option 1, after I return the chelsio and sfp+ adaptor it's not much more to pick up a X540.

I've contacted the retailer about the switch and asked for an RMA, the weird behaviour earlier on the other 10g port is still repeatable.

The switch itself is perfect for my use case at the moment I'm not running any VLANS, or link aggregation I just needed more 1gbe ports for some future expansion and two 10gbe ports for the PC and Server, all my management is done at the router level with my ER-4.

Thank you for your suggestions though I will keep them in mind if i want to explore in the future.
 
Last edited:

StoreMore

Dabbler
Joined
Dec 13, 2018
Messages
39
I do have a somewhat similar setup working. My custom NAS has a Chelsio 10G card installed (supermicro motherboard). Chelsio card was old used model I picked up for $50 from eBay. It was on the FreeNAS hardware list. This is plugged into my UniFi POE Switch via SFP+ (10G Tek Twin DAC Cable). My PC has an Aquantica 10G adapter on the motherboard and then I have this patched via a CAT6 cable out of my panel to SFP+ via an SFP+ to 10G Ethernet adapter.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
This is plugged into my UniFi POE Switch via SFP+ (10G Tek Twin DAC Cable)
I'm surprised using a DAC with ubiquity and chelsio worked.

 

StoreMore

Dabbler
Joined
Dec 13, 2018
Messages
39
Yep, I have had great luck with the 10G tek cables. They are using the switch as an intermediary so not sure if that makes a difference. I am using the DAC cables thay say for Ubiquiti. I did message 10Gtek originally before buying to confirm the cables from Chelsio cards to Ubiquiti switch and they did assured me they should work. Here's a link to the exact cable I bought from Amazon.
 
Top