10 Gig Networking Primer

10 Gig Networking Primer

Snow

Patron
Joined
Aug 1, 2014
Messages
309
I know this may be out of some home gamer budget's but the cloud router switch 317-1G-16s+RM is pretty cheap 370$ on amazon. I do not use it as a router just as a 10Gb switch. Also I use Mellanox connectx-2 cards they are great. You can find these cards for under 45$ for 2, they work with Freenas/FreeBSD Drivers. Mellanox say's these cards are not supported for Win on there web site. I am about 90% sure this is to push you to buy there newer cards. As you can use the connectx3 driver's and it works the same. I tested it in 4 different machines and it worked fine. I have all options for tuning in Windows ones the drivers where Installed, the drivers you need for the connectX-2 are 5.10 or 4.95. The cards are listed in the .inf so it's supported (unofficially).
 

bestboy

Contributor
Joined
Jun 8, 2014
Messages
198
While we're at the topic of Mikrotik:

The CRS305-1G-4S+IN coming out soonish looks very interesting for home users, too. It's a fanless
Five-port desktop switch with one Gigabit Ethernet port and four SFP+ 10Gbps ports
Prices are expected to be around 100 EUR resp. 120 USD
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,455

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,455

Snow

Patron
Joined
Aug 1, 2014
Messages
309
That's Kinda how I see it, if 10GB is going to be main Stream one day why not just get the big guy. Or If I setup a JBOD I have free 10Gb/s ports free.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
That's Kinda how I see it, if 10GB is going to be main Stream one day why not just get the big guy. Or If I setup a JBOD I have free 10Gb/s ports free.

10GbE is already a has-been in many data centers, as the race to adopt 25/40/100GbE has been on for quite some time.

https://www.datacenterknowledge.com/networks/year-100gbe-data-center-networks

The real question is what is going to happen in the data center. We've been observing a shift towards massive scale stuff that will quite frankly not be justifiable in the home for decades, or even longer. The data center technology traditionally drove the development of inexpensive consumer technology, but this seems to have broken down, and we're not seeing lots of affordable new 10G stuff, even more than 15 years after the introduction of 10G. A large part of this seems to be the inherent limitations in copper, and the fact that we reached a plateau where 1G is really sufficient for most uses. If that's the case, then that suggests that SFP+ is really the way to go for the time being, as the stuff is being stripped out of data centers and being dumped on eBay, and there are a few reasonable small-scale entrants for new SFP+ 10G hardware.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Yeah, but new Mikrotik against used Arista is like comparing a new Ford Fiesta to a used Porsche 911.
True. I would probably go for the used gear if it were me, but the Arista is probably a little louder than the Mikrotik so I am sure it isn't for everyone.
 

l@e

Contributor
Joined
Nov 4, 2013
Messages
143
Those mikrotik even when i swos they have the limited throughput based on different packet sizes. So better to go with a real switch. Im with @Chris Moore at using second hand for matter of cost. The 10g new is still at high price.
 
Last edited:

Snow

Patron
Joined
Aug 1, 2014
Messages
309
Those mikrotik even when i swos they have the limited throughput based on different packet sizes. So better to go with a real switch. Im with @Chris Moore at using second hand for matter of cost. The 10g new is still at high price.
If you like I could Post some ipref screen shots of Mellanox Ctx2 Via Om4 to MicroTik DAC to CiscoSG350X to DAC 3M to Mellanox Ctx2 at FreeNAS.
I have not had any problems with speeds to my Win10 Rig I can Transfer at 900MB/s to and from my FreeNAS. Not sure what to tell you I use route-os and was told not to use switch-os on the Mikrotik forums from a Dev do to problem's they had with it. Does it only do swos ?

One thing to note is yes they have problem's with laGG on more then 1 10Gb Connection. This is do with how the chip sets are handling the hand off to each other it could be fixed with a software update from my understanding. If you are looking to go beyond 10Gb/s looking at the used market will set you back a good chunk of change. I know I was looking at Cisco because I am a fan boy. If you need more then 10GB/s then I would say plain on spending some bank. At lest this will Drive other's like Net Gear, Cisco and others to price match a competitive new offering. I agree we will not see 25/40/100GB/s in the home market in the usa for awhile (10-20 Years If we are lucky).
 

rmccullough

Patron
Joined
May 17, 2018
Messages
269
I think a better option for home users is the Mikrotik CSS326-24G-2S+RM 24 port Gigabit Ethernet switch with two SFP+ ports. <$150, but only 2 SFP+ ports. I think this would work for most home users. Use the SFP+ connections to FreeNAS and between switches, then 1GbE to other machines.

@jgreco i think another factor that has affected the flow of data center network tech trickling into the home is the greater focus on WiFi. I don’t see this changing as the most home users are not going to install “smurf” tube or re-wire for new network technology. The investment here will be for greater wireless speed.
 

l@e

Contributor
Joined
Nov 4, 2013
Messages
143
If you like I could Post some ipref screen shots of Mellanox Ctx2 Via Om4 to MicroTik DAC to CiscoSG350X to DAC 3M to Mellanox Ctx2 at FreeNAS.
I have not had any problems with speeds to my Win10 Rig I can Transfer at 900MB/s to and from my FreeNAS. Not sure what to tell you I use route-os and was told not to use switch-os on the Mikrotik forums from a Dev do to problem's they had with it. Does it only do swos ?
Im not doubting that.
What intended to point is the difference in specs mikrotik (i use them a lot and im happy with them) compared to other hw switches. Basicaly a hw switch like cisco, brocade, hpe, will have a total throughput equal to the sum of the maximum of each port max. So ie a 4 port 10g + 1 port 1g will have a total throughput of 82Gbps. The difference with mikrotik crs305_1g_4s_in based on specs published is not guaranteeing those 82Gbps in throughput based on all packet sizes. Still it can achieve way more than the 2 ports only can generate. So for that reason you can not feel it at one to one connection. If you try to overload all ports at the same time for sure you will feel the difference.
 

rmccullough

Patron
Joined
May 17, 2018
Messages
269
And to clarify my suggestion, I think that mikrotik is a better option because I think you will have much fewer 10GbE connections than 1GbE. So the 24x1Gb connections satisfy your existing stuff, and use the 10Gb for backbone between switches and connection to FreeNAS.
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
Pretty much what I am doing.
And to clarify my suggestion, I think that mikrotik is a better option because I think you will have much fewer 10GbE connections than 1GbE. So the 24x1Gb connections satisfy your existing stuff, and use the 10Gb for backbone between switches and connection to FreeNAS.

I hope they fix the software problem with LAGG as it would be nice to Use more then 1x10Gb For my JBOD Setup I have Planed. If so I may have to buy one of the used Quanta LB6M old amazon switchs, I am a sucker for GUI's But you end up in the CLI 9 times out of 10 any way's lol.
 

rmccullough

Patron
Joined
May 17, 2018
Messages
269
I forgot to mention, the other big thing (for me anyway) is the mikrotik 24 port switch is fanless. In my crawl space where FreeNAS is I don’t care, but in office I do. This is a deal breaker and worth potentially slightly lower max throughput (all ports maxed), which I am much less likely to exhaust.

I had an older 24x100MbE switch there about 10 years ago. The fan drove me nuts and was the major catalyst for me upgrading to 1GbE.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
@jgreco i think another factor that has affected the flow of data center network tech trickling into the home is the greater focus on WiFi. I don’t see this changing as the most home users are not going to install “smurf” tube or re-wire for new network technology. The investment here will be for greater wireless speed.

That's definitely an angle, and it goes to reinforce the idea that there's a bit of a speed cap as to what's really necessary for consumer gear. People want their wifi, and they want good coverage, but they really don't care about speed, which shows in the proliferation of all these crappy "mesh" home wifi solutions, which have generally terrible performance because you're running so much stuff over the air. As wifi bandwidth increases, the distance between radios has to be reduced to increase speed, so the real solution is to deploy a bunch of wired access points, which can give you several gigabits of aggregate throughput without a ton of investment, when it's well designed. Most of these mesh systems, though, are crappy technology designed to "look elegant" that gets sited suboptimally and works even worse.
 

l@e

Contributor
Joined
Nov 4, 2013
Messages
143
That's definitely an angle, and it goes to reinforce the idea that there's a bit of a speed cap as to what's really necessary for consumer gear. People want their wifi, and they want good coverage, but they really don't care about speed, which shows in the proliferation of all these crappy "mesh" home wifi solutions, which have generally terrible performance because you're running so much stuff over the air. As wifi bandwidth increases, the distance between radios has to be reduced to increase speed, so the real solution is to deploy a bunch of wired access points, which can give you several gigabits of aggregate throughput without a ton of investment, when it's well designed. Most of these mesh systems, though, are crappy technology designed to "look elegant" that gets sited suboptimally and works even worse.
also there is the down side of having a lot of antennas near each other because of interference, since channels become overloaded.
but still 2-3 AP with AC capability are good to have since 5G has limited range.
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
also there is the down side of having a lot of antennas near each other because of interference, since channels become overloaded.
but still 2-3 AP with AC capability are good to have since 5G has limited range.
That's Why I love MikroTik you can cherry pick what you use off of there switches, if you look at a cisco system to set up a proper mesh network your looking at spending some money. Thats why I follow the RFC 1925 Rule you Can pick 2 and only 2 but you can not have all three, Good, Fast, & Cheap. I Do not want to break the bank but I want it to work right and have 99% up time. I was looking at Net Gears mesh network and it looks cheap.


RFC 1925 Rule
 

l@e

Contributor
Joined
Nov 4, 2013
Messages
143
That's Why I love MikroTik you can cherry pick what you use off of there switches, if you look at a cisco system to set up a proper mesh network your looking at spending some money. Thats why I follow the RFC 1925 Rule you Can pick 2 and only 2 but you can not have all three, Good, Fast, & Cheap. I Do not want to break the bank but I want it to work right and have 99% up time. I was looking at Net Gears mesh network and it looks cheap.


RFC 1925 Rule
If u can lay down some cables better avoid mesh wls. Give a look at unifi also. I use their ac ap and are not expensive. The rf part is ok (i wold buy ruckus if i had the money becase their rf is unbeatable) and the controller is free. The controller helps on distribution of channels also if any nearbt ap is interfearing will change the ap to best and no noise channel.
 
Top