10 Gig Networking Primer

10 Gig Networking Primer

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,455
and the controller is free.
...and can even be readily deployed in a FreeNAS jail, complete with automated Let's Encrypt certs.
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
If u can lay down some cables better avoid mesh wls. Give a look at unifi also. I use their ac ap and are not expensive. The rf part is ok (i wold buy ruckus if i had the money becase their rf is unbeatable) and the controller is free. The controller helps on distribution of channels also if any nearbt ap is interfearing will change the ap to best and no noise channel.
Why not use your MikroTik to run cAPs Man Mesh networking? it is pretty much the same thing as the Cisco controller.
There Aps are cheap I have 2 Indoors and one outdoors the waterproof ap was around 75$

CAPSMAN
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
if you look at a cisco system to set up a proper mesh network your looking at spending some money.

Yeah, but a Cisco mesh wifi network will still be crap compared to a Ubiquiti wired AP network. The Ubiquiti stuff is also relatively inexpensive.
 

l@e

Contributor
Joined
Nov 4, 2013
Messages
143
basically any mesh will be worse than having the AP wired to the switch since the first ap will condition all the throughput of the mesh. aslo every jump will add more latency, and in high speeds you will fell that it sucks.
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
I was taking about a wired ap mesh systems. The Cisco systems are awesome but you pay for it. I have not tried Ubiquiti. I can only add what others have told me about Ubiquiti. I've herd compared to Cisco Enterprises stuff is Night and day over Ubiquiti. But from my understand Ubiquiti is geared towards the Medium to Small business or home user's?
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
But from my understand Ubiquiti is geared towards the Medium to Small business or home user's?
I can't speak to Cisco, never configured that gear. In the past, Ubiquiti offered some pretty amazing hardware/software features compared to the competition. However, you had to work for it. Like RouterOS on Mikrotik, there were a bazillion options to play with and hence plenty of opportunities to screw it all up also. I was not a fan of the stability of early Ubiquiti gear, whereas Apple AP uptime has been limited strictly by the stability of the line power supply.

Apple APs are also much simpler to configure and set up by comparison (using a dedicated MacOS/iOS/Windows application helps immensely in that regard) but they also lack features such as the ability to set up guest VLANs that can talk to a limited set of on-site devices. (by default, all 1003 VLAN traffic on Apple APs can only go to the internet). FWIW, I use a combination of Apple base stations and Ubiquiti/Mikrotik networking gear here.

Now that Apple has left the AP market, I may switch to Ubiquiti - but only if there is a meaningful technical advancement in the market that my next generation of computer gear also supports. That's the part that a lot of folk seem to forget, i.e. it's all well and nice that they just bought a $800 AP but if nothing can connect to it at the claimed max speeds because no computer has yet shipped with compatible hardware/software then... why spend $800 on an AP? For the time being, 802.11ac serves me very well.

I would not characterize Ubiquiti as Medium/Small/Home-oriented. Rather, I'd say that Ubiquiti offers a gamut of gear from pro all the way down to the homeowner-oriented gear. The same was true of Cisco when they still owned Linksys.
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
@Constantin I see MikroTik as the new kids on the block they will have ups and downs. One thing I like is all there hardware uses pretty much the same OS so if they drop a Update for say there 24 port switch, it also goes to ever other device. I hope they did not bite off more then they can chew! The one thing that I dislike very much about Ubiquiti is you have to pay for their software/Key It feel's like Proprietary bull S***. I just think it should be included with the gear IMO.
 

bestboy

Contributor
Joined
Jun 8, 2014
Messages
198
to get back to the original topic 10 GbE:

The reason why I consider the 4+1 port mikrotik switch interesting for home users is that it is fanless and has just the right amount of ports to make the substantial but not exaggerated move towards 10 GbE adoption. 2 10GbE ports are too few. And more than 4 or 5 10GbE ports are typically not needed as the adoption is still slow and only a few selected applications will use 10GbE for the foreseeable future at home.
I'd say most home users will be fine connecting the NAS, their "workstation" and maybe a VM host to 10GbE. Then add another port of a future uplink for good measure and your done. What are home users supposed to do with the 24 10GbE ports of a screaming enterprise switch that needs cooling and ups the power bill?
So unless you are running a serious home lab I think the enterprise switches are either too small (2 ports) or too big (16+ ports) for regular home users that just want to get into the 10GbE game.
 
Last edited:

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,455
dislike very much about Ubiquiti is you have to pay for their software/Key
What? No, you never have to pay for their software. You can, if you choose, buy the Cloud Key, which is a self-contained computer (optionally with a NVR) running the controller software. Or you can run the controller software on another computer on your network (Windows, Mac, Linux, even in a FreeNAS jail). Or you can have UBNT host it for you in the cloud for whatever they charge. Or you can even host it in the cloud for free (for small networks, anyway) at hostifi.net. (Edit: or, for that matter, the controller software doesn't have to be kept running at all, though you do need to run it to configure your devices.)
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I would not characterize Ubiquiti as Medium/Small/Home-oriented.

They're not.

Rather, I'd say that Ubiquiti offers a gamut of gear from pro all the way down to the homeowner-oriented gear.

But this is also incorrect.

They're clearly targeted at market segments such as WISP and hospitality. Their product lines contain featuresets that were clearly designed for these purposes, and they would like to be taking market share away from Ruckus, Aruba, Cisco, etc. They have a complete stack for a hotel or restaurant where you want some AP's, a PoE switch to power them, a NAT router ("USG"), and possibly an on-site CloudKey to manage the thing. For the industries where they're used to paying many thousands of dollars for a solution, the low cost and tightly integrated featureset is a big draw. In the WISP market, you can see that there are a lot of building blocks for networks, especially things like the low end EdgeRouter series. When you're placing CPE on a mast or tower, you often don't have room for large gear, and you have special concerns for grounding and the like. They've done a good job addressing this kind of stuff.

So the flip side is that these features also make it great for techies who have other applications for it. I just finished configuring an EdgeRouter-X as an OOB OpenVPN server for data center management for a client. At $50, that's an amazing capability. Anyone who works with networking has probably worked with Vyatta, and Ubiquiti has done a good job on the featureset, so their gear is really like legos for networking geeks.

But none of it is really homeowner-grade stuff, i.e. pull it out of the box, turn it on, and it wizards you through a default setup. You do need to be willing to follow some more complicated directions and have some idea of what all the components are.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
So unless you are running a serious home lab I think the enterprise switches are either too small (2 ports) or too big (16+ ports) for regular home users that just want to get into the 10GbE game.
I agree to a point. It's entirely likely that the decision to "only" offer two 10GB/s interfaces was driven by thermal, cost, and use considerations. The first two are self-explanatory, the third depends on what marketing suggests to engineering as being the "best" features to offer.

I imagine a lot of SOHO applications use one of the the 10GB/s connection for a server and the other to link to other switches in the enterprise. It's a way to move a lot of data while only using one connection. In my home, I use one for the server, the other for a connection to the home office. That's where the bulk of the data flow is, hence the widest highway. As the chips for 10GB/s connections become less expensive and power hungry (i.e. hot), I expect the 10GB adoption to follow the same path as the prior evolution waves from 10->100 and then 100->1000 Mbit/s.

They're clearly targeted at market segments such as WISP and hospitality...But none of it is really homeowner-grade stuff, i.e. pull it out of the box, turn it on, and it wizards you through a default setup. You do need to be willing to follow some more complicated directions and have some idea of what all the components are.
On further reflection, you are absolutely right and thank you for the insight. I use some of their gear in a WISP-like application (linking multiple structures), bought more trying to adopt netblazr, and so on. However, allow me to quibble a little with the wizards comment. In my experience, the user experience depends a bit on the product and the time of adoption. In the past, I found the Ubiquiti AP management to be downright obtuse, the edgerouter web UI not much better.

However, Ubiquiti have improved the wizards in the Edgerouter significantly (I have no APs to play with ATM, so I can't comment on more recent implementations). The edgerouter is now a lot easier to use for default applications due to IP no longer being set on a weird address and the wizards. It is a shame that wizards are not (yet) available for IPSEC/L2TP, OpenVPN, etc. and that the setup of "advanced features" is typically described solely from a CLI perspective by the OEM. I am also not a fan of the auto-firewall feature silently adding firewall rules to the ruleset. making it that much harder to follow along and learn/troubleshoot.

To me, the Edgerouter is a lot like the FreeNAS. Very flexible, lots of options, and a steep learning curve for non-pro users if you want to get into "pro" features.
 

Sjöhaga

Dabbler
Joined
Apr 17, 2016
Messages
41
I decided to go with the Zyxel XGS1930-28 (which is 24 + 4SFP+) for my home lan. Cost me less than 300€ shipping included and is fan less.
If that was a good or bad choice is too early to tell but for the use I have had so far I'm happy with it. Doesn't do poe, but has a bigger sibling that does (but that comes with fans).

Around here the Zyxel XGS1930-28 is the cheapest option available with 4 SFP+ ports which is the main reason I choose it, plus I already have had some nice experience from other Zyxel gear.

and a link https://www.zyxel.com/products_serv...ith-4-SFP--Uplink-XGS1930-Series/introduction
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
While we're at the topic of Mikrotik:

The CRS305-1G-4S+IN coming out soonish looks very interesting for home users, too. It's a fanless

Prices are expected to be around 100 EUR resp. 120 USD
See if that thing came with a dual core CPU not sure why MikroTik is stuck on the 800Mhz Cpus! As a switch its great to get you feet wet in the 10GB/s realm, I get it allows them to make stuff cheap. I wish they would come out with ARM Socketits, I no that goes backwards in regards to ARM device architecture but the controller is pretty much that same along with the Chip set for most of the CPU Devices in its line. Would be nice If I could swap out from a 800Mhz single core for dual 800Mhz or a quad core 1.2Ghz One can dream!
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
See if that thing came with a dual core CPU not sure why MikroTik is stuck on the 800Mhz Cpus! As a switch its great to get you feet wet in the 10GB/s realm, I get it allows them to make stuff cheap.

So my understanding of this all is that the device will either operate with ports configured as attached to a switch, in which case it'll probably do fine as a switch and may be able to use the CPU to do routing or other tasks... not sure how feasibly... or ports connected to the CPU, at which point you are going to have some throughput/PPS issues. If you look at the Ubiquiti EdgeRouter Infinity (~$1500) they just barely manage to get to 18MPPS aggregate with their beefy 16 core 1.8GHz MIPS64 CPU. We really only passed the point where small CPU's such as the Cavium (used in, for ex., the Ubiquiti EdgeRouter Lite) were able to do credible routing of 1G at near-wire-speed (or actual wire speed in trite configurations) earlier this decade (~2014).

I'm excited that someone other than Ubiquiti is making an effort, and even if the thing works out such that it is just a decent little switch when running in layer 2 mode, that's significant all on its own. We can lament that it doesn't have better layer 3 capabilities, but the realities of the situation don't favor that, possibly not for another five years.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Our switch of choice at work uses a 64-bit MIPS Processor @1GHz clock, which is enough to provide 1280 Gbps switch bandwidth, 952 Mpps forwarding rate. This is a enterprise class solution with the associated price tag, but it goes to show that the processor isn't always doing all the heavy lifting. It would be interesting to pop the lid and see what some of their offload ASICs are, but I'm not going to chance voiding a warranty in doing so.

It is exciting to see 10Gbe prices beginning to drop in the pro-sumer market though.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Our switch of choice at work uses a 64-bit MIPS Processor @1GHz clock, which is enough to provide 1280 Gbps switch bandwidth, 952 Mpps forwarding rate. This is a enterprise class solution with the associated price tag, but it goes to show that the processor isn't always doing all the heavy lifting. It would be interesting to pop the lid and see what some of their offload ASICs are, but I'm not going to chance voiding a warranty in doing so.

It is exciting to see 10Gbe prices beginning to drop in the pro-sumer market though.

Well your switch of choice at work would continue to provide those packet forwarding rates even with a tenth of the CPU, because the processor isn't doing ANY of the heavy lifting. You are only using the CPU for doing the non-packet-forwarding features. Good switch silicon will not only handle packet forwarding, but will often do additional features such as layer 3 "switching" (i.e. more-or-less routing), packet filtering, etc., all at line (or near-line) rate, without involving a processor. The CPU then gets to handle "feature" stuff like STP, LACP, etc. Dedicated switching silicon should always be able to hit 20Gbps per port and 14.875Mpps per port. The only reason to put a beefier CPU in switches is because the incremental cost of doing so is pretty small, and you can run into tragedy in the field if your switch is suddenly suffering from an irreparable lack of CPU that's needed to remediate a problem. I remember the dear old Accton ES4624 ... hahaha.

Anyways the point here is that no currently practical amount of CPU will make a small switch into a workable line-rate 10G *ROUTER* but the possibility of small switches is definitely coming around... finally
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
I mean look at what cisco does with there SG line lets take the SG350X-24 I've seen up to 92.00 (Mpps) With 64-Byte Packets and well Switching Capacity (Gbps) it says it can handle 128 Gbps I have never gotin one to them speed's. As you would need a lot of stuff at home to push that limit.
I have seen both 10Gb/s SFP+ Ports hit 10 Gb/s and then the stacking ports also hit 8-9 Gb/s that's a whooping 29 Gb/s at the same time.

That's on a Dual Core 800Mhz ARM With 512MB of CPU memory and flash is at 256 MB. I think the age of doing more with less is a pound us. I know they say that it only use's 12 Mb of memory for a packet buffer not sure what the clock rate is on it, but if it can handle Gb/s speeds. They have differently mastered the trick of packing a 100 pounds of crap in to a 10 pound bag.
 

Andrew Ostrom

Explorer
Joined
Jul 28, 2017
Messages
57
Ok... After reading 21 pages of this thread I just want to make sure I'm not going to shoot myself in the foot, and I'm also looking for a recommendation on which switch to use. When I run CrystalDiskMark on my FreeNAS SMB share(s) I get Sequential Read (Q= 32,T= 1) : 110.878 MB/s and Sequential Write (Q= 32,T= 1) : 111.887 MB/s while neither my FreeNAS or PC is breaking a sweat, so I assume I'm network limited.

My configuration is simple - my Supermicro based FreeNASrver sits in the same rack as my primary switch in the basement "utility" room. My primary workstation/PC is in the room directly over the server room, and when I built the house in 2001 I installed extra conduit running downstairs, anticipating pulling fiber at some future point. (I worked in the network hardware/software business for many years). I have Comcast internet that usually yields about 300Mbps/50Mbps down/up. It comes into the same rack. I've set up 2 (soon to be 3?) ASUS routers using their AIMesh for WiFi everywhere though most of my devices other than phones and tablets are hardwired.

My goal is to connect my FreeeNAS and PC to the switch using 10GbE, I don't (currently) have any other 10GbE capable devices. I will buy 2 Chelsio NIC cards, they seem to be readily available on eBay for under $50 with the SR SFP+ transceivers included. I will buy a 1M and 10M OM3 fiber from fs.com.

My primary question is about the switch. I saw earlier that Chris Moore used an Aruba S2500-48T-4x10G switch which is still available on eBay for $129. I've also found a Cisco WS-C4948E 48 Port L3 Gigabit 4 x 10G SFP+ switch for $120 with a single power supply, $140 with dual supplies, and a Dell 5524 for $83. I would buy the appropriate SFP+ transcievers from fs.com.

Does this look like it should work? Which switch is the better solution? Since they already have the OS loaded I assume that I won't have to buy a license for IOS or whatever software Aruba or Dell use. Is that right? The last time I worked in the networks space was 2000, so all this gear is new since my time (we had OC-12 and OC-48 using SONET, and OC-192 was just starting, but that was for core switching only, and ungodly expensive). Thanks!
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,455
I can't speak to Arbua or Cisco at all. The Dell 5524 has no license or other recurring costs, and they're still releasing firmware updates. At least some of the Dells have lifetime warranties as well; I've been able to register both of the units I bought off eBay, and they're showing warranty out to 2040 or so.
 

Octopuss

Patron
Joined
Jan 4, 2019
Messages
461
Would anyone have any idea where could I find specific newer revisions of Chelsio cards? From what I've read thus far, the newer ones produce a lot less heat.
The cards don't seem to be in production anymore though, and I can't find them new anywhere near where I live in Europe.
I'm specifically looking for Chelsio T520-SO-CR, revisions 110-1188-50 or 110-1206-50.

I haven't seen any newer revisions listed on Ebay at all.
 
Top