How to use 1Gb network for some machines, 40Gb for others

[JustinClift]

Most 10GbE switches are SFP+ which will accommodate this connector. The fiber is only because that type of SFP+ is inserted.

Any idea if there's such a thing as an SFP+ to CX4 er... adapter, so the DAC breakout cable could fit into the back of something like this?

• http://www.ebay.co.uk/itm/152231176415

Just pointing that switch out, as I have two of the 24 port ones here and they've been rock solid (so far) with FreeNAS connected to one of the 10GbE ports on the back (via Mellanox ConnectX-1 card). :)

... also, extremely cheap on eBay. :D

 

[c32767a]

Any idea if there's such a thing as an SFP+ to CX4 er... adapter, so the DAC breakout cable could fit into the back of something like this?

• http://www.ebay.co.uk/itm/152231176415

Just pointing that switch out, as I have two of the 24 port ones here and they've been rock solid (so far) with FreeNAS connected to one of the 10GbE ports on the back (via Mellanox ConnectX-1 card). :)

... also, extremely cheap on eBay. :D

The QSFP breakout cable ends with a twinax SFP+ module.. There isn't enough of the right type of plumbing in the cable/connector to do anything but connect to a 10G connection. You might be able to get a SFP+ to 10G/1G BaseT media converter from someone like transition networks, but if it worked, it would be a hack.

A far simpler solution would be to do as others have suggested and use separate IP space on the 40 and 1G networks.
Then you just have to manipulate DNS so a DNS lookup from the 40G network returns the 40G IP and a DNS lookup on the 1G network returns the 1G IP of the Freenas. It's not too hard to pull off.. And depending on the file sharing protocol, might be completely hidden from the users anyway.

 

[friolator]

For now, we're going to keep the two separate. However, in looking through some of the release notes for the firmware updates for our switch, I found this in a version that came out last year:

"Added support for QSAX (QSFP to SFP+) 1G Transceiver"

So that sounds promising.

 

[MatthewSteinhoff]

I know you're dead set against doing it the right way (;)), @friolator, but...

You can get a refurbished Cisco SG300-10SFP for $190 (new: $350). That would allow you to make your entire network one big, happy family and eliminate any wonky behavior that could result from the subnetting you're suggesting. You're already comfortable with the Cisco SG300 family of products so this shouldn't be as worrisome as another brand's solution.

As an aside, the Cisco 'QSAX (QSFP to SFP+) 1G Transceiver' will likely cost about the same as the refurbished SG300-10SFP.

Cheers,
Matt

 

[depasseg]

DNS lookup from the 40G network returns the 40G IP and a DNS lookup on the 1G network returns the 1G IP of the Freenas

Does DNS exist of both networks? And since the 40G won't have internet access in the split config, devices that "migrate" to the 40G network will need to also have a 1G interface. So now how does your device know which interface and address to use for sharing. Like I said, this approach is doable, but I feel it will become a management headache. If this was set it up once and be done, then that would be one thing, but where you are going to be adding the 40G network to some machines, I think more thought of how exactly this will be accomplished is needed.

Or, spend $190 on a switch and be done with it.

 

[friolator]

"Right way," "wrong way," - is there really such a thing?

Right now the 40Gb network is not connected in any way to our larger gigabit network. I'd like for the gigabit machines to be able to access the stuff on the faster network in as seamless a way as is possible: both in terms of the end user and in terms of our maintenance of the setup. I'm willing to put in time to get the high performance network tuned so we can do our work, but I'd prefer to not have to think about networking all day, because that's not what we're here for.

So in my ideal scenario, machines that use 40GbE (and some that will use 10GbE on that same switch), will be removed from the 1Gb network as they're added to the faster network. From the end user (and my) perspective, this should be seamless: 40Gb machines should see the internet and SMB shares on individual 1Gb machines, as well as the network printers, without having to also have them directly attached to the 1Gb network.

I would like for the 40Gb machines to be able to address shared volumes on other machines by name, not IP. It seems to me the easiest way to do that will be to connect the two switches.

By "easiest" I mean both initial setup and ongoing maintenance. Am I wrong here?

 

[MatthewSteinhoff]

What I need is a 24+ port gigabit switch with a copper 10Gb uplink port that takes a passive DAC cable.

Have you looked at the Cisco SG500 series? We use both their 24-port and 48-port switches with FreeNAS. If you like the SG300 series, you'll be happy with the 500 series. Two or four 10G uplinks as well as 24 or 48 gigabit ports. Could swap your SG300 for an SG500 or just add an SG500. Price is good and the switch interface is darn near identical to the SG300 with which you're familiar. I can totally vouch for the SG500 line.

There are less expensive 10/1G switches but I know you're shy about other brands and are happy with the Cisco small business line.

Cheers,
Matt

 

[friolator]

You can get a refurbished Cisco SG300-10SFP for $190 (new: $350). That would allow you to make your entire network one big, happy family and eliminate any wonky behavior that could result from the subnetting you're suggesting.

How, though? Unless I'm mistaken, the SFP ports on this are the same mini-GBIC ports on our current SG300-28. They're 1Gbps, not 10. So how does that physically connect to the 40Gb network?

I haven't looked too much into the SG500 series, but it's an option. I was hoping there might be some inexpensive way to bridge the two switches, but it appears that's not the case.

 

[MatthewSteinhoff]

"machines that use 40GbE (and some that will use 10GbE on that same switch), will be removed from the 1Gb network as they're added to the faster network.

All machines with a 40G interface will need a 1G interface as well because the 1G subnet is the only side that has internet access, right? We've passed the point where a server can be effectively severed from the internet. How are you going to get your OS and application updates? You're either going to need a router between the two subnets or you're going to need a single subnet.

If you have a router sitting between the 1G and 40G networks, you're introducing overhead above and beyond what you'd have if there was but one subnet. Most routers should have no problem saturating a 1G pipe but you're adding latency and complexity.

The SG500X-24-K9-NA has two 10G SPF+ ports. The 48-port model has four 10G SPF+ ports. What specific 40G switch are you using? It certainly supports 10G, right?

Cheers,
Matt

 

[friolator]

The 40Gb switch is an IBM G8316. It has 16 40Gb ports, but it can use 40Gb->4x 10Gb breakout cables. Our plan is to put secondary machines that would benefit from a higher speed network, but don't require more than 10Gb, on those breakouts, adding them slowly to the network.

All machines with a 40G interface will need a 1G interface as well because the 1G subnet is the only side that has internet access, right?

Yes - this is the setup we have now.

If you have a router sitting between the 1G and 40G networks, you're introducing overhead above and beyond what you'd have if there was but one subnet.

Right - I guess this is what I'm asking: Is there something that's not a router that will allow connection between the 40Gb switch and our current 1Gb switch? If not, that's fine, we'll probably just upgrade the switch. But, we're talking about $800 vs something I had hoped would be a bit cheaper. I'd rather be spending the money on more drives, if I had a choice.

The SG500X-24 is probably what we'd get. We're currently using 22 of the 28 ports on our SG300, but if we only connect the 10 and 40Gb machines to the IBM switch, we free up ports on the gigabit switch, so I could go to less dense model.

 

[MatthewSteinhoff]

The refurbished Cisco SG300-10SFP solution is the least expensive at $250 all-in...

IBM G8316 -> 40G to 10G cable -> refurbished Cisco SG300-10SFP -> existing Cisco SG300 -> 1G clients

Everything lives in one subnet, no routing is required. Everything operates as fast as it is able. No squirrely subnet issues. No DNS issues. One interface per device.

If you want the same result (maybe a smidgen less latency) with one fewer point of failure and a more elegant - but $900 - solution...

IBM G8316 -> 40G to 10G cable -> Cisco SG500X-24 (replacing the existing SG300) -> 1G clients

Either of these dollar amounts - $250 or $900 - is more than worth the headache that comes when you start half-assing the network and end up with routing issues and start having to tell end-users how to map drives using IP addresses.

(Make sure to double-check with IBM on the 40G to 10G breakout cable. I've never had a problem with vendor-locked cables on the Cisco SG200/SG300/SG500 series but I don't know anything about IBM.)

Cheers,
Matt

 

[friolator]

Thanks - but unless I'm mis-reading the specs on the SG300-10SFP, aren't those the same 1Gb mini-GBIC ports we already have on our SG300-28? I don't see 10G ports on that machine.

 

[MatthewSteinhoff]

Aw, crap. I had the wrong model number. It's the Cisco SG350 that does 10gbe. My bad. Sorry to get your hopes up.

Cheers,
Matt

 

[friolator]

No problem. I just looked at the 350 series too, and they all seem to be 1Gb SFP as well, not 10. (I used the comparison tool on Cisco's site, and looked at all the models regardless of port count)

 

[Stux]

Ahem.

Netgear ;)

The switch I linked earlier does what you want. I can't speak to its reliability.

But it offers n gbaseT ports, 2 10gbaseT ports and 2 Sfp+ ports.

 

[friolator]

Got it, but again, I haven't had great luck with netgear hardware in the past. I'm willing to spend a bit for this if need be, but $800+ for the Cisco is a bigger pill to swallow.

 

[JustinClift]

We've passed the point where a server can be effectively severed from the internet. How are you going to get your OS and application updates?

Errrr no. Servers truly segregated from the internet don't generally need OS updates.

If there's no connection from the segregated network, and you're most likely not a target for state sponsored malware (eg usb stick / sound hopping / etc), then it's a reasonable approach.

 

[JustinClift]

Is there something that's not a router that will allow connection between the 40Gb switch and our current 1Gb switch?

How badly do you want to keep the cost down? If you have a spare PC around, it could be turned into a headless router fairly cheaply ($ wise, maybe not time).

 

[friolator]

How badly do you want to keep the cost down? If you have a spare PC around, it could be turned into a headless router fairly cheaply ($ wise, maybe not time).

I'd prefer to save the rack space if I can. The cost of getting a rack enclosure, building the PC, getting the NICs needed, might end up being more expensive than just swapping out the Cisco for an SG500X model...

 

[depasseg]

Got it, but again, I haven't had great luck with netgear hardware in the past. I'm willing to spend a bit for this if need be, but $800+ for the Cisco is a bigger pill to swallow.

Don't forget what @danb35 posted earlier -- Dell x1052 has 48 1GbE ports and 4 SFP+ (10GbE ports) for $629 new. There are other/older models on ebay for less than half that.