Box running - requesting some design advice

[boatymcboatface]

Hi everyone,

I have a FreeNAS box running on the latest release in a VM. Xeon 1275 processor, 36GB ECC memory assigned to FreeNAS (64GB total), 10 x 8 TB drives split across two M1015 controllers in two RaidZ vdevs (I crashplan all of it so I've accepted the RaidZ risk - I can afford to lose the data). I have VMWare ESXi with passthrough on both controllers.
All this sits on a Supermicro X11SSH-LN4F and it's working fine.

I have a desktop on which I do frequent photo editing. Indexing and editing large raw photos saturates a 1 gigabit connection very very quickly. I am trying to solve this problem and am contemplating a few scenarios, hoping someone can give me some tips.

The 10GB route part 1
Unfortunately due to the fact this motherboard has only two PCIe x 8 and one PCIe x4 slots, and the x8 slots being taken up by the M1015s, I only have one x4 slot left to use 10GB on. I would have to find a PCIe x4 based 10GB solution. Any advice on what I could use?

The 10GB route part 2 - controller
Alternatively I could ditch the M1015s and go with a single 16 port controller card, freeing up a PCIe x8 slot which might open up more (cheaper) options on the 10GB front.
A single slot HBA card might be nice anyway as I might get one of the Xeon-D boards in the future which only come with a single PCIe-x16 slot, but onboard 10 Gbit ports. I have an Areca 1280ML-24 card still lying around but deliberately picked up 2 x M1015 for the FreeNAS compatibility. Are there any 16 port cards as good/compatible as the M1015s?

The sync route
Alternatively, my desktop has a 1 TB SSD which is plenty fast for the photo editing. I could run some kind of sync between the two folders, so that I can edit locally but sync everything to the freenas box so it gets picked up by crashplan, and can be accessed from other machines too. I read a bit about Rsync, and about BTsync, any recommendations? Ideally I'd like two way sync, chance of conflicts is virtually non existent.

I'll admit this is not all pure problem solving, I get a kick out of building and messing with this stuff too so there is a distinct coolness factor as well :)

Many thanks for any tips and advice in advance!

 

[jgreco]

The 10GB route part 1
Unfortunately due to the fact this motherboard has only two PCIe x 8 and one PCIe x4 slots, and the x8 slots being taken up by the M1015s, I only have one x4 slot left to use 10GB on. I would have to find a PCIe x4 based 10GB solution. Any advice on what I could use?

Go get yourself a nice Intel X520 card, or a Chelsio T420-CR, put it in the server, and call it a day. You'll love it.

 

[jgreco]

https://forums.freenas.org/index.php?threads/10-gig-networking-primer.25749/

 

[boatymcboatface]

Thanks. I take it the fact it's an x8 card in a x4 slot doesn't matter then because you can't saturate it, right? Even if it has two ports?

Also, I looked at some of the other discussions, would you still recommend fiber over CAT6 ethernet now that that is gaining traction? (apart from the innate coolness factor fiber brings :D)
I think I have about 10 meters to cover and the whole house has CAT6a cabling everywhere currently making that potentially an easy upgrade path. I guess the Chelsio T420-CR SR can run 20+ meters though, correct?

Many thanks!

 

[jgreco]

Alternatively I could ditch the M1015s and go with a single 16 port controller card, freeing up a PCIe x8 slot which might open up more (cheaper) options on the 10GB front.
A single slot HBA card might be nice anyway as I might get one of the Xeon-D boards in the future which only come with a single PCIe-x16 slot, but onboard 10 Gbit ports.

By the way, this seems kinda odd. You're picking suboptimal hardware for the task. Get yourself a nice Xeon D board that has all this crap on it PLUS also has server-appropriate PCIe slots (x16 is really only useful for desktop/workstation).

https://forums.freenas.org/index.php?threads/supermicro-xeon-d-x10sdv-7tp4f.44237/

There ya go, the board I'm looking at for low power hypervisors. Add on a second 10G card to get a full set of four ports, and an LSI 9260 RAID controller to provide ESXi datastores, and you wind up with this totally mind-bending low power hypervisor.

 

[jgreco]

Thanks. I take it the fact it's an x8 card in a x4 slot doesn't matter then because you can't saturate it, right? Even if it has two ports?

Kind of a "who cares." So what happens if you saturate the PCIe? It still goes awesomely faster than 1Gbps ethernet. I mean, yeah, good, be aware of limitations. The X11SSH is all PCIe 3.0 so you're talking 500MB/sec per lane if you get a PCIe 2 card like the X520 or T420-CR. That's 2GBytes/sec in x4, so you wouldn't be able to saturate two 10G ethernets simultaneously, much less all four (two ports, two directions). So you can OCD about that and get a T520-CR with its PCIe 3 instead, or you can swap cards and put the slow HBA in the x4 slot on the theory that 8 drives probably can't pound 2GBytes/sec, or you can just not worry because it will be lots faster than gigabit and this is just a matter of "how much faster faster is."

Also, I looked at some of the other discussions, would you still recommend fiber over CAT6 ethernet now that that is gaining traction? (apart from the innate coolness factor fiber brings :D)
I think I have about 10 meters to cover and the whole house has CAT6a cabling everywhere currently making that potentially an easy upgrade path.

I would call it closer to transition. Honestly, back in 2013 when Netgear introduced their 10G copper "low cost" switches, I though that'd open the floodgates, but that was not the case, and the switches turned out to be jet engines. Netgear's released the latest model, the XS708T, which apparently idles at 15 watts and peaks maybe around 30. Unlike their previous offering, this is a full "smart" switch and can do vlans and all that, though probably thru the GUI. I could feel comfortable recommending this switch BUT I DON'T BECAUSE I'VE NOT TRIED IT. So this is only a "that looks kinda nice." Not a "go buy that."

The problem is that you will be paying a price premium for the 10G copper stuff. The wonderful thing about 10G SFP+ is that it is sufficiently old that people are actually retiring the gear, which means you can find good-to-great deals on fleabay.

However, my sense is that copper will win out... eventually. So if money isn't a big huge deal, go 10G copper if you don't mind being on the cutting edge.

I guess the Chelsio T420-CR SR can run 20+ meters though, correct?

If by "20+ meters" you mean "up to 300 meters," then, yes.

Many thanks!

Love the handle, by the way.

 

[boatymcboatface]

By the way, this seems kinda odd. You're picking suboptimal hardware for the task. Get yourself a nice Xeon D board that has all this crap on it PLUS also has server-appropriate PCIe slots (x16 is really only useful for desktop/workstation).

https://forums.freenas.org/index.php?threads/supermicro-xeon-d-x10sdv-7tp4f.44237/

There ya go, the board I'm looking at for low power hypervisors. Add on a second 10G card to get a full set of four ports, and an LSI 9260 RAID controller to provide ESXi datastores, and you wind up with this totally mind-bending low power hypervisor.

Agreed, that is nice stuff. The stuff I mentioned is what I already have though :)
I have a lot of Internet enabled crap around the house, cabled where I can, for which 1 gigabit is plenty. I have a 24 port Dlink switch that breaks up some provider VLans, and receives Internet from my Pfsense virtualized box. My Vms do nothing fancy, I just prefer to run separate arch Linux installs for things like crashplan, plex server etc rather than freenas jails - their data stores are on ssds and I don't care if they get hosed.
I got the xeon because in low power state it doesn't eat up that much, while it can quickly transcode for plex and other crap when needed.
I am still half thinking about running freenas on a dedicated board but honestly, it is doing quite well in a VM with passed through controllers.

 

[boatymcboatface]

Kind of a "who cares." So what happens if you saturate the PCIe? It still goes awesomely faster than 1Gbps ethernet. I mean, yeah, good, be aware of limitations. The X11SSH is all PCIe 3.0 so you're talking 500MB/sec per lane if you get a PCIe 2 card like the X520 or T420-CR. That's 2GBytes/sec in x4, so you wouldn't be able to saturate two 10G ethernets simultaneously, much less all four (two ports, two directions). So you can OCD about that and get a T520-CR with its PCIe 3 instead, or you can swap cards and put the slow HBA in the x4 slot on the theory that 8 drives probably can't pound 2GBytes/sec, or you can just not worry because it will be lots faster than gigabit and this is just a matter of "how much faster faster is."



I would call it closer to transition. Honestly, back in 2013 when Netgear introduced their 10G copper "low cost" switches, I though that'd open the floodgates, but that was not the case, and the switches turned out to be jet engines. Netgear's released the latest model, the XS708T, which apparently idles at 15 watts and peaks maybe around 30. Unlike their previous offering, this is a full "smart" switch and can do vlans and all that, though probably thru the GUI. I could feel comfortable recommending this switch BUT I DON'T BECAUSE I'VE NOT TRIED IT. So this is only a "that looks kinda nice." Not a "go buy that."

The problem is that you will be paying a price premium for the 10G copper stuff. The wonderful thing about 10G SFP+ is that it is sufficiently old that people are actually retiring the gear, which means you can find good-to-great deals on fleabay.

However, my sense is that copper will win out... eventually. So if money isn't a big huge deal, go 10G copper if you don't mind being on the cutting edge.



If by "20+ meters" you mean "up to 300 meters," then, yes.



Love the handle, by the way.

Thanks :)

I have just been offered a batch of enterprise grade cards for about 30$ each, I'll list them here if you don't mind me shamelessly picking your brain. There is a 520 in there but also some other stuff that looks decent.

I must say I appreciate the quick responses on this forum, plus it is refreshing to interact with people who have sufficient points in the mad scientist department to be able to ponder these things without going into a blank stare like, say, relatives or close friends ;)

 

[boatymcboatface]

Are any of these any good?

LPe12002-E Dual Fiber channel
Emulex LPE12000-E Single port Fiber channel
Chelsio communications Fiber channel (IBM MT1410 Chelsio SFP-FC PCIe Adapter 46M1811)

If not I can probably get my hands on a brand new X520-SR2 for a bit more.

Thanks!

bmcbf

 

[jgreco]

No, fiber channel isn't ethernet. It's similar in some ways, but not generally useful unless you already know why it's useful, sorry.

 

[boatymcboatface]

Bummer. Pardon my ignorance, I haven't dabbled with that stuff before and the learning curve is somewhat steep if you're not in hardware/infra anymore.

I will either pick up the 520 or go with a card supporting CAT6a depending on my finances... I'll let you know how I get on. Thanks for your advice sir!

Btw that supermicro board you linked seems quite rare, did some retail searches here and it's not even showing up at my regular supermicro vendor.

 

[jgreco]

Bummer. Pardon my ignorance, I haven't dabbled with that stuff before and the learning curve is somewhat steep if you're not in hardware/infra anymore.

No worries. Ignorance is correctable. There's a lot that can be learned from all the fine folks here in this community. One of the hardest things with technology is getting your foot in the door in the first place, when you're totally new to a given topic. We're happy to give people a shove in the right direction.

I will either pick up the 520 or go with a card supporting CAT6a depending on my finances... I'll let you know how I get on. Thanks for your advice sir!

Btw that supermicro board you linked seems quite rare, did some retail searches here and it's not even showing up at my regular supermicro vendor.

Yeah, it's newish. Overall the 1537 will beat an E3 Xeon but the per-core speeds aren't as hot. It looks like it'll make a good hypervisor for generic tasks.