Hardware, config & testing questions

[fatmal]

Hi All, first time poster here. I am tasked with providing approx 100Tb of picture & video loop archive for Radiology & Cardiology systems. Commercial offerings, while robust, are expensive. Images won't get read often from this environment, so it doesn't need to be particularly fast - just have high availability.

I would like to get some advice on hardware for test & production rigs, and what I should be testing for.

Hardware

I am proposing to using 2 x Lian Li EX-503 cases (no smarts - just USB3 JBOD), with 10 x 1Tb drives, and 2 x desktop PCs with USB3 PCI cards installed.

Does anyone have any wild objections to this rig?

Config

Config needs to be flexible i.e. 1 x PC with both external cases attached to test spanning a zPool over 2 x raidsets, then each PC with 1 x external case attached to test keeping a 2nd copy of the data on a second NAS. Each external case will hold a zRaid2 or zRaid3 set.

Does anyone have any experience with setting up these kinds of test environments?

Testing & Learning

I figure the tests I need to run are (and learn as I go along);

• Learn the install process, including attaching and configuring multiple raidsets and zPools
• Test whether a VM or 'native' install works best
• Learn how to add/remove devices from the raidsets (won't be able to test this in 'production'!)
• Fill the disk to capacity (probably just use a script to rename a large file and write to the NAS)
• Pull disks - just to see what happens. Replace with a blank disk to rebuild the raidset
• Use of snapshots for backup / restore - across the two physical environments
• Test deduplication
• Test keeping multiple copies on one NAS - see if that 'mirrors' across to the second machine
• Performance tests - any suggestions on tools would be most helpful.

Is this a sensible list of items to test? Have I missed anything?

Production

My thinking around the production system is;

• 1u 2 x ssd, big RAM system as the 'head' unit
• At least 1 24-bay 'shelf' of disk, attached via USB3, with 3Tb drives. Probably configured as 4 raidsets (6 disks each)
• Additional shelves of disk as required - daisychained up to infinity!
• All of this duplicated in a second data centre, with either snapshot copies (15min) or mirror on write.

Does anyone know of anyone running a similar setup that I could talk with?

Sorry for so many questions, but I've got a lot to learn!

Mal

 

[Joshua Parker Ruehlig]

I don't know why you'd opt for usb3 when you could go sata? usb doesn't have Native Command Queuing like sata, and I don't even know if you get SMART (disk health monitoring) through usb. usb is a Universal protocol while sata is designed for mounted harddrives so I really recommend sata. also you might max out your usb buses (don't know bandwith of usb3.0, or if freebsd supports usb 3.0).

I really recommend you go with a nice sata card that has sata multipliers. this site has a list of sata controllers supported by freebsd and the bandwidth you could expect for the number of ports you use. http://blog.zorinaq.com/?e=10

Also, why that case? It looks nice, but if your gonna have 20 harddrives hanging around it, your asking for something to get disconnected. http://www.newegg.com/Product/Product.aspx?Item=N82E16811219038
That case has 24 hotswap, and you'd have air blowing over the harddrive (keeping them cooler).

---

https://signup.netflix.com/openconnect/hardware
If you look at netflix' setup, they have freebsd rigs with something like 100TBs. You could use the same sata card as them even, you could probably save over them if you don't use a xeon cpu and just get an i3 or celeron..

Goodluck with the systems, your assignment sound fun!

 

[fatmal]

Joshua,

Thanks for your informative response! I hadn't thought of going direct to SATA connections, as I was only thinking of building a test rig. Doing that would also provide me with a more realistic performance measurement. If I used one of those 24-bay chassis I suppose that I could repurpose it as the production box, and save the boss some money!

Also, thanks for the link to the netflix setup - lots of good info there to help point me in the right direction.

Mal