BUILD 32tb 10GBE build

[rplucker]

Hi everyone,

I'm planing on building a new system that accomplishes 3 goals:
1) Long term reliability (I still want to have this data in 10+ years)
2) 800MB/s+ transfer speeds
3) Use as a render farm/plex server (less important)

I work as a pro freelance video editor/animator. It's not uncommon for me to work on projects that are 300gb+ in size, and I need access to that data FAST. The speed I get from the drives in my editing rig just aren't cutting it anymore. I'm constantly sitting waiting for media to load into my timeline- it's wasting a lot of time. I want to build a system that will hold all of my editing files that I will work off of live, and serve as a long term archive solution for completed projects.
I've read every word of cyberjocks guide for noobs and hardware recommendations thread. (thanks, those helped a bunch!)

This is my plan:
Supermicro X9SRL-F
Mushkin enhanced proline 16gb ECC registered DDR3 1600mhz 992063 (3x for a total of 48g)
Intel E5 1620 @3.6 OR E5 2670 @2.6 (used for about $70)
Rosewell RSV L4412
8x HGST 4tb 7200rpm NAS drives (I already have 2 for a total of 10)
EVGA 210-GQ-0650-V1 650w
Intel X540-T2 (I'll use the same NIC on my editing machine)

I'll be putting the 10 drives in RAIDZ2 for 32tb total. (is this the best config?)
In a few years, I'm going to build a system with more CPU horsepower for VMs and rendering, transfer all of the data that's on this system to that system, then use this one for nightly backups and snapshots.

I have a few questions:
1) I already have 2 4tb drives. I'll be buying identical ones. I've heard that it's bad to buy them at different times, but is it worth spending another $3o0 on 2 more so I can buy them all at the same time or can I use the 2 I already have?
2) I'd really like to go with the E5-2670 (8 core) because it's about the same price as the E5 1620 (4 core), but I know that when you're going for max data transfer speed, higher cpu speed trumps more cores. Is 2.6 ghz enough to get 800MB/s or do I need to go with the 1620? Those extra 4 cores would be really nice for the windows VM I'm going to run as a render farm.
3) The ram listed is registered. Will this work on that board? I can't seem to find any documentation on that particular board as to if it supports registered ram- I'm probably missing something really obvious. I also can't seem to find any unregistered in that price point except for kingston, which isn't recommended.
4) If I also put a X540-T1 in my gaming PC, is it a good idea to put my steam and origin games on the nas to take advantage of faster loading times or is that just a dumb idea?
5) Is 48g of ram enough considering I'll be allocating 12-16g to a windows VM for rendering?

Please correct me if I'm using any terms wrongly or if I'm missing anything.

I know this is kind of a long post, thanks in advance for your help!

 

[Ericloewe]

I've heard that it's bad to buy them at different times

First time I've heard that one. Feel free to disregard that postulate.

2) 800mb/s+ transfer speeds

I assume you want more than 800 millibits (mb) per second. 800 megabits (Mb) per second sounds silly with 10GbE, too, so you mean you want 800 megabytes (MB) per second, meaning 6.4 gigabits (Gb) per second?

 

[rplucker]

I assume you want more than 800 millibits (mb) per second. 800 megabits (Mb) per second sounds silly with 10GbE, too, so you mean you want 800 megabytes (MB) per second, meaning 6.4 gigabits (Gb) per second?

Thanks for the reply. Yes, I do mean 800 megabytes per second.

 

[Ericloewe]

In that case, I don't have any numbers that might help you. The lower clock should be fine if you're dealing with multiple users, though (Samba spawns a thread per user).

3) The ram listed is registered. Will this work on that board? I can't seem to find any documentation on that particular board as to if it supports registered ram- I'm probably missing something really obvious. I also can't seem to find any unregistered in that price point except for kingston, which isn't recommended.

Theoretically, it'll work. In practice, it may not. That's what Supermicro's QVL is for.

4) If I also put a X540-T1 in my gaming PC, is it a good idea to put my steam and origin games on the nas to take advantage of faster loading times or is that just a dumb idea?

It'll probably be slower than a local SSD, despite 10GbE. And more expensive.

5) Is 48g of ram enough considering I'll be allocating 12-16g to a windows VM for rendering?

Try it out, if the ARC hit rate is low, add more RAM.

 

[rplucker]

@Ericloewe Thanks, I'm going to build this hopefully in a few weeks. I'll let you know.

 

[snaptec]

I think 8-10 drives in a single vdev won't give you 800+ mb/s.
Performance wise you schould use mirrored vdevs.





Gesendet von iPhone mit Tapatalk

 

[rplucker]

I think 8-10 drives in a single vdev won't give you 800+ mb/s.
Performance wise you schould use mirrored vdevs.

The main thing I need is fast read speeds. I'm okay with slower write speeds, but do you think I could get 800 read with a 10 drive VDev in raidZ2?

 

[hertzsae]

First, I don't see mention of a backup solution. I hope you're backing up offsite. RAID will give you uptime, but you should still have backups. I have some friends that like Crashplan, but there are plenty of other solutions. You're probably already taking care of this, but I didn't want to leave it unmentioned.

I work in enterprise storage. I run freenas at home. I'm not a freenas expert, but I know storage. Freenas could be different, but generally raid 6 (similar to raid z2) reads just as fast as mirrored raid. Mirrored raid has a huge advantage on writes, especially random writes.

The real question you need to answer is what is your current bottleneck? Is it bandwidth or latency? If the answer is bandwidth, then read no further, your proposed solution will likely work. If the answer is latency, then you may want to design something else. You mentioned reads being a primary factor, so I'd say that adding SSD to host an L2ARC for read cache may be the way to go. Alternatively, if you work on one project at a time, I'd put large SSDs in your workstation, work with the files locally and have nightly backups transferring to your NAS. No NAS will ever be as fast as a local storage solution when it comes to latency. Then you can save money by not building as powerful of a NAS, because you'll just be backing things up there.

Keep in mind that spinning drives have great sequential transfer rates and terrible random transfer rates (latency). You may be able to get 800 MBps out of a 10 drive raidZ2 assuming it's a large sequential transfer.

 

[jgreco]

First time I've heard that one. Feel free to disregard that postulate.

The "bad to buy different drives" thing is a fallacy advocated by people who've drunk delicious kool-aid provided by storage array providers.

https://forums.freenas.org/index.php?threads/purchasing-drives-in-batches.42229/#post-273159

 

[SweetAndLow]

my system with 2vdevs with 8 disks in raidz2 reads and writes at about 800-900MB/s with a streaming workload. With only a single vdev it reads and writes around 450MB/s. This is using WD red drives, if you jump to 7200 rpm drives you can get around 600MB/s read/write per vdev.

Also mirrors are going to read much faster because you can read from each drive independently. And it will also help with random IO.

 

[CookiesLikeWhoa]

You can also expand a mirrored pool much more easily because your vdevs are only going to be 2 to 4 disks each instead of 5-8 disks each.

 

[rplucker]

I ordered every thing for this build on the 29th (2 days ago) with a few small changes. I'll let you guys know how everything turns out.

 

[rplucker]

I got almost all of the components in the mail a few days ago and I've spent some time putting everything together and testing. I'm very happy with my new build! I haven't gotten my nics in yet so I'm still on 1gbe but most everything is working great. Thanks everyone for your help!

My only problem is that normal booting rarely works. Sometimes it gives me a "This is a nas data disk and cannot boot system. system halted" error even though it's booting off of my freenas flash drive as usual. Other times, it will stop at different points during the boot process.

Has anyone seen this before? This may belong to a different area of the forum, someone please let me know if I should move it.

Thanks!

 

[SweetAndLow]

If you get the data disk error that means your bios is not configured to boot from the USB stick correctly or maybe your USB stick is dead and doesn't work every time.

 

[rplucker]

I'm really happy. I know most benchmarks aren't completely accurate but this is really impressive. Thanks everyone!

 

[SweetAndLow]

I'm really happy. I know most benchmarks aren't completely accurate but this is really impressive. Thanks everyone!

Is that over cifs share? Something is wrong with your write speeds.

 

[rplucker]

Is that over cifs share? Something is wrong with your write speeds.

It is. I assumed that those speeds are normal considering write speeds generally take a hit on raidz2.

 

[SweetAndLow]

Test the local write speeds of your pool. You should be able to write at 400MB/s+.

 

[rplucker]

Test the local write speeds of your pool. You should be able to write at 400MB/s+.

What's the best way to do that? I copied and pasted a large file in the same directory on the share and I got about 170 MB/s

 

[SweetAndLow]

Make new test dataset with compression off then use dd if=/dev/zero of=/not/pool/datasetpath/testfile bs=1M count=20000

That should create a 20GB file and give you a good idea of pool speed top compare to other people.