NAS for VFX production & File sharing

[timvanhelsdingen]

Hey guys, me again.
So sorry for the bunch of posts for the past week, but the responses have been very helpful. I've put together a new build based on the feedback and i'd like to hear your take on how it's looking now. It's more aimed at being at expandability, with 4 ram slots still left open and having a ton of empty pci-e slots which I could add SAS controllers on to add external drive bays later.

Budget didn't even come out a lot higher because of some savings in other parts (cheaper case, no SAS controller)
Reason I didn't go for second hand xeon parts is because i prefer to source from one vendor as that gives me more tax deductables, plus I didn't find many good single cpu motherboards in the older series.


What it will be used for:
I am a freelance visual effects artists. I do a lot of simulation work, which generates a ton of data (some overnight sims can generate hundreds of gigabytes)
High sequential read speeds are pretty import here, more important than fast random IO. If I do a water simulation it might load 1gb from disk per frame, so that's more of a sequential read then anything else.
Also I frequently have to send caches to clients / co-workers. I am currently using crushFTP for that but I am thinking of starting to use Nextcloud for that once I move all my data to my freenas build. I could possibly stick with crushftp if a VM would alow me to mount drives locally (don't think crush can share network storage) but nextcloud seems pretty great for my usecase. Especially as I could give others acces to my server as a dropbox replacement when doing team projects.

I will probably be using 1 or 2 VM's on it as licence server, repository and some other stuff, but nothing that requires very high compute.
(all the repository does is manage a database of ongoing renders and where the data lives, and it contains currently submitted jobs)

I will be connecting through 10gbe with one workstation, and additional 1 or 2 other system will be pulling or writing data for renders / simulations but not 24/7 (I assume I can also limit their speeds in Freenas so they don't pull full reads?..)


System I have planned right now:
CPU - Xeon Silver 4410
Motherboard - Supermicro X11SPi-TF
Memory - 4x Samsung 16gb RDIMMS (appear on QVL)
HDD - 8 x 6Tb Seagate Ironwolf
SSD (for VM, in mirrored vdev) - 2x Crucial 256gb SATA ssd
Case - Fractal Design Define R6
PSU - Corsair TX 550M v2

I will be booting from dual USB sticks. I know this isn't the fastest, but i'm not planning to really reboot this thing a lot (will just run 24/7)
The 8 HDD + 2x SSD will take up the 10 sata ports on the motherboard and re-introducing the SAS controller would add to the budget.

How it will be configured:
The 8 HDD will be put into mirrored striped vdevs as this is aimed at performance and i'm ok with the slightly less storage capacity of it. Still seems safe enough for redundancy as well (can lose 1 or 2 drives, and rebuild is a lot less taxing)

Don't have anything for SLOG or L2ARC right now, but those can be configured at a later time right?... Could always buy a intel optane if I feel like i'd need more performance on SLOG, or add a m.2 on the slot if i want L2ARC later.

Let me hear your thoughts!
Feeling pretty good at this current setup.

 

[Newfoundland.Republic]

My thoughts...

• Have you considered used? FreeNAS does not really need current hardware and you could save some money there to "invest" in other parts of the system.
• USB thumb drives - SSDs like the Kingston AS400s are about the same cost as a good USB thumb drive (about $20-25/each). While 120GB is large for a boot device, it would likely be more reliable.
• A host bus adaptor (a/k/a SAS controller) can be found on eBay for under $40. That would free up the onboard SATA ports for boot drives (see above) See M1015 on eBay. Make sure to flash it to IT mode.

 

[Herr_Merlin]

Hi,
I understand the reason why you go for a certain configuration. Booting from USB for example. I do the same. Heck the system isn't rebootet in a year and if so there there a maintenance window and I am happy with the slightly longer reboot.
Limiting the access speed is not so easy from my point of view. I would limit them by limiting the network connectivity e.g. 1G vs your workstation 10G.
Regarding skipping the SAS HBA, I would not do so. The onboard ports might not be connected equal to the chipset.

For your use case I don't see a L2ARC useful. The SLOG might be handy with spinning rust. But I would prefer to go for a SAS HBA first.

Additionally I can understand the reason to go for a new system regarding tax write offs.

 

[timvanhelsdingen]

My thoughts...

• Have you considered used? FreeNAS does not really need current hardware and you could save some money there to "invest" in other parts of the system.
• USB thumb drives - SSDs like the Kingston AS400s are about the same cost as a good USB thumb drive (about $20-25/each). While 120GB is large for a boot device, it would likely be more reliable.
• A host bus adaptor (a/k/a SAS controller) can be found on eBay for under $40. That would free up the onboard SATA ports for boot drives (see above) See M1015 on eBay. Make sure to flash it to IT mode.

Thanks for the reply!
I have considered used, but it comes out more expensive if I go that route. But as I pointed out I get an extra 28% tax writeoff If my total invoice is above 450 euro (excl tax) it comes out cheaper to source motherboard + cpu from same vendor, and I wasn't able to find a good board + cpu combo from a second hand vendor (needs to be a vendor, as I need an invoice for tax)
Plus most of the x10 boards with everything that I want on them (10gbe, IPMI) are dual cpu boards and getting a pair of second hand xeons will cost about the same as the current build. Rather build it new if the cost will be similar.

I do have a SATA pci-e controller (6 port) laying about so i could hook some bootdrive up to that, but does it actually make much of a difference?.. if im mirroring the boot devices and all they do is boot then it doesn't seem like that much of a benefit, i can always replace one of the usb if one fails since its super cheap. Or am I overlooking something here apart from the boot part?... Read Freenas runs in RAM.

 

[timvanhelsdingen]

Hi,
I understand the reason why you go for a certain configuration. Booting from USB for example. I do the same. Heck the system isn't rebootet in a year and if so there there a maintenance window and I am happy with the slightly longer reboot.
Limiting the access speed is not so easy from my point of view. I would limit them by limiting the network connectivity e.g. 1G vs your workstation 10G.
Regarding skipping the SAS HBA, I would not do so. The onboard ports might not be connected equal to the chipset.

For your use case I don't see a L2ARC useful. The SLOG might be handy with spinning rust. But I would prefer to go for a SAS HBA first.

Additionally I can understand the reason to go for a new system regarding tax write offs.

Thanks! Is there a benefit to still get a SAS controller if all the drives can fit on the motherboard through sata?.. Was planning to get a SAS controller later if I ever want to attach an additional box with more drives, but for the internal drives it didn't seem too useful because it also takes up a pci-e slot. Or am I missing something here? Performance should be similar?...

Good to know there isn't any management for speed on the freenas. I'm still looking into a new switch so I guess I could do it on the switch level if I get a managed one.

 

[Herr_Merlin]

Okay - skip that. I assume that the 10 SATA ports where connected to a different controllers on the board..
but as it turns out they are on the same:

On-Board Devices
Chipset	Intel® C622
SATA	Intel® C622 controller for 10 SATA3 (6 Gbps) ports; RAID 0,1,5,10

So keep it as designed. Don't forgot to turn off RAID for the C622.
I personally prefer a SAS HBA anyways with SAS disks. The bandwidth and performance is much superior

 

[Herr_Merlin]

btw. from where in Europe are you? As you can buy aboard and save the VAT in total and write off the rest and thus save a lot of money.
By doing so there are some huge refurbished server dealers where you could get an entire enterprise box with new HDD and even with 4h onsite business critical support.

 

[timvanhelsdingen]

btw. from where in Europe are you? As you can buy aboard and save the VAT in total and write off the rest and thus save a lot of money.
By doing so there are some huge refurbished server dealers where you could get an entire enterprise box with new HDD and even with 4h onsite business critical support.

I live in the Netherlands.
There is an additional 28% tax writeoff if you invest minimum of € 2301 in a year and a minimum of € 450 per invoice.
That is in addition to the VAT writeoff (21%) and writeoff on income tax (between 36% and 52% depending om bracket) so only end up paying about half the cost of the system after all the writeoffs are done.

I was looking into refurbished servers, but office i'm moving too in two weeks doesn't have a seperate server room (servers located on top of a platform in open space) so I rather not have a super loud server bothering everybody else working there hahaha. Else that would have probably been a good option.

Plus I generally like building systems as a hobby so I don't mind if it takes a bit of time to configure this.

 

[timvanhelsdingen]

Okay - skip that. I assume that the 10 SATA ports where connected to a different controllers on the board..
but as it turns out they are on the same:

On-Board Devices
Chipset	Intel® C622
SATA	Intel® C622 controller for 10 SATA3 (6 Gbps) ports; RAID 0,1,5,10

So keep it as designed. Don't forgot to turn off RAID for the C622.

I personally prefer a SAS HBA anyways with SAS disks. The bandwidth and performance is much superior

Oh thanks for pointing that out. Freenas wil experience issues if the raid controller isn't turned off?...
Will I see a difference in performance if i'd hook them up to a SAS controller? I guess SAS 3gb would be good enough for spinning rust right? Or guess I could also just get a 6g one.

 

[Herr_Merlin]

Go for 6G or 12G. 3G would be limited bandwidth wise for your large sequential read and write tasks.
I don't know how good the C622 performances on that but my experience with those controllers aren't the best in general regarding performance.
Remember that you are writing and reading to a huge amount of disks.

 

[timvanhelsdingen]

Go for 6G or 12G. 3G would be limited bandwidth wise for your large sequential read and write tasks.
I don't know how good the C622 performances on that but my experience with those controllers aren't the best in general regarding performance.
Remember that you are writing and reading to a huge amount of disks.

Thanks! will look into getting one.
On a sidenote, is it possible to move drives to different controllers for example if i'd try on the SATA but performance is slow move them to a SAS controller, or does freenas need them to remain connected to the same controller?...

 

[Herr_Merlin]

it should find the disk.. with a rescan and a dismount of the pool first.. but I would wait for the answer from other people here..

 

[timvanhelsdingen]

it should find the disk.. with a rescan and a dismount of the pool first.. but I would wait for the answer from other here..

cool. thanks. Sounds like perhaps getting SAS controller from the getgo might be safest option then though. Seems cheap enough to not risk it.