Upgrade existing FreeNAS server

[Ixian]

I've had the following running for about 4 years now:

Supermicro X10SL7-F
Intel Xeon E3-1230v3
Crucial ECC DDR3 UDIMM (4x8) 32GB
(6) WD 5TB Red - RAIDZ2 (primary storage pool)
(2) WD 5TB Red - Mirrored (backup pool)
(1) Intel 500GB SSD (Jails, snapshotted to backup pool)
Housed in a Fractal Design R4 case with an Icy Dock 4-2 bay 5.25 bay expansion to fit more drives.

Currently using FreeNAS 11.1U6

Home file storage, media for the most part, Emby server, most it gets asked to do is transcode a lot of media (remote play for Emby users, and I have scripts that post-process recordings, plus there's thumbnail generation for Roku's going on, etc. - basically a lot of home-server ffmpeg/ffprobe duty). I run some other stuff like SABNZBD, a Nginx reverse proxy, and the like, fairly standard media server stuff.

Spent a lot of time on it back in the day, backed by plenty of advice here, and I'm back again. There's nothing wrong with the system - it is humming along quite nicely and has for years - but I need more storage.

I want to go with (8) WD Red 10TB drives using RAIDZ2. Two questions:

Do I need to up my memory? I know the rule of thumb is 1GB per TB of storage, but also it's a loose rule. My board tops out at 32GB so I'd need to replace it, and the CPU as well. I'm looking at:

Supermicro X11SSL-CF-0
Intel Xeon E3-1245v6
Crucial ECC DDR4 UDIMM (4x16)

However other than the obvious doubling of memory I don't know how much bang for the buck vs. what I have now I'd be getting. Do I need to do it anyway for an 80TB (60TB) RAIDZ2 pool?

Second question:

Ideally I'd hook the new drives up, stress test them with Spearfoot's excellent utility, then snapshot/send the data from the old pool to new. After checking, delete the old pool and figure out what I want to do with the old drives.

Except I'm going to be out of SATA ports; both the old board and potential new one max out at 14 and I'm using SATA-DOM for the FreeNAS boot device. Even if I temporarily disconnect the Jail SSD and the Backup pool drives I'm still short at least a port, I think. Both boards have 6 SATA plus two SFF 4x ports for a total of 14, and unless I am mistaken one of those ports is used for the DOM.

Any ideas on the best way to do this? Maybe I am overlooking something obvious.

Thanks!

 

[Ender117]

For your use case more RAM may not be required. I would think video stuff mostly will not benefit from more ARC, unless you replay a few episode frequently. You also have a SSD for jails. I would say upgrade the disks first, then decide if you need more RAM

My 0.02

 

[Ixian]

If it does turn out to be the case, would adding an L2ARC cache be a good step? Certainly cheaper than buying a new MB, ram, and CPU.

 

[Ender117]

If it does turn out to be the case, would adding an L2ARC cache be a good step? Certainly cheaper than buying a new MB, ram, and CPU.

Many recommend against using L2ARC (if that's what you're talking) before 64G RAM. But I am no expert in this so don't take my word for it

I think there are also 16G sticks for your boards out there, might be a better option

 

[Ixian]

The X10SL7-F only supports 32GB if I am not mistaken, though. 64GB means more ram, new MB, new CPU for my setup.

Sounds like I should just try it (the drive upgrade) and see if my read speeds go to hell, though?

Now to figure out the ports. If I disconnect everything but the existing 6 pool drives I can connect the 8 new ones; that means I'll need to temporarily swap out the Sata-dom I am using for boot with a USB drive. I assume I can use a "temp" FreeNAS install off USB to:

Create/check new zpool
Import existing zpool
Snapshot/send existing pool>new pool

Then remove old drives, hook everything back up, and re-import new pool in to FreeNAS? Will this work?

 

[Ender117]

The X10SL7-F only supports 32GB if I am not mistaken, though. 64GB means more ram, new MB, new CPU for my setup.

Sounds like I should just try it (the drive upgrade) and see if my read speeds go to hell, though?

Now to figure out the ports. If I disconnect everything but the existing 6 pool drives I can connect the 8 new ones; that means I'll need to temporarily swap out the Sata-dom I am using for boot with a USB drive. I assume I can use a "temp" FreeNAS install off USB to:

Create/check new zpool
Import existing zpool
Snapshot/send existing pool>new pool

Then remove old drives, hook everything back up, and re-import new pool in to FreeNAS? Will this work?

Ahh... I was thinking about DDR4. Well yeah looks like ddr3 16g is RECC only...

I cannot see any problem in the plan, but I would wait for someone more experienced than me to confirm ;)

 

[anmnz]

would adding an L2ARC cache be a good step

A read cache cannot possibly help you unless you are frequently reading the same data multiple times from disk. Your described use case doesn't sound like it's doing a lot of that. So it sounds like L2ARC would be a waste of time. Even if you had enough RAM already.

 

[Ixian]

On the same line, I'm guessing now that for my use case - where I have at most 2-3 people streaming a video off the server at the same time, and some background moves from my SSD when transcoding is done/etc. - 32GB is no real problem even with a 60TB (usable) pool?

 

[IQless]

Supermicro X11SSL-CF-0
Intel Xeon E3-1245v6
Crucial ECC DDR4 UDIMM (4x16)

If I understand the SM page correctly that MB does support IPMI, so you should not need the integrated graphics.
Since FreeNAS won't utilize the integrated graphics of the 1245v6, you might be better off getting either a 1240v6, but this all depends on the cost.

 

[Inxsible]

here's nothing wrong with the system - it is humming along quite nicely and has for years - but I need more storage.

Options:

1. Replace drives with larger capacity drives
2. Get a case that can support more drives -- like a 4U rackmount (for example) that can support 24 or even more drives.

Option 1 has a point of diminishing returns after a certain time

 

[Ixian]

For the expand, yes, probably the easiest way is to replace one drive at a time, wait for resliver, then rinse and repeat. I believe I have at least 1 open SATA port I can use for this. And it's safer since no loss of redundancy.

However I have 8 drives now and unless something has changed very recently I can't expand the pool with the 2 additional disks. Hence my original plan being, find a way to create a second new volume with the new disks and move everything to it from the old one.

I could get an expansion card and hook them up that way but then I'm back to buying things I won't need just to upgrade the volume.

 

[Inxsible]

However I have 8 drives now and unless something has changed very recently I can't expand the pool with the 2 additional disks.

You should have a backup of your data for that.

You could nuke everything and create a new pool configuration from scratch.

 

[Ixian]

I have a backup of my important data. Not the 20TB of media files I want to add storage to.

Think I have figured out the right plan, though.

 

[Ericloewe]

I think there are also 16G sticks for your boards out there, might be a better option

Nope. Support for 16 GB DIMMs was introduced with Skylake. That said, 32 GB is definitely worth trying out before spending tons of money on upgrades for the whole system.