Citadel - Build Plan and Log

[ctag]

Cool, thanks for clearing that up for me :)

My dad is a world-class planner, and will often spend much longer figuring out exactly how a project will be accomplished than actually carrying it out. I envy that, but I'm still pretty impulsive and rash sometimes... Yesterday I saw that WD EasyStore 8TB external drives were selling for $160 at BestBuy. 4TB drives are $130, so for $30 extra I get twice the capacity, which sounds like a deal to me.

I bought four and shucked them. I'd never shucked drives before, but it was pretty straightforward, and I wound up with four red-label 256MB cache NAS drives.

So now I'm thinking about arrangements. Both Raidz2 and mirrors would apparently achieve 50% capacity, and I've read that mirrors in cases like this perform faster.

I'm also considering getting a few more drives while they're on sale and just having way more capacity than I know what to do with.

And I installed FreeNAS 11.1 on a virtual machine and am playing around with it. Right now it has 4x 8GB disks in Raidz2.

 

[LTCM]

Did you happen to do any research on doing burn in with the drives still in the enclosures? Can you even do that? That much work over a few days could create heat problems, maybe? But it certainly would be easier to warranty a drive still in the case.

Sweet deal and some good drives...congrats.

 

[ramar]

Thanks. I think I'm just going to go ahead and pick up that used server and see how it does, and save building my own as an exercise for next time.

did you customize the used server order? did you order the 6 core in the link (12 with 2nd CPU Riser)? it seems like it would be a good idea to get the second CPU if the price is low enough

 

[Evertb1]

So now I'm thinking about arrangements. Both Raidz2 and mirrors would apparently achieve 50% capacity, and I've read that mirrors in cases like this perform faster.

It will be actually (much) less then 50 % of the raw capacity. For a calculation see this url: https://jsfiddle.net/Biduleohm/paq5u7z5/1/embedded/result/. Especially with Raidz2, a pool of 4 drives is not very economically.

But take this as well in consideration: With Raidz2 you can loose any 2 disks in the pool without total data loss. With 2x2 disks in mirror, loosing a second disk in the same mirror means that all the data in that pool is gone.

 

[rivey]

Did you happen to do any research on doing burn in with the drives still in the enclosures? Can you even do that? That much work over a few days could create heat problems, maybe? But it certainly would be easier to warranty a drive still in the case.

Sweet deal and some good drives...congrats.

I also have 4 of these drives, all WD Red, and have shucked all 4 of them without any damage to the shells. I am now running badblocks on the second of the four. Takes about 5 days per drive on the temporary system I am using. No errors yet reported and since they can be reinstalled in the original cases, am not worried about the warranty. I think you should remove them and then run the tests. That way, no heat issues to worry about. Good luck.

 

[ctag]

Did you happen to do any research on doing burn in with the drives still in the enclosures? Can you even do that? That much work over a few days could create heat problems, maybe? But it certainly would be easier to warranty a drive still in the case.

Sweet deal and some good drives...congrats.

Some people were discussing that over on r/dataHoarder and it sounded like it would be worth avoiding any potential heat issues. And putting the drives back in an enclosure seems pretty straightforward.

As for actually doing the burn-in testing, I'm reading this post that mentions smartctl and badblocks.

did you customize the used server order? did you order the 6 core in the link (12 with 2nd CPU Riser)? it seems like it would be a good idea to get the second CPU if the price is low enough

I did not :-/ now that I'm thinking about it I really should have asked for more RAM too. And I didn't opt for the second CPU because the hardware guide said it probably wouldn't be worthwhile.

It will be actually (much) less then 50 % of the raw capacity. For a calculation see this url: https://jsfiddle.net/Biduleohm/paq5u7z5/1/embedded/result/.

Thanks for the link, it looks like I'd have around 12.6TB with Raidz2.

Especially with Raidz2, a pool of 4 drives is not very economically.

If I get two more of these drives while they're on sale, that would make a 6-disk Raidz2 with ~25TB of usable space. I'm considering it..

With Raidz2 you can loose any 2 disks in the pool without total data loss. With 2x2 disks in mirror, loosing a second disk in the same mirror means that all the data in that pool is gone.

The only reason I'm considering mirrors is because of this blog post about it. I'm definitely inclined toward 2 disks of redundancy before losing the pool is possible, but that article argues that it just isn't worthwhile. So I need to do some more reading and see what works best for people.

I also have 4 of these drives, all WD Red, and have shucked all 4 of them without any damage to the shells. I am now running badblocks on the second of the four. Takes about 5 days per drive on the temporary system I am using. No errors yet reported and since they can be reinstalled in the original cases, am not worried about the warranty. I think you should remove them and then run the tests. That way, no heat issues to worry about. Good luck.

Cool, I'm following along then ;)

 

[rivey]

Cool, I'm following along then ;)

As a clarification, due to the capabilities of my temporary FreeNAS system, I am only running the badblocks tests on one drive at a time. This is due to the small amount of ram and the old CPU that I am using. When I tried to run two drives at one I got errors on both drives. I then have run the badblocks on the first drive by itself with no errors and am currently running the test on the second drive with no errors and I am about 70% complete on this test. When this one is done, I will run the tests on the remaining two drives one at a time. Since I can't afford the parts I need for the new FreeNAS box yet this just seems to make sense and allows me to learn the software at the same time. By the way, my goal is to get two more of these drives and ultimately run them in a Z2 configuration. Way more storage than I need, but I would rather have more than I need than not have enough later on. Good Luck

 

[Evertb1]

The only reason I'm considering mirrors is because of this blog post about it. I'm definitely inclined toward 2 disks of redundancy before losing the pool is possible, but that article argues that it just isn't worthwhile. So I need to do some more reading and see what works best for people.

I am very familiar with that blog. This has been subject to discussion more then once. And that blog is the reason that I started out with mirrors with my first FreeNAS build. A little more then a year ago I decided to reconfigure my storage and go for RAIDZ2. This based on a better economy of my storage and a (for me) better form of redundancy. But you are right if you feel that you need to read up on this subject some more to come to a for you best solution.

 

[ctag]

When I tried to run two drives at one I got errors on both drives.

That's kinda scary.. I'm not sure which computer I'm going to use for the drive testing yet, but I'll keep this in mind.

By the way, my goal is to get two more of these drives and ultimately run them in a Z2 configuration. Way more storage than I need, but I would rather have more than I need than not have enough later on.

Sounds like a plan. Good luck!

 

[Chris Moore]

The only reason I'm considering mirrors is because of this blog post about it. I'm definitely inclined toward 2 disks of redundancy before losing the pool is possible, but that article argues that it just isn't worthwhile. So I need to do some more reading and see what works best for people.

I would not use a pool of mirrors to store my data. It only provides one drive of redundancy and I have had two drives fail at the same time. If you don't have a specific need for speed, and most people don't, RAID-z2 is the way to go and people that are particularly cautious are using RAID-z3.
You will still have more than enough performance for a 1GB network and if you moved to a 10GB network, you would still see performance in the area of 300MB/s.
The author has a specific use case in mind, and for him mirrors are the way to go. Everyone has an opinion, it doesn't mean it is right for you. You have to think about what you are storing. Is it a database or a website that is going to need heavy random IO?

 

[Evertb1]

The author has a specific use case in mind, and for him mirrors are the way to go. Everyone has an opinion, it doesn't mean it is right for you. You have to think about what you are storing. Is it a database or a website that is going to need heavy random IO?

I have read the blog numerous times and it didn't strike me as targetting a specific use case. Though soho and hobby use are mentioned. The writer is just going to bat for the use of mirrors, period. At least that is how I read it. And he has some strong/valid arguments. Luckily he also stresses the importance of backups. Because, no matter what configuration you choose for your storage pool, you need backups if you want to keep your data safe.

After living a couple of years with FreeNAS I am happy with my current Raid-Z2 pool. To me and for my use case, it is a nice balance between a reasonable amount of redundancy (keeping the data available for use), utilizing the available raw storage space and performance of the pool. But I need to confess that I never had the need (knock on wood) to replace a failed disk. No doubt that will be a pain in the b**. I have done it with a mirrored set and that was a breeze.

 

[Chris Moore]

I began my journey with FreeNAS using retired drives purchased from eBay. Since the beginning in 2011, I have replaced every drive and then replaced the replacement drives. It isn't that bad as long as you do it right. Also, it is a good idea to have a spare drive on hand for when you need it. I keep a couple of each size.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[ctag]

I would not use a pool of mirrors to store my data. It only provides one drive of redundancy and I have had two drives fail at the same time. If you don't have a specific need for speed, and most people don't, RAID-z2 is the way to go and people that are particularly cautious are using RAID-z3.
You will still have more than enough performance for a 1GB network and if you moved to a 10GB network, you would still see performance in the area of 300MB/s.
The author has a specific use case in mind, and for him mirrors are the way to go. Everyone has an opinion, it doesn't mean it is right for you. You have to think about what you are storing. Is it a database or a website that is going to need heavy random IO?

Cool. I would like to run some lightweight jails or VMs on this machine, but primarily just backups and media archival, so the safety of Raidz2 is appealing.

I've started the long SMART test on two of the drives. Should complete by tomorrow evening. I'm going to try and set up SMART temperature monitoring on the host computer tonight. One of the drives is already at 34C, and it's only been plugged in a few minutes...

 

[Chris Moore]

I've started the long SMART test on two of the drives. Should complete by tomorrow evening. I'm going to try and set up SMART temperature monitoring on the host computer tonight. One of the drives is already at 34C, and it's only been plugged in a few minutes...

Here is a batch of scripts that help with the monitoring of the NAS:
Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

All the drives need good airflow to keep them cool. That is the hard part of picking a good chassis. What do you have them mounted in?

 

[ctag]

I just have them laying out on a desk :-/

I've got a small office fan pointed at them from two feet back, and the temperatures seem to be decreasing a bit and holding.

 

[Chris Moore]

I just have them laying out on a desk :-/

I've got a small office fan pointed at them from two feet back, and the temperatures seem to be decreasing a bit and holding.

More airflow is better.
I feel that ideal temperatures are in the range between 28°C and 38°C, but there is no real industry standard because some brands of drives run hotter than others. If I recall correctly, the WD Red drives are rated to operate at temperatures between 0°C and 70°C, but pushing the limit can't be good for long life.

Correction. The top temp should be 65°C...
https://www.wdc.com/content/dam/wdc/website/downloadable_assets/eng/spec_data_sheet/2879-800002.pdf

 

[ramar]

i thought that when running ZFS, it was highly recommend (on this site?) that you do not run RAID, i'll have to reread the info

 

[Evertb1]

i thought that when running ZFS, it was highly recommend (on this site?) that you do not run RAID, i'll have to reread the info

RAIDZ(x) is NOT the same as RAID. RAIDZ is controlled by ZFS. See the ZFS primer in the FreeNAS manual http://doc.freenas.org/11/zfsprimer.html

 

[ramar]

Thanks Evertb1, that is exactly what i was confused about. RAIDZ2 seems to be generally a good balance between storage efficiency and data safety. I would be using it for music production. I might be tempted to go one safer. I like the idea of having extra storage bays for replacing drives.

 

[danb35]

it was highly recommend (on this site?) that you do not run RAID

It is highly recommended that you not use a hardware RAID controller. But some form of RAID (RAIDZn or mirrors) is strongly encouraged.