Recovering Configuration After Hard Drive Swap

[William Bryan]

Thank you. I understand that. I inadvertently added an additional drive to my pool when I was trying to swap one out. It's what brought me here. I'm having to destroy my pool and start over. One could say this is the "nuclear option".

Thank you again,

Bill

 

[joeschmuck]

Bill,
We never discussed the hardware you are running and which hard drives you purchased. I'd like to understand your hardware configuration (do you have at least 8GB of ECC RAM, the motherboard you are using, the CPU in your system) to ensure you are doing it safely for your data and that you are not spending too much money for hard drives. Also tell me about any extra hard drives you have laying around.

You can add more drives a little more cost effectively by purchasing WD Green drives. Many people swear by them like Cyberjock. Newegg has them for $225 delivered to your door (using the promo code ESCATAN26 which expires on the 6th of June). I know it's more money than you wanted to spend but if you purchased 2 more drives, that would put you up to 6 drives total and get you up to ~22TB of storage.

 

[William Bryan]

Here is my hardware:

• 32 GB thumb drive for operating
• ASRock B75M-DGS 1155 Intel motherboard
• Pentium G860 LGA 1155 65W Dual-core processor
• 16 GB DDR3 SDRAM 1600
• 430W power supply - I suspect this is my weakest link....?
• 2 WD 3 TB drives
• 1 Seagate 3 TB drive - I originally bought two of each, trying to reduce the chance of all 4 going down together.....
• 1 HGST Deskstar NAS 3.5 6TB 7200RPM SATA III with 128MB cache (just installed to replace the Seagate that went bad)
• 4 more of the same 6TB drives (I just ordered those last night) I found them on Amazon for $280.
• I also ordered a SYBA control card with 4 more SATA ports because my motherboard only has 4.

I only have one extra drive laying around. It's only a 200 GB drive. It has Windows Vista installed on it and I used it once to recover data off my operating drive when Windows become corrupt due to a power loss. I wasn't putting everything on my server prior to then. A few things (like Microsoft Money) were writing to my local drive. Now it is also writing to the file server. It makes it nicer, because it can be updated from any of our computers.

One question that is more academic than anything else now (because I will have filled all the bays in my case for drives): When you delete the vdev and make a new one to add additional drives, do you lose your data?

I found a couple articles last night on RAIDz3. Not sure what I will do there yet.

 

[William Bryan]

I had enough to buy them, it's just I'm trying to save up for the first computer for the business:

CPU: Intel® Core™ i7-5960X Extreme Edition Eight-Core 3.00GHz 20MB Intel Smart Cache LGA2011-V3
HDD: 1TB Samsung 850 EVO Series SATA-III 6.0Gb/s SSD - 540MB/s Read & 520MB/s Write
HDD 2: 1TB Samsung 850 EVO Series SATA-III 6.0Gb/s SSD - 540MB/s Read & 520MB/s Write [+519] (1TB x 2 (2TB Capacity) Raid 0 Extreme Performance
MEMORY: 64GB (8GBx8) DDR4/2800MHz Quad Channel Memory [+1234] (Corsair Vengeance LPX
NVIDIA GeForce GTX TITAN X 12GB GDDR5 PCIe 3.0 x16 Video Card (Maxwell Dual Card (SLI))

I'm looking at right around $8,000 for this. It's a first step, interim computer for the main one I will eventually buy to process the scan data. This one will then be used to run AutoCAD and extract line work from the data.

Other more established companies are running a dual processor system with 128 GB of RAM. Their systems are costing more in the neighborhood of $15,000.

 

[cyberjock]

Here is my hardware:

• 32 GB thumb drive for operating
• ASRock B75M-DGS 1155 Intel motherboard
• Pentium G860 LGA 1155 65W Dual-core processor
• 16 GB DDR3 SDRAM 1600
• 430W power supply - I suspect this is my weakest link....?
• I also ordered a SYBA control card with 4 more SATA ports because my motherboard only has 4.

/faints

Just a word of warning.. that SYBA card has a very low chance of even working.

 

[William Bryan]

It should be in within the next couple days, I guess that will be the moment of truth. I read the reviews, and the 2nd one I saw said they were using it with FreeNAS, so I took the risk.

I guess the other hardware I have will handle this?

 

[Ericloewe]

I had enough to buy them, it's just I'm trying to save up for the first computer for the business:

CPU: Intel® Core™ i7-5960X Extreme Edition Eight-Core 3.00GHz 20MB Intel Smart Cache LGA2011-V3
HDD: 1TB Samsung 850 EVO Series SATA-III 6.0Gb/s SSD - 540MB/s Read & 520MB/s Write
HDD 2: 1TB Samsung 850 EVO Series SATA-III 6.0Gb/s SSD - 540MB/s Read & 520MB/s Write [+519] (1TB x 2 (2TB Capacity) Raid 0 Extreme Performance
MEMORY: 64GB (8GBx8) DDR4/2800MHz Quad Channel Memory [+1234] (Corsair Vengeance LPX
NVIDIA GeForce GTX TITAN X 12GB GDDR5 PCIe 3.0 x16 Video Card (Maxwell Dual Card (SLI))

I'm looking at right around $8,000 for this. It's a first step, interim computer for the main one I will eventually buy to process the scan data. This one will then be used to run AutoCAD and extract line work from the data.

Other more established companies are running a dual processor system with 128 GB of RAM. Their systems are costing more in the neighborhood of $15,000.

Unless you're specifically looking for cheap single-precision floating point performance, Titan Xs are a huge money sink. A system with a Xeon E5-1650 v3 and ECC RAM would also only be marginally more expensive...

 

[William Bryan]

What are your recommendations for a graphics card? I have applications that max out everything. An 8 core processor at times will be running 95-100% for all 8 cores, 40 seconds out of every minute. Almost 100% of the RAM will be used. There are huge reads and writes to the hard drive. I am displaying point clouds from a 3D laser scanner. The scanner can collect up to almost 1,000,000 points per second. The resolutions I run it's collecting around 500,000 points per second with a 10 minute scan. Multiply this times the number of scans. My largest is 53, but this is modest. It's easy to have 200 to 500 scans on industrial complexes, apartment buildings, etc. I need something that can display these and allow me to rotate the cloud without lagging.

 

[joeschmuck]

Here is my hardware:

• 32 GB thumb drive for operating
• ASRock B75M-DGS 1155 Intel motherboard
• Pentium G860 LGA 1155 65W Dual-core processor
• 16 GB DDR3 SDRAM 1600
• 430W power supply - I suspect this is my weakest link....?
• 2 WD 3 TB drives
• 1 Seagate 3 TB drive - I originally bought two of each, trying to reduce the chance of all 4 going down together.....
• 1 HGST Deskstar NAS 3.5 6TB 7200RPM SATA III with 128MB cache (just installed to replace the Seagate that went bad)
• 4 more of the same 6TB drives (I just ordered those last night) I found them on Amazon for $280.
• I also ordered a SYBA control card with 4 more SATA ports because my motherboard only has 4.

I only have one extra drive laying around. It's only a 200 GB drive. It has Windows Vista installed on it and I used it once to recover data off my operating drive when Windows become corrupt due to a power loss. I wasn't putting everything on my server prior to then. A few things (like Microsoft Money) were writing to my local drive. Now it is also writing to the file server. It makes it nicer, because it can be updated from any of our computers.

One question that is more academic than anything else now (because I will have filled all the bays in my case for drives): When you delete the vdev and make a new one to add additional drives, do you lose your data?

I found a couple articles last night on RAIDz3. Not sure what I will do there yet.

Is this your FreeNAS hardware?

I'm sorry to be the bearer of bad news but your system isn't capable of using ECC RAM which is a very big issue. If the above is the Motherboard, CPU , and RAM that you have installed to run your FreeNAS system, I'm going to recommend you do not use FreeNAS as your NAS software of choice. I certainly hope no one guided you down this hardware path with FreeNAS in mind.

The ZFS file system is highly robust for data integrity but part of what makes it a great product for data integrity is it's error checking. The downside is you must have reliable RAM and able to detect a "bit flip" during these error checks or if there was a bit flip, be it a one time thing or worse a stuck bit, all your data could be destroyed during an error checking process (a.k.a. Scrub).

So if this is the case, you can still establish your system however I would caution you to set aside some money over the next few months to purchase a new motherboard, CPU and RAM to complete your system. Once you have your new parts just shut down your current system, swap out the parts and boot it up, it should work flawlessly. Af course ensure that your BIOS has the ECC RAM enabled (might be by default but you should check).

I have some comments about the other parts you purchased, please take my comments as constructive feedback....

The 6TB hard drives, I would not have purchased those. I wouldn't say they have the best record for longevity. Also they are 7200 RPM so they will demand more power and produce more heat. In a simple NAS like you want to create a 5200 RPM drive is more than fine. It will let you saturate your 1Gb Ethernet port with ease. You can mix hard drives, so if you wanted to purchase another 6TB drive, I'd recommend the WD Red 6TB drive (that is because I like the Red line) but you could also buy a WD Green 6TB drive (those are good too). The difference between the Red and Green is for the most part a 1 year longer warranty on the Reds.

RAM, again because of your desired usage I'd recommend 16 GB of ECC RAM or at the most 32GB of ECC RAM (it needs to be compatible with your motherboard), cost is roughly $10 per GB.

CPU & MB, ask around, there are various options but in general a Supermicro motherboard and a decent Intel CPU. You are looking at around $400+ cost (sorry). Do not purchase these items without asking if the parts you want to purchase are compatible.

I wish I had better news for you.

As for recommendations for your main computer system for your Laser thing... I have no idea but I'd see what the software package recommends and maybe check out a forum where those folks hang out. I would definitely ensure that you know what the minimum system specs are and how fast you want things to be processed. As you indicated before, some companies spend up to $15K on a system like what you want to build but I don't see why that would be needed unless the graphics card is very expensive (some are). And many time when using a graphics application, the graphics card does most of the rendering so you do need to figure out what would be best for your software.

 

[joeschmuck]

Hey, maybe you could sell your son the CPU, RAM, and Motherboard and give him that failing hard drive too as a bonus. Maybe you could get a few bucks to help out purchasing the right components :)

 

[Ericloewe]

What are your recommendations for a graphics card? I have applications that max out everything. An 8 core processor at times will be running 95-100% for all 8 cores, 40 seconds out of every minute. Almost 100% of the RAM will be used. There are huge reads and writes to the hard drive. I am displaying point clouds from a 3D laser scanner. The scanner can collect up to almost 1,000,000 points per second. The resolutions I run it's collecting around 500,000 points per second with a 10 minute scan. Multiply this times the number of scans. My largest is 53, but this is modest. It's easy to have 200 to 500 scans on industrial complexes, apartment buildings, etc. I need something that can display these and allow me to rotate the cloud without lagging.

The question is "are you looking for single-precision or dual-precision?". Real-time stuff is typically done in single-precision, while final renders and similar processing is done in dual-precision (depends on the specific case, obviously). Even if it turns out you're ok with having a crapton of single-precision performance, a GTX 980 Ti is a few hundred bucks less and performs almost exactly the same - more than likely, more RAM, better disks or a faster CPU would be a better use of those ~700 bucks you'd save.

 

[William Bryan]

I'm assuming this is real-time rendering. I'm loading the point cloud in a program, viewing it, and extracting data out of it.

 

[joeschmuck]

Specifically which pieces of software are you going to run? If you are going to run AutoDesk/AutoCAD, the site has a recommended hardware tool. For instance I picked AutoDesk 2016 Premium and one of the selections was this NVIDIA Quadro 5000, but NewEgg sells the newer version of this Quatro K5200 for a mere $1839 (note that there is a $200 rebate right now too, not included in my price quote). You need to have a graphics card which is compatible with your software since you will have very specific requirements and cannot wing it or it will cost you more in the long run.

I'm just cautioning you to be careful in what you purchase since you are starting up a new business, you can't afford to make costly mistakes.

EDIT: If you want the cream of the crop, get this graphics card.

 

[William Bryan]

Compatibility. That is the pickle. Autodesk wants Quatro, the software for my scanner doesn't like Quadro, even though it's NVidia.

The problem I've found with minimum specs (especially for the scanner) is they work really good if I'm going out and doing 4 or 5 scans. (I've only done one of those since I've been scanning.) I can process those on my 4 year-old, AMD processor with 16 GB or RAM and a standard HDD. In real world scanning, the minimum specs lead to errors, crashes, and corrupted data.

 

[joeschmuck]

Yup, that is why you should visit some of the forums which support what you are trying to do so you can ensure yo make the best choice you can before dropping all that money.

As for FreeNAS, I still think you will need to drop about $500 minimum for the MB, CPU, and RAM (8GB Absolute Minimum and later add another 8GB stick if you find out you need it). I'll have to let you make your own decisions about how you want to proceed. And as for RAIDZ2 or RAIDZ3, it's only a number of how many drives can fail and you still retain access to your data. RAIDZ2 means 2 drives can fail, RAIDZ3 means 3 drives can fail. Now if one more drive fails then all your data is gone and unrecoverable. With larger drives like yours (6TB) it takes time to resilver (the process to recreate all the data on the new drive) all the data to the new drive. So during this process if you have another failure, well you can see how this works. If the data is extremely important like a crime scene for Austin, TX CSI, then you cannot afford to take any risks with the data.

Data risks brings up another point. I'm not sure how established you are just yet but if the location where your server is located burns to the ground, or a theft occurs, what type of off-site data storage will you have? These are things you should consider but they of course do not have to be addressed right now. This is how I think. If I had to keep a backup off-site, I'd place a second FreeNAS server at my sons house and use the high speed internet to synchronize my data to the off-site server. Many people in these forums do this so they could offer up some advice once you get to this point in your business.

 

[William Bryan]

Thanks again for your advice.

I plan on setting up the server so it emails me when there is a problem. With RAIDz2, once I start having a problem, I can replace the drive right away. Hopefully I won't have two more drives fail while it's resilvering.

I forked out the money for an upgrade. I purchased your motherboard, processor, 32 GB of RAM, and a new power supply (one with 12 SATA cables). I think I should have this covered now. It only cost me about $500. I talked with my son about the non ECC issue. He told me he upgraded his system about a year ago to ECC memory. Would have been nice to share, but he's busy with a life of his own. He spent about $700.

I've thought about the offsite back up. Problem is I'm on satellite internet. (Wait, I think I hear banjos.) I have a monthly data cap. I do need to figure something out, even it it's building another server and putting it in the outhouse, chicken coup, or pump house. (Don't really have an outhouse, but the pump house is dimensioned like one.) ;)

 

[joeschmuck]

RAIDZ2 should be fine, just make sure the email is working properly so you are aware.

As for offsite backups, because you are talking about so much data, if it looks like the raw data you collect per project is say less than 25GB, then you could purchase a Blu-Ray Burner and blank 25GB or 50GB media, copy off the original data (before processing) and send it to your sons house for safe keeping, or even a safety deposit box. I say only the original data because you could always rebuild whatever you are creating as long as you have the original scan data, but you must have that at a minimum. This is a less expensive option than a second server. And if you could get away using DVD-R media (4.7GB or 8GB), then you will save some money there. But always verify the media was created properly.

Take care.

-Mark

 

[William Bryan]

For that matter a thumb drive stored in my utility shed would work too. Sometimes the best solution is right there and not so high tech (as having a second file server). Too bad it's not always so easy to see.

 

[joeschmuck]

True, a thumb drive would work as well, just place it in a ziplock bag to keep the humidity down (you folks in TX are getting a bit of rain these days). To keep things actually safe, you could install a small fire proof safe as well. Eh, lots a person could do.

 

[William Bryan]

Hope things are going great. Other than still fighting the server issue, things are good here.

I was looking through the manual of my motherboard, and since I purchased the same one you have, I thought I would ask you what you have yours set to. It appears the EDD Mode is disabled, but either way there are five other choices: basic, good, super, max, user. Which one do you recommend?

Also, after I destroyed my old volume I checked how many disks FreeNAS recognized. It listed all 8. I had to manually create a new volume, because not all my drives are the same size. When it had finished one of my new drives was not listed. I'm thinking bad cable or bad drive? I'm going to have to wait until later to check the cable issue. (I did this with the old MB, CPU, and RAM and have taken it out of the box.)

My parts came in a couple weeks ago. I put it all together, pushed the power button, and nothing happened. Not even any fans turning. I pulled the power supply out and tested it. It failed the test, so I exchanged it for a new won. The new one arrived today and I tested it before continuing. This time it passed. However, after I put it all together and turned on the power button again.....nothing. The fans run, the hard drives make noise, but the BIOS won't load on my monitor. I'm thinking the power supply fried the motherboard, CPU, or memory. I'm leaning toward the MB. I ordered a speaker so I can at least hear how many beeps it makes....