Yorick
Wizard
- Joined
- Nov 4, 2018
- Messages
- 1,912
TL;DR: SAS drives are inexpensive on eBay
--
My 5-wide raidz2 was at 75% used, and rapidly getting to the point where a single 14TB disk wouldn't hold the data I cared about - everything with the exception of the PC backup dataset. I pondered moving it all to a 14TB external, then blowing away the pool and redoing it 8-wide.
That'd have been USD 260 for the external, and returning the external is a) a little iffy ethically and b) maybe not that easy once the packaging has been opened, depending on store policy.
Which is when I realized that I had already built my solution, near-as. I had set up a raidz expansion test system in my desktop, so could I expand that further and use it as a replication target so I could redo my main raidz2? Of course I can.
Assumption: There's a tower system, no matter how old, that has space for 8 drives. In some way shape or form. It'll be used as a temporary replication target. Depending on what its daily use is, it may also become a permanent backup target, or a test system. Or it can be torn down again after, to reduce the cost.
To stay under USD 260 and still have a working system - again, assumption is CPU/board/memory/PSU/case is already in place - I went for eBay SAS drives. That's the secret sauce, 2TB and 3TB SAS drives are being removed from server systems en masse and are quite affordable as a result.
HBA IT mode IBM M1015 or similar: USD 36
Intel NIC i210t: USD 23 (did not arrive in time, Realtek worked, but, probably a good idea just in case)
Cables SAS-to-SATA or SAS-to-SAS: Between 7.50 and 12, per cable; two needed. Let's call it 24 total on the high end.
Without drives: USD 83
Drives: 2TB SAS in 8-wide raidz1 gives me 11.8 TiB (just enough for me), 3TB SAS in 8-wide raidz1 would be 17.7 TiB
2TB goes for 17 to 20 bucks per drive, depending. I got six for 100 and then two more for 19 each. USD 138 total.
3TB goes for 24 to 26 bucks per drive, depending. Assuming 25 it's 200 for the drives. Slightly above the external, also more space.
As built with 2TB: USD 221, including shipping. Not bad.
4xhotswap in 3x5.25", aluminum, I found for USD 85. Totally optional and a luxury, but it worked well for my case and allowed a "cleaner" drive arrangement. I could also have removed the GPU for the period of transfer, and that would have allowed me to use the second internal drive cage and not taken that hit.
I may keep the 2TB drives for raidz expansion testing, still thinking about that. Or put them up on eBay again to recoup USD 100 or so.
Steps taken to make it all work:
- Flash HBA in EFI shell or DOS, this may not be necessary depending on how it arrives
- Cable the drives. If they are just mounted to the case, use SAS-to-SAS breakout (SFF-8087 to SFF-8482) with SATA power connectors; if they are mounted to a hot-swap, SAS-to-SATA breakout. Yes these hotswaps support SAS and SATA drives both, and yes you can access the SAS drive as a SAS drive over a SAS-to-SATA breakout cable that connects to the hotswap cage. It's all just connectors, the protocol doesn't get touched by anything but the drive and the HBA.
- Use a Linux boot to change SAS drives from 520 sectors to 512 sectors, was necessary on 6 of them, not on the other 2. Depends on how you get them.
- badblocks in Linux for all SAS drives to make sure they are okay. Some had a "grown defect list", some didn't. Good enough for my use case here. Could also have run badblocks in TrueNAS instead.
- Boot into TrueNAS, make a raidz1 pool, set up replication, and move data over to temporary system
- While all of the above happens, on source TrueNAS, badblocks the additional 3 drives.
- When replication is complete, verify that all data came over, verify again; verify a third time.
- Blow away the source raidz2, redo as 8-wide
- Replicate "the other way" - back to my main TrueNAS system
Between badblocks (a week), replication two ways (6 days - roughly another week), it took two weeks to do. It's not as neat as actual raidz expansion, but it got the job done, and I can do it now, before my data grows to the point where 2TB drives wouldn't have done the trick any more.
And I have a full-blown test system. That could become a backup system, but it's actually my daily driver desktop, so it won't become a backup system until I allow myself an SFF-PC upgrade sometime 2022 :).
Edited to add: This basic recipe could also be a main "starter" TrueNAS. Sure no ECC, but, where cost is the main factor, that's a choice people make anyway. Between 8 and 12 drives, as raidz2 or raidz3. If over 8 drives, an inexpensive SAS expander can be stuck into one of the PCIe slots, making it possible to address 12 (or more) drives easily. Any tower system with lots of 5.25" space should be able to handle 12 drives easily, by adding hot-swap cages as desired. Maybe even 15 or 16, depending.
--
My 5-wide raidz2 was at 75% used, and rapidly getting to the point where a single 14TB disk wouldn't hold the data I cared about - everything with the exception of the PC backup dataset. I pondered moving it all to a 14TB external, then blowing away the pool and redoing it 8-wide.
That'd have been USD 260 for the external, and returning the external is a) a little iffy ethically and b) maybe not that easy once the packaging has been opened, depending on store policy.
Which is when I realized that I had already built my solution, near-as. I had set up a raidz expansion test system in my desktop, so could I expand that further and use it as a replication target so I could redo my main raidz2? Of course I can.
Assumption: There's a tower system, no matter how old, that has space for 8 drives. In some way shape or form. It'll be used as a temporary replication target. Depending on what its daily use is, it may also become a permanent backup target, or a test system. Or it can be torn down again after, to reduce the cost.
To stay under USD 260 and still have a working system - again, assumption is CPU/board/memory/PSU/case is already in place - I went for eBay SAS drives. That's the secret sauce, 2TB and 3TB SAS drives are being removed from server systems en masse and are quite affordable as a result.
HBA IT mode IBM M1015 or similar: USD 36
Intel NIC i210t: USD 23 (did not arrive in time, Realtek worked, but, probably a good idea just in case)
Cables SAS-to-SATA or SAS-to-SAS: Between 7.50 and 12, per cable; two needed. Let's call it 24 total on the high end.
Without drives: USD 83
Drives: 2TB SAS in 8-wide raidz1 gives me 11.8 TiB (just enough for me), 3TB SAS in 8-wide raidz1 would be 17.7 TiB
2TB goes for 17 to 20 bucks per drive, depending. I got six for 100 and then two more for 19 each. USD 138 total.
3TB goes for 24 to 26 bucks per drive, depending. Assuming 25 it's 200 for the drives. Slightly above the external, also more space.
As built with 2TB: USD 221, including shipping. Not bad.
4xhotswap in 3x5.25", aluminum, I found for USD 85. Totally optional and a luxury, but it worked well for my case and allowed a "cleaner" drive arrangement. I could also have removed the GPU for the period of transfer, and that would have allowed me to use the second internal drive cage and not taken that hit.
I may keep the 2TB drives for raidz expansion testing, still thinking about that. Or put them up on eBay again to recoup USD 100 or so.
Steps taken to make it all work:
- Flash HBA in EFI shell or DOS, this may not be necessary depending on how it arrives
- Cable the drives. If they are just mounted to the case, use SAS-to-SAS breakout (SFF-8087 to SFF-8482) with SATA power connectors; if they are mounted to a hot-swap, SAS-to-SATA breakout. Yes these hotswaps support SAS and SATA drives both, and yes you can access the SAS drive as a SAS drive over a SAS-to-SATA breakout cable that connects to the hotswap cage. It's all just connectors, the protocol doesn't get touched by anything but the drive and the HBA.
- Use a Linux boot to change SAS drives from 520 sectors to 512 sectors, was necessary on 6 of them, not on the other 2. Depends on how you get them.
- badblocks in Linux for all SAS drives to make sure they are okay. Some had a "grown defect list", some didn't. Good enough for my use case here. Could also have run badblocks in TrueNAS instead.
- Boot into TrueNAS, make a raidz1 pool, set up replication, and move data over to temporary system
- While all of the above happens, on source TrueNAS, badblocks the additional 3 drives.
- When replication is complete, verify that all data came over, verify again; verify a third time.
- Blow away the source raidz2, redo as 8-wide
- Replicate "the other way" - back to my main TrueNAS system
Between badblocks (a week), replication two ways (6 days - roughly another week), it took two weeks to do. It's not as neat as actual raidz expansion, but it got the job done, and I can do it now, before my data grows to the point where 2TB drives wouldn't have done the trick any more.
And I have a full-blown test system. That could become a backup system, but it's actually my daily driver desktop, so it won't become a backup system until I allow myself an SFF-PC upgrade sometime 2022 :).
Edited to add: This basic recipe could also be a main "starter" TrueNAS. Sure no ECC, but, where cost is the main factor, that's a choice people make anyway. Between 8 and 12 drives, as raidz2 or raidz3. If over 8 drives, an inexpensive SAS expander can be stuck into one of the PCIe slots, making it possible to address 12 (or more) drives easily. Any tower system with lots of 5.25" space should be able to handle 12 drives easily, by adding hot-swap cages as desired. Maybe even 15 or 16, depending.
Last edited: