My first server up and running

[Krackster]

ok first NAS Build, I am a totally a newby at the server stuff. I can build computers been doing it for 20 years. This server stuff is a foreign language. It works. Any advise or info would be greatly appreciated.
Current Apps: PLEX, ISCSI Steam Library

TRUENAS SCALE BUILD:

AMD Ryzen 7 PRO 4750G Processor
Asus AMD PRIME B550M-A AC Motherboard
128GB (4x32) ECC UDIMM DDR4-2666 PC4-21300 Memory
NICGIGA 2.5G Base-T PCIe Network Adapter
LSI 9201-16i 6G 16P SAS HBA P19 IT Mode ZFS FREENAS,UNRAID
10 x HGST HUS724040ALE640 4TB 3.5" SATA III 6Gb/s 7200RPM HDD (40TB RAID Z2 (HBA card))
2 x Intel SSDSC2BB120G6K 120GB 2.5" SATA 6Gbps SSD (Mirrored OS (motherboard))
Corsair AX760i Digital ATX Power Supply
Fractal Design Define 7 XL Black Brushed Aluminum Case

 

[Krackster]

ok first NAS Build, I am a totally a newby at the server stuff. I can build computers been doing it for 20 years. This server stuff is a foreign language. It works. Any advise or info would be greatly appreciated.
Current Apps: PLEX, ISCSI Steam Library

TRUENAS SCALE BUILD:

AMD Ryzen 7 PRO 4750G Processor
Asus AMD PRIME B550M-A AC Motherboard
128GB (4x32) ECC UDIMM DDR4-2666 PC4-21300 Memory
NICGIGA 2.5G Base-T PCIe Network Adapter
LSI 9201-16i 6G 16P SAS HBA P19 IT Mode ZFS FREENAS,UNRAID
10 x HGST HUS724040ALE640 4TB 3.5" SATA III 6Gb/s 7200RPM HDD (40TB RAID Z2 (HBA card))
2 x Intel SSDSC2BB120G6K 120GB 2.5" SATA 6Gbps SSD (Mirrored OS (motherboard))
Corsair AX760i Digital ATX Power Supply
Fractal Design Define 7 XL Black Brushed Aluminum Case

I also Have Comcast 1 Gig+ internet service
Average:
Download 1340 Mbps
Upload 40+ Mbps

 

[Arwen]

It is helpful for you to specify what your 10 x 4TB disks pool layout will be.

Some things like RAID-Z1 are not recommended with larger than say 1TB/2TB disks. And wider stripes, (>8), tend to start performing less optimally. Really, >12 disk stripes are pretty bad... but 10 might be okay.

Also iSCSI tends to want Mirrored vDevs instead of RAID-Zx.

Next, 2.5Gbps Ethernet cards can have spotty driver support in TrueNAS. It is changing, just warning you. Perhaps someone else can chime in on that specific card.

If you have not purchased yet, consider a real server board from one of the various manufacturers. They tend to be built to last longer, leave off the desktop fluff like audio, over-clocking and tons of USB.

 

[NugentS]

NICGIGA 2.5G Base-T PCIe Network Adapter is a Realtek chipset. Your mileage may vary as to how well it works
And just to echo @Arwen for block storage you need mirrors

 

[Krackster]

It is helpful for you to specify what your 10 x 4TB disks pool layout will be.

Some things like RAID-Z1 are not recommended with larger than say 1TB/2TB disks. And wider stripes, (>8), tend to start performing less optimally. Really, >12 disk stripes are pretty bad... but 10 might be okay.

Also iSCSI tends to want Mirrored vDevs instead of RAID-Zx.

Next, 2.5Gbps Ethernet cards can have spotty driver support in TrueNAS. It is changing, just warning you. Perhaps someone else can chime in on that specific card.

If you have not purchased yet, consider a real server board from one of the various manufacturers. They tend to be built to last longer, leave off the desktop fluff like audio, over-clocking and tons of USB.

currently as this is just a actual working homelab, the motherboard and cpu are really parts I had left over. the 2.5 nics was just a need due to the high speed internet to get all my band with. And I don't know anything about the Iscsi block thing. I can tell you where I am going with this I want a solid NAS for storage that above all else. PLEX is an experiment to see how it works. I am going with a Asrock rack x470d4u2-2t motherboard for my final build.

 

[Krackster]

currently as this is just a actual working homelab, the motherboard and cpu are really parts I had left over. the 2.5 nics was just a need due to the high speed internet to get all my band with. And I don't know anything about the Iscsi block thing. I can tell you where I am going with this I want a solid NAS for storage that above all else. PLEX is an experiment to see how it works. I am going with a Asrock rack x470d4u2-2t motherboard for my final build.

It is helpful for you to specify what your 10 x 4TB disks pool layout will be.

Some things like RAID-Z1 are not recommended with larger than say 1TB/2TB disks. And wider stripes, (>8), tend to start performing less optimally. Really, >12 disk stripes are pretty bad... but 10 might be okay.

Also iSCSI tends to want Mirrored vDevs instead of RAID-Zx.

Next, 2.5Gbps Ethernet cards can have spotty driver support in TrueNAS. It is changing, just warning you. Perhaps someone else can chime in on that specific card.

If you have not purchased yet, consider a real server board from one of the various manufacturers. They tend to be built to last longer, leave off the desktop fluff like audio, over-clocking and tons of USB.

By pool layout you mean how I have them configured. They are in a 10 drive Raidz2. Iscsi is up and running with my steam games so far everything seems to work ok.

 

[jgreco]

the 2.5 nics was just a need due to the high speed internet to get all my band with.

Then perhaps instead of a crappy Realtek solution, you should head on over to the 10 Gig Networking Primer and learn how to do it the right way.

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

And I don't know anything about the Iscsi block thing.

Then it would be prudent to learn before doing.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

 

[NugentS]

Have looked again at the OP - he wants iSCSI as a steam library I am assuming as a disk on his workstation, so used only by one client. IMO this doesn't need mirrors. RAIDZ and its performance is going to be fine, just as its fine for Plex.

The shitty 2.5Gb NIC is more of a concern as its almost certainly a piece of crap and is the OP's most likley performance failure point. The 10 Wide Z2 - conventional wisdom here seems to think up to 8 is good, up to 12 is OK but don't go beyond 12. I don't use RAIDz at all - so I haven't tried it

 

[jgreco]

IMO this doesn't need mirrors. RAIDZ and its performance is going to be fine

Performance isn't the only reason to use mirrors. RAIDZ used for block storage can end up using a shocking amount of additional overhead space, and a few other considerations listed in the block storage resource.

 

[Krackster]

ok, ok, you don't need to whip me with a tree branch.... the new motherboard coming has 2 intel 10gig nics on it. I also have an intel nic coming for my gaming rig.

Keep in mind i went from 1gig network to fully upgraded 2.5 gig network and now wanting a storage server upgrading to a 10 gig network. Nic's, Switches, cables.....my money only goes so far....

 

[jgreco]

Used 10G gear on eBay is generally cheaper than new 2.5G gear. Just sayin'.

 

[Krackster]

So if I LEARNED correctly, I should rework my homelab with mirrored drives. 10 4TB harddrives in a mirriored 20tb solution and 512gig ARC2 cache drive for ISCSI for steam

 

[NugentS]

Performance isn't the only reason to use mirrors. RAIDZ used for block storage can end up using a shocking amount of additional overhead space, and a few other considerations listed in the block storage resource.

That I don't know - I only use mirrors so its something I do not come up against. Maybe I should read your resource!!
- which I have now done
RAIDZ used for VM's block storage = bad.
RAIDZ used for a single client via iSCSI. Obviously I don't know without testing but I suspect would still work (just not as well as mirrors), just make sure you have plenty of space as the 50% overall utilisation, 50% ZVOL utilisation would probably still apply (so for 2TB of disk space you get 0.5TB of sensible use block storage). 10Gb will also help. Experimentation and testing is your friend here to see whats acceptable to you

 

[NugentS]

So if I LEARNED correctly, I should rework my homelab with mirrored drives. 10 4TB harddrives in a mirriored 20tb solution and 512gig ARC2 cache drive for ISCSI for steam

That should work very well

 

[jgreco]

RAIDZ used for a single client via iSCSI.

Will be moderate for performance. However, there is still a space allocation problem. Think about this: TrueNAS defaults to 4KB (ashift=12) sector sizes. So as an example: If you have an 8-bay chassis (fairly common), and put a RAIDZ3 in there, and then specify an often-recommended blocksize of 16KB for the zvol, your block will be written out as three sectors of parity, and up to five sectors of data-plus-pad (depending on compression). That is 32KB raw stored for 16KB of data (2:1 ratio). And if you lower that to 4KB blocksize, that zooms up to 4:1.

 

[Krackster]

Will be moderate for performance. However, there is still a space allocation problem. Think about this: TrueNAS defaults to 4KB (ashift=12) sector sizes. So as an example: If you have an 8-bay chassis (fairly common), and put a RAIDZ3 in there, and then specify an often-recommended blocksize of 16KB for the zvol, your block will be written out as three sectors of parity, and up to five sectors of data-plus-pad (depending on compression). That is 32KB raw stored for 16KB of data (2:1 ratio). And if you lower that to 4KB blocksize, that zooms up to 4:1.

So if I understand what (you told me to learn-haha) and what you are saying here is it raid z2 writes 3 times (1 time for the original block and 2 more blocks for the parity) high latency causes timeouts end user

mirrored just writes the single block duplicated simultaneously to 2 drives no extra blocks

 

[Etorix]

So if I LEARNED correctly, I should rework my homelab with mirrored drives. 10 4TB harddrives in a mirriored 20tb solution and 512gig ARC2 cache drive for ISCSI for steam

Or serve this Steam library through NFS or SMB (file storage: raidz# OK) rather than iSCSI (block storage: mirrors).

 

[Krackster]

Or serve this Steam library through NFS or SMB (file storage: raidz# OK) rather than iSCSI (block storage: mirrors).

the iscsi is on smb raid z2 pool currently

 

[NugentS]

Or serve this Steam library through NFS or SMB (file storage: raidz# OK) rather than iSCSI (block storage: mirrors).

Can you do that - I though Steam needed to be a "local" drive.
[Pause]
Ooo - thats changed - you can now use network drives. That might be a much better idea

 

[Krackster]

OK, so after taking some time to LEARN about building a server I can certainly see some of my errors. Although this current server is running without errors (doing what I am asking of it) I am going to upgrade it this Friday. This will be the new specs:

TRUENAS SCALE SERVER BUILD 2:
AMD Ryzen 7 PRO 4750G Processor
AsRock Rack X470D4U2-2T Micro ATX Server Motherboard**
128GB (4x32) ECC UDIMM DDR4-2666 PC4-21300 Memory
LSI 9201-16i 6G 16P SAS HBA P19 IT Mode ZFS FREENAS,UNRAID
10 x HGST HUS724040ALE640 4TB 3.5" SATA III 6Gb/s 7200RPM HDD - (20TB mirrored pool (HBA card))
6 x HGST Ultrastar HUS724030ALE640 3TB SATA III 3.5" - (15TB RAIDZ pool(HBA card))**
2 x Intel SSDSC2BB120G6K 120GB 2.5" SATA 6Gbps SSD - (120GB Mirrored OS (motherboard))
2 x TEAMGROUP MS30 512GB M.2 SATA3 - (Cache drive for each pool)**
Corsair AX760i Digital ATX Power Supply
Fractal Design Define 7 XL Black Brushed Aluminum Case


Dell X540-T2 Ethernet Adapter - 03DFV8, Dual Port, 10Gb install on Primary computer**

**-Denotes Changes