Bad DD performance on all the pools of my system

[flotueur]

Hi everyone,

Here is a little bit of a Monday night challenge for you :)
I have a nice little NAS with the following parts:

• Motherboard A2SDi-4C-HLN4F (with an Intel Atom™ C3558 embedded)
• RAM 64GB RDIMM
• Drives:
  • OS - 1x Kingston 128GB SSD
  • Storage [Virtue] - 1x 4TB WD Red
  • Storage [Virtue] - 3x 2TB WD Red (I willr eplace the 2TB as they die by 4TB)
  • Storage Caching [Virtue] - 1x NVMe WD Blue (500GB) for the Storage Pool
  • Apps & VMs [Server] - 2x Samsung 850 pro 250GB

Initially i felt the system was a little slow (it is an old Freenas install i converted into TrueNAS scale).
But after setting up another TrueNAS scale on a different PC with junk harddrives and bad SSD, I realized i may have a problem: my junk PC is actually reporting faster read/write speed!
I am confused.. Here are the DD tests i performed:

Testing the "Server" pool (mirrored SSDs, 24% used):

Code:

truenas[~]# sudo dd if=/dev/zero of=/mnt/Server/appdata/test.dat bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 16.7773 s, 1.2 GB/s

Testing the home pool (single ssd):

Code:

truenas[~]# sudo dd if=/dev/zero of=/home/admin/test.dat bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 17.5856 s, 1.2 GB/s

Testing the "Storage" pool (4 disks RAIDZ1 77% used)

Code:

truenas[~]# sudo dd if=/dev/zero of=/mnt/Virtue/downloads/test.dat bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 16.7427 s, 1.3 GB/s
 

The other system with lower specs (3rd gen i3) would show 2/3+ GB/s - am i stuck on SATA 1?
Any idea of a setting i could change?

Further thoughts:

• "Virtue" is full at 77% so i would hqve expected performance drop only on that pool, not on the rest of the system;
• The MB does support SATA 6 - I am using both mini-SAS and SATA 3:0 configuration;
• I searched across the forum before posting - it seems that performance issues can have so many causes but I couldn't find the same case... That being said, i could read about compression and single disk performance. I wonder how this is connected... Because my pool "Server" shouldn't be affected by a sub-optimal configuration of "Virtue"?

Regards,
Florian

 

[sretalla]

The first problem you have is your testing is crap.

dd with if=/dev/zero is highly compressible, so you won't be testing anything real if you haven't first disabled compression on the dataset you're working with.

1.3 GB/sec is much faster than SATA3 speed (about 550MB/sec for most SSDs).

Depending on your dataset settings, you may also want to look at matching the blocks in dd to the recordsize (128K by default).

If you're really serious about testing properly, you'll look into using fio.

 

[Patrick M. Hausen]

The i3 has got a way higher clock frequency than the Atom so I don't really buy the "lower specs" assumption - at least for single core performance which is exactly what you are measuring. How fast can one core compress a stream of zeroes?

A speed of > 1GB/s would be incredible if you were really hitting storage. A typical NAS HDD (WD Red Pro 8 TB) has got a sustained write rate of "up to 235MB/s".

 

[flotueur]

Thanks both!

Quick comments:

• Understood DD is not optimal and roger the fact compression will be creating irrelevant results;
• I thought the output was in gigabits and not gigabytes;
• Looking at fio :)
• i3 3220 is indeed quicker 3.2ghz vs 2.2 ghz but it is dual core vs quad. My comment on lower spec was related to the fact the Atom based system is 64gb ram vs 16gb and using the more optimized Supermicro MB vs. the DELL Optiplex MB the i3 came with :)

I am looking at fio now - the initial results still show that the i3-based system to be quicker than the Atom based one.
I found a script that generates a nice table, and will post after reviewing the data.

I guess, for my understanding, what do you think should be my expectation for my Atom based system?

 

[flotueur]

Hi all!

So I did have a play with fio, with the objective of comparing my two systems.
It was my first time playing with it so please be patient with me if I made errors measuring!

The table below shows clearly that for each test, the i3 based operating system is faster (with the exception of the Single_1 MiB_random_write_process. Note I added the configuration parameters for reference but what really matters is to compare the pools for each test as opposed to figure out the best test.

For info, the i3 system is:

• Motherboard - Dell Optiplex 7010 with i3-3220
• RAM 16GB DDR3 (non ECC)
• Drives:
  • OS / Boot - 1x KingFast M.2 NVMe through PCIE adapter
  • Storage pool [Storage] -
    • 2x 2TB drive (Hitachi_HDS722020ALA330 and ST2000DM001-1CH164)
    • 1x 1TB drive (SAMSUNG_HD103UI)

Appart from the boot drive, all the other components are not optimised for NAS environment so I expected fluctuations of course but not like that.
What surprised me:

• The i3 system HDD are faster to the SSDs of the Atom based system;
• The SSD of the Atom system are often slower than the HDD of the same system

It is very random - even if the tests could be refined in a more scientific way, to me, they are directional to pointing towards a configuration issue (after all, i do not understand fully Truenas) or sub-optimal hardware but I really don't know where to continue my investigation!

What do you think I should do to improve the speed of the Atom based system?

Here are the results:

Test

NAS

Pool

Type

BW Avg MiB/s

BW Avg bw MB/s

IO MiB

IO MB

Run Low

Run Hi

BS

Size

Numjobs

16_parallel_128KiB_random_read_processes

Atom

Virtue

Read

256​

269​

11981​

12800​

42361​

46713​

128k

256m

16​

16_parallel_128KiB_random_read_processes

Atom

Server

Read

97,4​

102​

5235​

5489​

42423​

53743​

128k

256m

16​

16_parallel_128KiB_random_read_processes

Atom

Boot

Read

21,8​

22,8​

3967​

4160​

182214​

182268​

128k

256m

16​

16_parallel_128KiB_random_read_processes

i3

Storage

Read

118​

124​

5570​

5840​

37061​

47054​

128k

256m

16​

16_parallel_128KiB_random_read_processes

i3

Boot

Read

496​

520​

6858​

7191​

12463​

13839​

128k

256m

16​

16_parallel_64KiB_random_write_processes

Atom

Virtue

Write

118​

123​

6997​

7337​

59382​

59440​

64k

256m

16​

16_parallel_64KiB_random_write_processes

Atom

Server

Write

87,3​

91,5​

3563​

3736​

39633​

40814​

64k

256m

16​

16_parallel_64KiB_random_write_processes

Atom

Boot

Write

8,6​

8,8​

1656​

1736​

175122​

193133​

64k

256m

16​

16_parallel_64KiB_random_write_processes

i3

Storage

Write

364​

382​

43418​

46694​

119210​

119221​

64k

256m

16​

16_parallel_64KiB_random_write_processes

i3

Boot

Write

28,8​

30,2​

1101​

1154​

33844​

38162​

64k

256m

16​

Single_1 MiB_random_write_process

Atom

Virtue

Write

620​

650​

27034​

28979​

43583​

43583​

1m

1g

1​

Single_1 MiB_random_write_process

Atom

Server

Write

192​

202​

8930​

9364​

46440​

46440​

1m

1g

1​

Single_1 MiB_random_write_process

Atom

Boot

Write

199​

208​

15667​

16794​

79015​

79015​

1m

1g

1​

Single_1 MiB_random_write_process

i3

Storage

Write

294​

309​

26214​

28160​

89176​

89176​

1m

1g

1​

Single_1 MiB_random_write_process

i3

Boot

Write

294​

308​

12902​

13824​

43805​

43805​

1m

1g

1​

Single_128KiB_random_read_process

Atom

Virtue

Read

2054​

2154​

61645​

66150​

30001​

30001​

128k

256m

1​

Single_128KiB_random_read_process

Atom

Server

Read

2136​

2240​

21402​

22938​

10001​

10001​

128k

256m

1​

Single_128KiB_random_read_process

Atom

Boot

Read

2108​

2210​

21094​

22630​

10001​

10001​

128k

256m

1​

Single_128KiB_random_read_process

i3

Storage

Read

3623​

3799​

108544​

1E+05​

30001​

30001​

128k

256m

1​

Single_128KiB_random_read_process

i3

Boot

Read

2771​

2906​

27750​

29798​

10001​

10001​

128k

256m

1​

Single_4KiB_random_read_process

Atom

Virtue

Read

111​

116​

3321​

3482​

30001​

30001​

4k

256m

1​

Single_4KiB_random_read_process

Atom

Server

Read

111​

116​

1106​

1160​

10001​

10001​

4k

256m

1​

Single_4KiB_random_read_process

Atom

Boot

Read

117​

123​

1169​

1226​

10001​

10001​

4k

256m

1​

Single_4KiB_random_read_process

i3

Storage

Read

225​

236​

6746​

7074​

30001​

30001​

4k

256m

1​

Single_4KiB_random_read_process

i3

Boot

Read

158​

166​

1585​

1662​

10001​

10001​

4k

256m

1​

Single_4KiB_random_write_process

Atom

Virtue

Write

52,2​

54,7​

1616​

1695​

30965​

30965​

4k

256m

1​

Single_4KiB_random_write_process

Atom

Server

Write

80,8​

84,7​

829​

869​

10254​

10254​

4k

256m

1​

Single_4KiB_random_write_process

Atom

Boot

Write

24,8​

26​

823​

863​

33148​

33148​

4k

256m

1​

Single_4KiB_random_write_process

i3

Storage

Write

113​

118​

3517​

3688​

31194​

31194​

4k

256m

1​

Single_4KiB_random_write_process

i3

Boot

Write

169​

177​

1812​

1900​

10721​

10721​

4k

256m

1​

 

[MrGuvernment]

I guess my question is, what performance were you expecting from it?

Also,
Is this set up as cache for TrueNAS? if so, you do not need it, it wont do anything for you, especially with only 16GB of system ram. WD Blue drives also have no DRAM cache on them so they will run slower once used enough. Really everything you have in your system is low level quality, so do not expect super duper performance from it.

• Storage Caching [Virtue] - 1x NVMe WD Blue (500GB) for the Storage Pool

 

[sretalla]

OK, a couple of questions...

Why are you testing the boot pools? (they don't matter... and it looks like your Atom system is connected via USB2... not really a problem for performance of the system in general).

Are your WD RED drives RED Plus or old RED? if not, you may have SMR disks.

I agree with the post above that L2ARC probably isn't helping (you can just remove it and test again to prove that).

 

[PK1048]

You have not stated your use case. What are you using this for? That should drive your testing. ZFS is very hard to performance test due to lots of internal optimizations and features. These can lead to test results very different from real world performance.

• Does your use case include multiple clients or one client?
• What size files?
• Is the load read heavy or write heavy?
• Are the operations large sequential (media files) or small random (database)?
• Try to get your test case to match your real world use.

If you are just testing for academic (or the fun of it), that is OK too. In that case, disable as many complex features as you can, turn compression off, test with files at least 10x your ARC (or restrict the ARC to 0.1x your test data), for example. Then watch both `zpool iostat -v 1` and `iostat -x 1` during your test run. Look for latencies, bandwidth, and saturation (the %util column in the iostat -x output). If your disks (either HDD or SSD) are close to 100%util then memory and CPU will have less effect on results. Note that the `iostat -x` values will peak every 5 seconds on writes due to how ZFS caches writes, if you can turn on SYNC writes in your test suite that should smooth those out and get you better numbers.

Don't forget to account for disk controllers as well, a good SAS controller *may* outperform a basic SATA controller. You will see that in `iostat -x` numbers, especially if you use the same disks to test with on each system.

But, you are best to benchmark with the real data load You can still watch `zpool iostat -v 1` and `iostat -x 1` to get a feel for what is going on under the covers.

 

[flotueur]

Waow so many great answers! I am learning a lot here, thanks for that! I want to start saying that i appreciate the time each of you spent replying and I am going to do my best to integrate this in my response.

I have a lot to unpack here but you are right, I can always make my intentions clearer!

Use cases for each system:

• i3 - SOHO on the cheap - this is for my partner to have a cheap NAS with spare part lying around so that she can have storage and apps for her small business;
  • Users - up to 5 local
  • Workload - file server, VM for Virtual Assistants, up to 5 apps running and potentially blog hosting
• Atom - SOHO and homelab - this is for my household use and my extracurricular activities, I want to optimize the performance to enhance the user experience
  • Users - up to 5 local
  • Workload - file server, VM for experimentations (and home assistant), 15+ apps running and potentially blog hosting

Objectives:

• Build up the Atom machine as a great home lab and make it faster as TrueNAS and the apps feels running a bit sluggish;
• Build up the i3 machine as an entry level NAS that we can evolve as the company makes more profit.

Assumptions:

• The i3 system being made out of spare non-NAS optimized part should have overall worse performance;
• The Atom "Server" pool should outperform the other HDD based pools;
• A cache drive will help speed up the slower pool of HDD;
• 870EVO mirror to perform better than the single kingston SSD on the Atom system;

Goal of the post:

• Validate/invalidate assumptions
• Learning more about my home lab use cases and ZFS so that ultimately i can build better NAS
• Identifying performance sinks
• Because better at testing and diagnosing NAS related issues

Allright, now on to the responses:

I guess my question is, what performance were you expecting from it?

Just added info in the assumptions above but essentially the Atom based server should be faster than the i3.

Is this set up as cache for TrueNAS? if so, you do not need it, it wont do anything for you, especially with only 16GB of system ram. WD Blue drives also have no DRAM cache on them so they will run slower once used enough. Really everything you have in your system is low level quality, so do not expect super duper performance from it.

Yes - for my storage pool. Note that I have 64GB of RAM. Your answer made me realize I know too little about ZFS caching so I read about it and this is definitely not what I thought. In this case, I would agree with you that this may not be necessary as i rarely max out my ram now (althought the apps are taking a huge amount of it. here is a quick snapshot of my current usage for reference:

• Free: 19.4 GiB
• ZFS Cache: 31.2 GiB
• Services: 12.1 GiB

Even at that level of quality I was expecting more but I ignore a few key factors such as ZFS caching not working the way I thought and DRAM-less SSD. I am still puzzled by my SSD RAID which shoudl perform better (the SSD are decents).

Why are you testing the boot pools? (they don't matter... and it looks like your Atom system is connected via USB2... not really a problem for performance of the system in general).

Simply for reference! I wanted to check how well the drives are performing. It also helped me contextualized certain assumptions like, on the Atom system, I would expect the 870EVO mirror to perform better than the single kingston SSD.

Are your WD RED drives RED Plus or old RED? if not, you may have SMR disks.

Thanks! This one was a killer, one of them is SMR indeed (see below)! Since i understand that a pool will run as fast as its slower disk, that isn't helping my case :(

I agree with the post above that L2ARC probably isn't helping (you can just remove it and test again to prove that).

Yes, I am rethinking the whole thing now... It doesn't seem to make sense at this point of time but i want to review when it will.

You have not stated your use case. What are you using this for? That should drive your testing. ZFS is very hard to performance test due to lots of internal optimizations and features. These can lead to test results very different from real world performance.

• Does your use case include multiple clients or one client?
• What size files?
• Is the load read heavy or write heavy?
• Are the operations large sequential (media files) or small random (database)?
• Try to get your test case to match your real world use.

Thanks for that! You are right and I hope my little write up helps more. Good point on the test, I have very little experience on those so it is an opportunity to learn for me.

If you are just testing for academic (or the fun of it), that is OK too. In that case, disable as many complex features as you can, turn compression off, test with files at least 10x your ARC (or restrict the ARC to 0.1x your test data), for example. Then watch both `zpool iostat -v 1` and `iostat -x 1` during your test run. Look for latencies, bandwidth, and saturation (the %util column in the iostat -x output). If your disks (either HDD or SSD) are close to 100%util then memory and CPU will have less effect on results. Note that the `iostat -x` values will peak every 5 seconds on writes due to how ZFS caches writes, if you can turn on SYNC writes in your test suite that should smooth those out and get you better numbers.

Don't forget to account for disk controllers as well, a good SAS controller *may* outperform a basic SATA controller. You will see that in `iostat -x` numbers, especially if you use the same disks to test with on each system.

But, you are best to benchmark with the real data load You can still watch `zpool iostat -v 1` and `iostat -x 1` to get a feel for what is going on under the covers.

That is great guidance! I will definitely use that going forward.




With all that in mind, considering all the points above and some research about my drives I reached the following conclusions:

• My newest 4TB drive is SMR and may be the performance sink;
• The cache drive doesn't work as I thought it would and won't be useful unless i max out the RAM;
• If a cache becomes relevant as I expand my disk storage, select an NVMe with DRAM;
• Need to change my testing methodology!

Open questions/points:

• I still don't understand the relative poor performance of the samsung SSDs compared to other pools: is a mirror supposed to be that slow? The specs of the drive seem decent, am I missing something in their config?
• if I were to replace my HDDs in Atom, what brand would you recommend? My strategy was to expand the Virtue pool by slowly replacing the 2TB with 4 TB from WD but I need more space and the SMR 4TB is not optimal for my use cases.
• Even with suboptimal tests, in the spirit to compare at the same level, what could make the Atom "Storage"(Virtue) pool faster ?

Details about the drives:

System	Pool	Use	Drive	Capacity	Speed (RPM)	Type	SMR?	DRAM?
Atom	Virtue	Storage	WDC_WD20EFRX-68AX9N0	2TB	5400​	HDD	N	N/A
Atom	Virtue	Storage	WDC_WD20EFRX-68EUZN0	2TB	5400​	HDD	N	N/A
Atom	Virtue	Storage	WDC_WD20EFRX-68EUZN0	2TB	5400​	HDD	N	N/A
Atom	Virtue	Storage	WDC_WD40EFAX-68JH4N0	4TB	5400​	HDD	Y	N/A
Atom	Virtue	Cache	WD Blue SN570 500GB	500GB	N/A	NVMe	N/A	No
Atom	Server	Apps	Samsung_SSD_870_EVO_250GB	250GB	N/A	SSD	N/A	Yes
Atom	Server	Apps	Samsung_SSD_870_EVO_250GB	250GB	N/A	SSD	N/A	Yes
Atom	Boot	Boot	KINGSTON_SKC400S37128G	128G	N/A	SSD	N/A	Yes
i3	Boot	Boot	KingFast nvme 120G	120G	N/A	NVMe	N/A	No
i3	Storage	Storage	Hitachi_HDS722020ALA330	2TB	7200​	HDD	N?	N/A
i3	Storage	Storage	SAMSUNG_HD103UI	1TB	5400​	HDD	N	N/A
i3	Storage	Storage	ST2000DM001-1CH164	2TB	7200​	HDD	N	N/A

 

[MrGuvernment]

@flotueur okay, since this is for your partner, and hosting files for 5 people, and possibly a blog site. I just say FULL STOP!

Do not use this hardware for anything someone may have to rely on for work...

What is your backup / restore plan if this system crashes and burns?

Want to host a blog? pay for a cheap DigitalOcean wordpress hosting and be done with it, more secure than self hosting from a truenas system

Consider a dropbox/OneDrive subscription for file hosting / sharing between workers? Get a MS Office M365 sub for $100 or so for 5 people and be done with it?

 

[flotueur]

@flotueur okay, since this is for your partner, and hosting files for 5 people, and possibly a blog site. I just say FULL STOP!

Do not use this hardware for anything someone may have to rely on for work...

What is your backup / restore plan if this system crashes and burns?

Want to host a blog? pay for a cheap DigitalOcean wordpress hosting and be done with it, more secure than self hosting from a truenas system

Consider a dropbox/OneDrive subscription for file hosting / sharing between workers? Get a MS Office M365 sub for $100 or so for 5 people and be done with it?

Apologies, I could have added more here but I didn't want to confuse people with the i3 system and focus more on the Atom one!
So she has already everything online, the NAS would be a backup copy for now to enjoy some local performance gains and self host things like Wireguard, Nextcloud and PiHole. She is on OneDrive but complains it is slow and doesnt find cold storage practical to manage timely. Her site is also online but is not critical so she prefers self hosting temporarily to optimise expenses.
Our approach is to get something up and running quick and test out TrueNas with the i3 system. If she likes the convenience of the NAS then we would be looking into investing more, building something a bit similar to what I would like the Atom server to be (hence all the questions to make it good). Online+NAS+cold storage should bring us closer to a proper 3-2-1 backup strategy.

The atom system in comparison has a bit more resiliancy with mirroring and z1 plus cloud sync and auto copy critical data to another Nas.
Only the performance is a bit lackluster (especially compared to the junky i3 system), certainly due to poor choices as we know and are investigating

 

[flotueur]

and it looks like your Atom system is connected via USB2...

I failed responding to this: that is my problem exactly, this is an SSD with dram cache! Not the greatest I confess but it shouldn't give the impression it is usb2 based!

 

[sretalla]

My newest 4TB drive is SMR and may be the performance sink;

Agreed, and even if not, it's something that will eventually cause issues for that pool.

 

[flotueur]

From a HDD replacement, would you recommend going for WD Red Plus (I don't need pro) or Iron wolf? (ST4000VNZ06)
The spec sheets says they are fairly similar for the data points I need (especially, they are both CMR at the 4 TB capacity) but I lack the experience of understanding their respective reliability.

 

[sretalla]

The Red Plus drives are well understood around the forum and I don't see any bad reports about them.

 

[PK1048]

Yes - for my storage pool. Note that I have 64GB of RAM. Your answer made me realize I know too little about ZFS caching so I read about it and this is definitely not what I thought. In this case, I would agree with you that this may not be necessary as i rarely max out my ram now (althought the apps are taking a huge amount of it. here is a quick snapshot of my current usage for reference:

• Free: 19.4 GiB
• ZFS Cache: 31.2 GiB
• Services: 12.1 GiB

Those numbers look good for 5 users and general purpose file service. Having said that, you really want more RAM and ARC for VM storage.

Simply for reference! I wanted to check how well the drives are performing. It also helped me contextualized certain assumptions like, on the Atom system, I would expect the 870EVO mirror to perform better than the single kingston SSD.

The EVO870 is a consumer SSD with great large block sequential performance, but relatively poor small block random write performance. Assume any SSD that only provides sequential performance specs is a consumer grade and has poor small, random performance.

EVO870 spec: https://www.samsung.com/us/computin...870-evo-sata-2-5-ssd-1tb-mz-77e1t0b-am/#specs

Compare to the Solidigm (used to be Intel) DC-S4510 (yes, and old product, but I am using 4 of them) which is rated at 490 MB/sec sequential write and 18,000 I/Ops (4k block size) random write. https://www.solidigm.com/products/data-center/d3/s4510.html#form=2.5" 7mm&cap=480 GB

Because of the way ZFS handles writes, they are almost all random. There are exceptions, but assume for VM use all your writes will be random.

Open questions/points:

• I still don't understand the relative poor performance of the samsung SSDs compared to other pools: is a mirror supposed to be that slow? The specs of the drive seem decent, am I missing something in their config?

In general the speed of any zpool will be the speed of the slowest top level vdev * the number of top level vdevs. So if we assume 2-way mirrors and you use 2TB drives in a 2x2 configuration (2 vdevs each with 2 drives) you will get a certain performance. But if you used 4TB drives in a 1x2 configuration you would have the same capacity but only 1/2 the write bandwidth. If we go back to the 2TB drives but use a raidZ2 configuration of all 4 drives you end up with the same write performance as the 1x2 4TB drive mirror because you have only 1 top level vdev.
For read operations, the speed will largely depend on how much of it is coming from the ARC, and that will be limited by CPU speed (single threaded) and memory bandwidth. I say single threaded because a single read operation is one ZFS read thread.

• Details about the drives:

System	Pool	Use	Drive	Capacity	Speed (RPM)	Type	SMR?	DRAM?
Atom	Virtue	Storage	WDC_WD20EFRX-68AX9N0	2TB	5400​	HDD	N	N/A
Atom	Virtue	Storage	WDC_WD20EFRX-68EUZN0	2TB	5400​	HDD	N	N/A
Atom	Virtue	Storage	WDC_WD20EFRX-68EUZN0	2TB	5400​	HDD	N	N/A
Atom	Virtue	Storage	WDC_WD40EFAX-68JH4N0	4TB	5400​	HDD	Y	N/A
Atom	Virtue	Cache	WD Blue SN570 500GB	500GB	N/A	NVMe	N/A	No
Atom	Server	Apps	Samsung_SSD_870_EVO_250GB	250GB	N/A	SSD	N/A	Yes
Atom	Server	Apps	Samsung_SSD_870_EVO_250GB	250GB	N/A	SSD	N/A	Yes
Atom	Boot	Boot	KINGSTON_SKC400S37128G	128G	N/A	SSD	N/A	Yes
i3	Boot	Boot	KingFast nvme 120G	120G	N/A	NVMe	N/A	No
i3	Storage	Storage	Hitachi_HDS722020ALA330	2TB	7200​	HDD	N?	N/A
i3	Storage	Storage	SAMSUNG_HD103UI	1TB	5400​	HDD	N	N/A
i3	Storage	Storage	ST2000DM001-1CH164	2TB	7200​	HDD	N	N/A

Ugh .... 5400 RPM drives are generally a bad idea for performance.
Mixing drives of different performance in one zpool is a bad idea (mixing 5400 and 7200 RPM drives).
Mixing drives of different sizes in one zpool is a bad idea.
You can do all of the above, but it will reduce the performance, remember, the zpool is only as fast the slowest drive in a vdev * the number of top level vdevs.

See https://pk1048.com/zfs-resilver-observations/ for a blog post that shows the exact behavior for a raidZ2 pool. This was FreeBSD, but the underlying ZFS behavior is the same. Shortly after this I moved to a mirror configuration and have not looked back.

In terms of drive reccomendations, I have had very good results with Western Digital RED Pro and RE (now Gold). I generally avoid consumer drives and any drive with less than a 5 year warranty. I have had mixed results with Seagate, all of my ES-2 drives failed within warranty, some more than once, but Seagate replaced them all, many of them came back as ES-3. I have had not failures of the ES-3 drives (but they are a bit nosier).

The other trick I use for drives, is I try not to have all the drives in one zpool from the same make/model/batch. I might have 2 WD RED Pro and 2 Seagate ES-3, and even there I look for drives with different date codes. I have seen a manufacturer have a bad batch that all failed within weeks of each other. That does not help preserve your data.

In short, look for drives with a real 5 year warranty, no matter the make, those tend to be well built. Be careful of getting OEM pulled drives as those will not have a valid warranty. For example, a WD drive sold as an OEM to Dell is warrantied by DELL and not WD (and usually for only 1 year, even if the WD warranty would be 5 years).

 

[flotueur]

The Red Plus drives are well understood around the forum and I don't see any bad reports about them.

In terms of drive reccomendations, I have had very good results with Western Digital RED Pro and RE (now Gold). I generally avoid consumer drives and any drive with less than a 5 year warranty. I have had mixed results with Seagate, all of my ES-2 drives failed within warranty, some more than once, but Seagate replaced them all, many of them came back as ES-3. I have had not failures of the ES-3 drives (but they are a bit nosier).

The other trick I use for drives, is I try not to have all the drives in one zpool from the same make/model/batch. I might have 2 WD RED Pro and 2 Seagate ES-3, and even there I look for drives with different date codes. I have seen a manufacturer have a bad batch that all failed within weeks of each other. That does not help preserve your data.

In short, look for drives with a real 5 year warranty, no matter the make, those tend to be well built. Be careful of getting OEM pulled drives as those will not have a valid warranty. For example, a WD drive sold as an OEM to Dell is warrantied by DELL and not WD (and usually for only 1 year, even if the WD warranty would be 5 years).

Thanks both for those recommendations! A mix of drive makes sense and I feel i will go with that.

Those numbers look good for 5 users and general purpose file service. Having said that, you really want more RAM and ARC for VM storage.

Thanks! I will monitor this closely - since I only run 1 VM for now and a few apps, i rarely go over the numbers i pasted for now.

The EVO870 is a consumer SSD with great large block sequential performance, but relatively poor small block random write performance. Assume any SSD that only provides sequential performance specs is a consumer grade and has poor small, random performance.

EVO870 spec: https://www.samsung.com/us/computin...870-evo-sata-2-5-ssd-1tb-mz-77e1t0b-am/#specs

Compare to the Solidigm (used to be Intel) DC-S4510 (yes, and old product, but I am using 4 of them) which is rated at 490 MB/sec sequential write and 18,000 I/Ops (4k block size) random write. https://www.solidigm.com/products/data-center/d3/s4510.html#form=2.5" 7mm&cap=480 GB

Because of the way ZFS handles writes, they are almost all random. There are exceptions, but assume for VM use all your writes will be random.

That is really useful - thanks! That explains a lot why, abeit decent, the performance is not as good as I was expecting.

In general the speed of any zpool will be the speed of the slowest top level vdev * the number of top level vdevs. So if we assume 2-way mirrors and you use 2TB drives in a 2x2 configuration (2 vdevs each with 2 drives) you will get a certain performance. But if you used 4TB drives in a 1x2 configuration you would have the same capacity but only 1/2 the write bandwidth. If we go back to the 2TB drives but use a raidZ2 configuration of all 4 drives you end up with the same write performance as the 1x2 4TB drive mirror because you have only 1 top level vdev.
For read operations, the speed will largely depend on how much of it is coming from the ARC, and that will be limited by CPU speed (single threaded) and memory bandwidth. I say single threaded because a single read operation is one ZFS read thread.

So in this case, I configured the system to have 1 pool and 1 vdev for all the HDDs (1x4TB and 3x2TB) - Raid Z1 - , does your scenario still apply? What i have noticed is that off course i have half of the capacity of the 4TB.
Noted for the read, the Atom is one of the smallest i could get which i feel is a bad idea, I should have bought the 8C version at the time :(

Ugh .... 5400 RPM drives are generally a bad idea for performance.
Mixing drives of different performance in one zpool is a bad idea (mixing 5400 and 7200 RPM drives).
Mixing drives of different sizes in one zpool is a bad idea.
You can do all of the above, but it will reduce the performance, remember, the zpool is only as fast the slowest drive in a vdev * the number of top level vdevs.

Oh yes, for the i3 system indeed... I am addressing that but the system is mainly for test :)

 

[MrGuvernment]

5400 RPM drives can often be fine in a TrueNAS system when configured in a proper pool, run cooler and less power and you dont see a big a hit as you would using a 7200rpm vs 5400rpm in a desktop or laptop like back in the day...

 

[PK1048]

So in this case, I configured the system to have 1 pool and 1 vdev for all the HDDs (1x4TB and 3x2TB) - Raid Z1 - , does your scenario still apply? What i have noticed is that off course i have half of the capacity of the 4TB.

So you have one top level vdev consisting of a 4 disk raidZ1? In this case you will get one drive's worth of random write performance and you will have 1/2 of the 4TB drive unused (as you noted).

Including output from `zpool status` is the easy way to show us all your configuration. Here is my database pool consisting of 4 enterprise SSD in a 2x2 way mirror.

root@brandywine:~ # zpool status DB
pool: DB
state: ONLINE
scan: scrub repaired 0 in 0 days 00:00:03 with 0 errors on Mon Oct 30 23:09:07 2023
config:

NAME STATE READ WRITE CKSUM
DB ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
diskid/DISK-PHYG012301D8480BGN%20%20 ONLINE 0 0 0
diskid/DISK-PHYG012301X0480BGN%20%20 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
diskid/DISK-PHYG012301XE480BGN%20%20 ONLINE 0 0 0
diskid/DISK-PHYG012301XF480BGN%20%20 ONLINE 0 0 0
errors: No known data errors
root@brandywine:~ #

The zpool is named `DB`
It is has 2 top level vdevs 'mirror-0' and 'mirror-1'
Each of the vdevs is a 2-way mirror
I am due for another scrub, last one was Oct 30.
Note that this is from a FreeBSD 12.x system.

This zpool has 2x the random write performance of any individual SSD.

 

[flotueur]

5400 RPM drives can often be fine in a TrueNAS system when configured in a proper pool, run cooler and less power and you dont see a big a hit as you would using a 7200rpm vs 5400rpm in a desktop or laptop like back in the day...

I think you are right - although I am seeking performance, I have to be reasonable and for my use case, 5400rpm drive would be just fine and cost effective too.
As a result of our conversation, I have set my mind to:

• 2x WD Red Plus 4TB - WD40EFPX
• 2x Ironwolf 4TB - ST4000VNZ06

Looking at the specs they are fairly identical, same speed, CMR, etc. and as PK1048, a little mix/match helps mitigating surprises!
I am now pondering on the SSDs... Thanks for all this learning and I hope people reading this thread in the future will find this helpful!

So you have one top level vdev consisting of a 4 disk raidZ1? In this case you will get one drive's worth of random write performance and you will have 1/2 of the 4TB drive unused (as you noted).

Yes, and am I right saying this one drive's worth of write performance is going to be representative of the slowest member of the pool? Meaning, the SMR drive perfromance in my case?

Including output from `zpool status` is the easy way to show us all your configuration. Here is my database pool consisting of 4 enterprise SSD in a 2x2 way mirror.

Wonderful way to share! Apologies for not doing that earlier but here we go:

Code:

root@truenas[~]# zpool status       
  pool: Server
 state: ONLINE
  scan: scrub repaired 0B in 00:01:41 with 0 errors on Sun Jan 28 00:01:43 2024
config:

        NAME                                      STATE     READ WRITE CKSUM
        Server                                    ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            cad5550d-4ceb-4a1a-afb6-3b872f1f2598  ONLINE       0     0     0
            e9243970-3673-47bd-82fc-991bc9ff63e2  ONLINE       0     0     0

errors: No known data errors

  pool: Virtue
 state: ONLINE
  scan: scrub repaired 0B in 04:52:41 with 0 errors on Sun Jan  7 04:52:43 2024
config:

        NAME                                      STATE     READ WRITE CKSUM
        Virtue                                    ONLINE       0     0     0
          raidz1-0                                ONLINE       0     0     0
            e4b9645e-ac44-11ea-9247-3cecef002bf4  ONLINE       0     0     0
            7fda8cbb-2d6c-11ea-b782-3cecef002bf4  ONLINE       0     0     0
            3d185fe1-21f8-11ea-803b-3cecef002bf4  ONLINE       0     0     0
            c7a52272-2dc7-11ea-a126-3cecef002bf4  ONLINE       0     0     0
        cache
          65508c70-6447-4e3b-816e-1449bd1e5ed3    ONLINE       0     0     0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:56 with 0 errors on Sat Jan 27 03:45:58 2024
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sdf3      ONLINE       0     0     0

errors: No known data errors