Neville005
Dabbler
- Joined
- Sep 18, 2021
- Messages
- 10
Ok, let me be honest: I'm at a loss here and could really use some help. As such, I greatly appreciate any and all feedback and/or suggestions!
The problem
I'm looking to improve my small file SMB performance, but can't seem to find the current bottleneck. I'm pretty new to TrueNAS and ZFS in general, so I wouldn't be surprised if I'm just missing a basic setting somewhere or maybe just have wrong expectations of my hardware. I've been trying to rule things out, but haven't been able to find my exact bottleneck.
My hardware
I realize that this isn't exactly server grade hardware. Most of the hardware is stuff that I already had laying around and it seemed to be enough for an initial test. I would like to upgrade to a Supermicro A2SDi-H-TF motherboard (with the Intel Atom C3758 processor) and 16GB of ECC memory in the coming weeks, but I'd first like to make sure that such a move would actually help my performance. I'm also aware that the Samsung 870 QVO SSD's are somewhat controversial (mainly due to their QLC memory). However, they should be sufficient for my particular workload. I expect a maximum utilization of 10-20 TBW per drive per year, so they should long outlast their warranty period and performance should be decent enough with their TurboWrite-cache. I'll be using a separate system for back-up purposes.
My test scenario
To test the performance of my TrueNAS system, I'm using one of my previous projects. The project consists out of a total of 68,249 files, which are spread over 1,342 folders and have combined size of 11.5 GB. Most of the files are extremely small, with just a few files responsible for most of the combined size. I've chosen this project as it's a great reflection of my average use case for this system. I'm measuring the performance using the following three metrics:
My main system consists of the following (relevant) hardware:
All copies in the test scenario are done to or from the M.2 boot drive of my main system. My main system and TrueNAS system are connected via a QNAP QSW-1105-5T switch and Cat 6a cabling (with a length of max 20 meters).
My hope/expectations
As a point of refence, I ran my test scenario against the built-in (4TB Samsung 960 EVO) SSD in my main system. This resulted in the following performance figures:
My hope would be to get within reach of these performance figures with my TrueNAS system. I'm fully aware that small files have always been somewhat of a sore point for SMB file transfers, so I'm not expecting to exceed the local performance figures and/or to get anywhere close to the maximum speed of my 2.5Gbps NIC.
My current performance figures
My test scenario currently results in the following figures when using the onboard 1 gigabit connection of my TrueNAS motherboard:
When I use the 2.5 gigabit connection of my PCIe Network Adapter, I get the following figures:
Without the few large files in the sample project, the average upload and download speeds would probably be closer to between 2 and 5 MB/sec.
As a reference, I also ran the test scenario over a gigabit connection against a WD My Cloud EX2 Ultra NAS with 2x 8TB WD Red HDD's (which is my previous setup) and this resulted in the following figures:
My current performance is definitely better than my previous setup was, but it's still not as close to the performance of a local drive as I'd like it to be.
Potential hardware bottlenecks
I've currently tried to look at the following potential hardware bottlenecks:
Potential software bottlenecks
I'm currently using just three minor tweaks to the default TrueNAS settings:
Please help
As you can see, I've been unable to locate the bottleneck of my current setup. My current performance looks to be somewhat similar to what one might expect out of an HDD. It's as if the TurboWrite-cache of the SSD's isn't used at all, even though this feature appears to be enabled on all four SSD's when doing a check from the CLI.
As I mentioned at the beginning of my post: I'm at a complete loss here and don't know where to look anymore. Your help is greatly appreciated!
The problem
I'm looking to improve my small file SMB performance, but can't seem to find the current bottleneck. I'm pretty new to TrueNAS and ZFS in general, so I wouldn't be surprised if I'm just missing a basic setting somewhere or maybe just have wrong expectations of my hardware. I've been trying to rule things out, but haven't been able to find my exact bottleneck.
My hardware
Motherboard | ASUS A320M-K |
CPU | AMD Ryzen 3 2200G |
Memory | 1x 8GB DDR4-2666 Corsair Vengeance LPX |
Boot Drive | 1x 250GB Samsung 980 (NVME M.2) |
Data Drives | 4x 2TB Samsung 870 QVO (SSD) in a RAID-Z2 configuration |
Network Interface Controller | EDUP 2.5GBase-T PCIe Network Adapter with an RTL8125 controller running in a PCIe 2.0 x1 slot |
GPU | Sapphire Pulse Radeon RX570 8GB |
PSU | Corsair CX450M (450W) |
Operating System | TrueNAS Core 12.0 U5.1 |
I realize that this isn't exactly server grade hardware. Most of the hardware is stuff that I already had laying around and it seemed to be enough for an initial test. I would like to upgrade to a Supermicro A2SDi-H-TF motherboard (with the Intel Atom C3758 processor) and 16GB of ECC memory in the coming weeks, but I'd first like to make sure that such a move would actually help my performance. I'm also aware that the Samsung 870 QVO SSD's are somewhat controversial (mainly due to their QLC memory). However, they should be sufficient for my particular workload. I expect a maximum utilization of 10-20 TBW per drive per year, so they should long outlast their warranty period and performance should be decent enough with their TurboWrite-cache. I'll be using a separate system for back-up purposes.
My test scenario
To test the performance of my TrueNAS system, I'm using one of my previous projects. The project consists out of a total of 68,249 files, which are spread over 1,342 folders and have combined size of 11.5 GB. Most of the files are extremely small, with just a few files responsible for most of the combined size. I've chosen this project as it's a great reflection of my average use case for this system. I'm measuring the performance using the following three metrics:
- Upload - The time it takes to copy the project from my main system to the TrueNAS system
- Download - The time it takes to copy the project back from the TrueNAS system to my main system
- Delete - The amount of files per second that can be deleted from my TrueNAS system
My main system consists of the following (relevant) hardware:
Motherboard | ASUS ROG Strix B250F Gaming |
CPU | Intel i7-7700 |
Memory | 2x 8GB DDR4-2400 Corsair Vengeance LPX |
Boot Drive | 1x 500GB Samsung 960 EVO (NVME M.2) |
Additional Drives | 1x 4TB Samsung 870 EVO (SSD) 1x 2TB WD Blue (HDD) |
Network Interface Controller | ASUS XG-C100C (with one 10GBase-T connection) |
Operating System | Windows 10 Pro 21H1 |
All copies in the test scenario are done to or from the M.2 boot drive of my main system. My main system and TrueNAS system are connected via a QNAP QSW-1105-5T switch and Cat 6a cabling (with a length of max 20 meters).
My hope/expectations
As a point of refence, I ran my test scenario against the built-in (4TB Samsung 960 EVO) SSD in my main system. This resulted in the following performance figures:
Upload (write) | 6 minutes (average of 32 MB/sec) |
Download (read) | 5.5 minutes (average of 35 MB/sec) |
Delete | 1200 files per second |
My hope would be to get within reach of these performance figures with my TrueNAS system. I'm fully aware that small files have always been somewhat of a sore point for SMB file transfers, so I'm not expecting to exceed the local performance figures and/or to get anywhere close to the maximum speed of my 2.5Gbps NIC.
My current performance figures
My test scenario currently results in the following figures when using the onboard 1 gigabit connection of my TrueNAS motherboard:
Upload (write) | 14 minutes (average of 14 MB/sec) |
Download (read) | 20 minutes (average of 10 MB/sec) |
Delete | 200 files per second |
When I use the 2.5 gigabit connection of my PCIe Network Adapter, I get the following figures:
Upload (write) | 12 minutes (average of 16 MB/sec) |
Download (read) | 18 minutes (average of 11 MB/sec) |
Delete | 220 files per second |
Without the few large files in the sample project, the average upload and download speeds would probably be closer to between 2 and 5 MB/sec.
As a reference, I also ran the test scenario over a gigabit connection against a WD My Cloud EX2 Ultra NAS with 2x 8TB WD Red HDD's (which is my previous setup) and this resulted in the following figures:
Upload (write) | 29 minutes (average of 7 MB/sec) |
Download (read) | 29 minutes (average of 7 MB/sec) |
Delete | 50 files per second |
My current performance is definitely better than my previous setup was, but it's still not as close to the performance of a local drive as I'd like it to be.
Potential hardware bottlenecks
I've currently tried to look at the following potential hardware bottlenecks:
- Network Interface Controller: Switching from a 1 gigabit connection to a 2.5 gigabit connection resulted in a mere 10% performance increase (mainly due to the few large files). I did a separate test with a single large file to confirm that my maximum transfer speed is indeed around 270 MB/sec, so the 2.5 gigabit connection appears to be working properly.
- SSD's: While I don't have any replacement SSD's for the pool, I did instead try to run the pool as a striped VDEV with all four SSD's. This did not lead to any appreciable difference in the performance figures. With the current RAID-Z2 pool, performance should theoretically be limited to the IOPS of a single SSD, with other performance figures actually performing better due to the number of drives. These theoretical RAID-Z2 performance figures are nowhere close to being met.
- Memory: According to the TrueNAS dashboard, most of the memory is either free or being used as ZFS cache. While 8GB may not be a lot, it appears to be sufficient for this rather small pool (with a total usable storage capacity of around 3.5TB).
- CPU: I do not have any drop in replacements for the CPU of my TrueNAS system or for the one of my main system. However, my TrueNAS system averages around 15% load during uploads and downloads, which is less then the 25% load I would expect in case of a continuous max load on a single thread. My main system averages around an 8% load for Windows Explorer, which is also less the potential 12.5% average in case of a max load of a single thread. Both systems should theoretically be able to use Receive Side Scaling (RSS) in order to utilize multiple threads, but I haven't seen any indication of it being used on this single connection. The workload is already divided between multiple threads (even with the onboard 1 gigabit connection), but the CPU load doesn't rise above single thread levels. I'm not sure if the "re" driver for the RTL8125 chip actually supports RSS for TrueNAS, but the chip itself is capable of using this feature.
Potential software bottlenecks
I'm currently using just three minor tweaks to the default TrueNAS settings:
- I've turned off the compression of my SMB dataset. This seemed to result in a rather small performance improvement (less than 3%).
- I've disabled the sync option of SMB dataset. This seemed to result in a rather small performance improvement (less than 3%).
- I've added the "server multi channel support = yes" auxiliary parameter to my SMB service config. This did not seem to impact performance.
Please help
As you can see, I've been unable to locate the bottleneck of my current setup. My current performance looks to be somewhat similar to what one might expect out of an HDD. It's as if the TurboWrite-cache of the SSD's isn't used at all, even though this feature appears to be enabled on all four SSD's when doing a check from the CLI.
As I mentioned at the beginning of my post: I'm at a complete loss here and don't know where to look anymore. Your help is greatly appreciated!