my hardware:
Motherboard msi tomahawk b550 https://www.msi.com/Motherboard/MAG-B550-TOMAHAWK/Specification
CPU AMD Ryzen 5 3600 https://www.amd.com/en/products/cpu/amd-ryzen-5-3600
RAM 32GB Kit 2x16GB DDR4-3200 PC4-25600 ECC 2Rx8 Unbuffered Server Memory
Controller ADAPTEC ASR-71605 SFF8643 16 PORT SAS SATA 6Gb/s HBA/RAID in HBA mode
network 2 mobo nics + Intel X520-DA2 10GB Dual Port PCI-E E10G42BTDA
2 pools, primary and secondary.
primary 1 vdev raidz2, 6 ssd, connected to mobo 6 sata ports:
vdev1 3x Samsung_SSD_870_QVO_2TB
vdev1 Samsung_SSD_870_EVO_2TB
vdev1 Samsung_SSD_870_QVO_2TB
vdev1 SPCC_Solid_State_Disk
secondary, 2 vdevs mirrored rotational, connected to adaptec controller:
vdev1 ST8000VN004-2M2101 8tb
vdev1 WDC_WD80EDAZ-11TA3A0 8tb
vdev2 TOSHIBA_HDWE140 4tb
vdev2 WDC WD4000FYYZ-01UL1B3 4b
software:
TrueNAS-SCALE-22.02.3
non-standard settings:
secondary has zfs set cache=metadata secondary
Being used as a backup server for a few desktop pcs, ip camera nvr and vm disk storage repository for xcp-ng (12 vms moderate to low disk io) via the 10g nic, nfs share, sync disabled and directly connected to truenas w/ jumbo frames enabled, nic speeds are as expected.
primary pool hosts vm storage repository and the backups for the desktops. Secondary pool has the drive for frigate to write the ip camera recordings. Primary pool also replicates to the secondary pool as a backup.
The problem... My rotational array performs better than my ssd array. Originally was setup with a 5 disk raid 6 array on an hp server. The drives after a year of use started acting slow. So I decided to standup truenas so
I could get better insight to what was going on, centralize the storage and feel safer with my data.
What I am experiencing:
Transfers of large vm disks to truenas error and fault drives on the array if the copy is for longer than 20 minutes. It is usually the qvo drives crapping out. I can zfs clear them and they resilver and jump back in.
Moving applications or drives to the secondary pool preform 3x better than the primary ssd pool.
I did not buy my hardware with truenas in mind, I stood up the server with what I had available. After doing a lot of forum searching it seems people have problems with the samsung qvo drives. I also see the controller I have doesn't seem to be a favorite either. It is in hba mode but when I have the controller doing the ssd array, I can not trim the drives. This is why I moved it to the onboard sata controller. I am happy to switch out parts gradually to make this a much better storage server. I can roughly buy a part or disk once a month to upgrade/fix. I had 4 qvo drives but I used this months budget to replace one with the evo.
What I am asking is how do I troubleshoot this with my existing hardware and/or what should I replace to get my ssd pool preforming. I mostly care about the vm's running in my lab and I don't have a huge budget to make costly changes. My largest concern is that when the disk work load is high for 20 minutes the array degrades.
Any help is greatly appreciated.
Motherboard msi tomahawk b550 https://www.msi.com/Motherboard/MAG-B550-TOMAHAWK/Specification
CPU AMD Ryzen 5 3600 https://www.amd.com/en/products/cpu/amd-ryzen-5-3600
RAM 32GB Kit 2x16GB DDR4-3200 PC4-25600 ECC 2Rx8 Unbuffered Server Memory
Controller ADAPTEC ASR-71605 SFF8643 16 PORT SAS SATA 6Gb/s HBA/RAID in HBA mode
network 2 mobo nics + Intel X520-DA2 10GB Dual Port PCI-E E10G42BTDA
2 pools, primary and secondary.
primary 1 vdev raidz2, 6 ssd, connected to mobo 6 sata ports:
vdev1 3x Samsung_SSD_870_QVO_2TB
vdev1 Samsung_SSD_870_EVO_2TB
vdev1 Samsung_SSD_870_QVO_2TB
vdev1 SPCC_Solid_State_Disk
secondary, 2 vdevs mirrored rotational, connected to adaptec controller:
vdev1 ST8000VN004-2M2101 8tb
vdev1 WDC_WD80EDAZ-11TA3A0 8tb
vdev2 TOSHIBA_HDWE140 4tb
vdev2 WDC WD4000FYYZ-01UL1B3 4b
software:
TrueNAS-SCALE-22.02.3
non-standard settings:
secondary has zfs set cache=metadata secondary
Being used as a backup server for a few desktop pcs, ip camera nvr and vm disk storage repository for xcp-ng (12 vms moderate to low disk io) via the 10g nic, nfs share, sync disabled and directly connected to truenas w/ jumbo frames enabled, nic speeds are as expected.
primary pool hosts vm storage repository and the backups for the desktops. Secondary pool has the drive for frigate to write the ip camera recordings. Primary pool also replicates to the secondary pool as a backup.
The problem... My rotational array performs better than my ssd array. Originally was setup with a 5 disk raid 6 array on an hp server. The drives after a year of use started acting slow. So I decided to standup truenas so
I could get better insight to what was going on, centralize the storage and feel safer with my data.
What I am experiencing:
Transfers of large vm disks to truenas error and fault drives on the array if the copy is for longer than 20 minutes. It is usually the qvo drives crapping out. I can zfs clear them and they resilver and jump back in.
Moving applications or drives to the secondary pool preform 3x better than the primary ssd pool.
I did not buy my hardware with truenas in mind, I stood up the server with what I had available. After doing a lot of forum searching it seems people have problems with the samsung qvo drives. I also see the controller I have doesn't seem to be a favorite either. It is in hba mode but when I have the controller doing the ssd array, I can not trim the drives. This is why I moved it to the onboard sata controller. I am happy to switch out parts gradually to make this a much better storage server. I can roughly buy a part or disk once a month to upgrade/fix. I had 4 qvo drives but I used this months budget to replace one with the evo.
What I am asking is how do I troubleshoot this with my existing hardware and/or what should I replace to get my ssd pool preforming. I mostly care about the vm's running in my lab and I don't have a huge budget to make costly changes. My largest concern is that when the disk work load is high for 20 minutes the array degrades.
Any help is greatly appreciated.