Slow speed ssd pool over 40gbit iSCSI

[flwstern]

Hello everyone,

I have a TrueNAS SCALE system based on an HP DL380 G9 with 256GB ECC RAM, 20 cores 40 threads, and a ConnectX-3 40Gbit DAC 0.5M connected to a Juniper QFX5100 switch.

I never exceed 8 Gbit/sec on any of the pools. On the Micron pool, which is a 9200 Pro Micron that's supposed to handle 840k read IOPS / 180k write IOPS, I only get 105k IOPS with fio and exactly the same benchmark on the dc1000m pool. The ssd-san pool consists of a 3x mirror SATA Kingston DC550M 3.84TB SSDs. I am running an MTU size of 8900 all the way from TrueNAS > Juniper > VMware ESXi, solely on layer 2.

I have 3 SSD storage pools:

nvme_dc1000 3.48T 50.0G 3.44T - - 0% 1% 1.00x ONLINE /mnt
nvme_micron 6.97T 82.8G 6.89T - - 0% 1% 1.00x ONLINE /mnt
ssd-san 10.5T 1.76T 8.70T - - 38% 16% 1.00x ONLINE /mnt

Pool: ssd-san
State: ONLINE
Scan: scrub repaired 0B in 00:50:07 with 0 errors on Sun Jan 28 00:50:09 2024
Config:

Code:

    NAME                                      STATE     READ WRITE CKSUM
    ssd-san                                   ONLINE       0     0     0
      mirror-0                                ONLINE       0     0     0
        183ad214-d779-4609-a361-85b0aa5e6b86  ONLINE       0     0     0
        a5813287-f58b-43ba-9638-6b2c34e2d04e  ONLINE       0     0     0
      mirror-1                                ONLINE       0     0     0
        77cdbc87-2c79-4ab7-995f-0686ff284dc0  ONLINE       0     0     0
        9aef7328-2054-4a0c-bc2f-28a915e3dd76  ONLINE       0     0     0
      mirror-2                                ONLINE       0     0     0
        d11584c1-418f-42fd-a2d0-ac4cd5549e96  ONLINE       0     0     0
        56b95dd7-236d-4ff7-83ce-c3b11f92892e  ONLINE       0     0     0

Pool: nvme_micron (Micron 9200)
State: ONLINE
Scan: scrub repaired 0B in 00:00:00 with 0 errors on Sun Jan 28 00:00:02 2024
Config:

Code:

    NAME                                      STATE     READ WRITE CKSUM
    nvme_micron                               ONLINE       0     0     0
      mirror-0                                ONLINE       0     0     0
        26fd441b-b087-4815-b0b6-3638da107f06  ONLINE       0     0     0
        a342e2b6-28c3-409f-98ab-99d5a416c6bd  ONLINE       0     0     0

Pool: nvme_dc1000
State: ONLINE
Scan: scrub repaired 0B in 00:00:00 with 0 errors on Sun Jan 28 00:00:02 2024
Config:

Code:

    NAME                                      STATE     READ WRITE CKSUM
    nvme_dc1000                               ONLINE       0     0     0
      mirror-0                                ONLINE       0     0     0
        fc4b9f5e-99d6-450e-a3c3-290bfcfe4b7d  ONLINE       0     0     0
        89e96c8b-9e14-4d1a-9e2b-4022a87e3741  ONLINE       0     0     0

 

[flwstern]

here are the stats on 1 VM running on the micron 9200 over iscsi that the only load on the pool