JaymzMac
Cadet
- Joined
- Mar 13, 2019
- Messages
- 5
I'm seeing some strange behaviour for one of my vdevs and I wondered if someone here could shed some light on what is happening.
Here's my pool configuration:
For a period of time, we see that the disks in the 2nd vdev are extremely busy, and have high latency, while the other vdevs are performing as expected.
As you can see from the following gstat output, the disks in the 2nd vdev are running at 100% busy and the latency for flushing is over 250ms:
I've made the following observations while experiencing the issue:
1. According to the gstat output, the IO size of the write operations average around 6-8KB on the 2nd vdev, whereas on the other vdevs the size is around 20-30KB.
2. The number of writes operations to the disks in the second vdev is higher than the writes to the other vdevs.
3. All the disks in the second vdev are running on average at 90-100% busy. The other disks are running at around 40-50% busy.
4. There is not one specific disk in the 2nd vdev which is causing the issue.
5. We're running with sync set to always, and when I run zilstat, there are very few operations between 4-32kB being written to the slog from the clients.
6. The issue normally lasts for a few hours before it disappears and the performance becomes uniform across all the vdevs again.
My theory is that the small write size is making the disks in the vdev extremely busy, and causing the high latency, slowing down the pool performance. But where are the small IOs coming from, if they're not coming from the clients (I can't see them in zilstat)? And why are they isolated to the 2nd vdev? Shouldn't it be spread across the vdevs assuming equal usage across the pool? Could it be some kind of internal ZFS workload that I'm not aware of causing the issue?
Appreciate any thoughts / ideas from the community.
Here's my pool configuration:
Code:
zpool list -v NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT ex-freenas-01 195T 91.6T 103T - 28% 46% 1.00x ONLINE /mnt raidz2 65T 30.1T 34.9T - 28% 46% gptid/1231bea4-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/12fd9bab-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/13c714d0-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/15d05b7c-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/17cac22c-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/19dd2db3-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - raidz2 65T 30.7T 34.3T - 28% 47% gptid/1b01046c-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/1be27e07-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/1cb75a2e-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/1d8901a8-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/1e5e6bea-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/1f3fc5c7-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - raidz2 65T 30.7T 34.3T - 28% 47% gptid/20554da7-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/21386d9c-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/2229a259-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/23172de4-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/24068c91-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/250f5938-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - log - - - - - - mirror 184G 174M 184G - 0% 0% gptid/2641fb9e-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - gptid/26cd51b6-54e1-11e8-a35e-ac1f6b8d5ba6 - - - - - - cache - - - - - - gptid/25b4e86c-54e1-11e8-a35e-ac1f6b8d5ba6 373G 357G 15.7G - 0% 95%
For a period of time, we see that the disks in the 2nd vdev are extremely busy, and have high latency, while the other vdevs are performing as expected.
As you can see from the following gstat output, the disks in the 2nd vdev are running at 100% busy and the latency for flushing is over 250ms:
Code:
dT: 1.006s w: 1.000s filter: ^da[0-9]+$ L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w d/s kB kBps ms/d o/s ms/o %busy Name 5 194 61 12 700 18.2 131 24 3107 0.7 0 0 0 0.0 2 45.6 35.1 da0 11 274 138 8 1122 12.1 134 22 3015 0.2 0 0 0 0.0 2 21.2 42.6 da1 14 268 109 6 708 15.4 157 21 3258 0.8 0 0 0 0.0 2 24.7 40.4 da2 3 243 86 7 557 13.9 155 21 3198 0.2 0 0 0 0.0 2 41.0 32.3 da3 3 298 136 7 891 15.6 160 20 3174 0.2 0 0 0 0.0 2 24.2 41.6 da4 2 250 89 10 887 10.3 158 21 3290 0.2 0 0 0 0.0 2 24.1 34.4 da5 ----------------------------------------------------------------------------------------------------------------------- 2 362 118 15 1786 47.2 242 7 1647 0.3 0 0 0 0.0 2 257.1 84.4 da6 2 383 128 14 1794 56.2 253 7 1718 0.2 0 0 0 0.0 2 283.1 86.0 da7 3 293 105 16 1683 97.6 186 8 1448 0.3 0 0 0 0.0 2 366.3 98.7 da8 6 309 128 14 1806 65.9 179 8 1404 0.3 0 0 0 0.0 2 279.9 100.4 da9 6 386 104 22 2291 99.5 279 7 1834 0.3 0 0 0 0.0 2 346.6 99.8 da10 3 371 125 15 1889 67.7 244 7 1671 0.2 0 0 0 0.0 2 298.3 93.8 da11 ----------------------------------------------------------------------------------------------------------------------- 1 362 189 22 4244 4.9 171 17 2828 0.2 0 0 0 0.0 2 25.5 33.7 da12 1 290 126 24 3063 15.8 162 17 2812 0.4 0 0 0 0.0 2 34.7 40.0 da13 1 240 73 15 1090 25.3 165 18 2900 0.2 0 0 0 0.0 2 33.1 32.6 da14 1 200 48 28 1325 17.6 150 19 2788 0.2 0 0 0 0.0 2 33.3 27.7 da15 1 290 147 29 4284 10.7 141 20 2788 0.3 0 0 0 0.0 2 28.5 40.2 da16 1 498 327 14 4519 2.8 169 17 2916 0.3 0 0 0 0.0 2 33.9 35.4 da17 ----------------------------------------------------------------------------------------------------------------------- 0 3152 0 0 0 0.0 1605 36 57103 0.1 0 0 0 0.0 1547 0.0 21.7 da18 0 3152 0 0 0 0.0 1605 36 57103 0.1 0 0 0 0.0 1547 0.0 21.4 da19 ----------------------------------------------------------------------------------------------------------------------- 0 1361 1190 4 4936 0.1 171 121 20759 0.7 0 0 0 0.0 0 0.0 6.4 da20
I've made the following observations while experiencing the issue:
1. According to the gstat output, the IO size of the write operations average around 6-8KB on the 2nd vdev, whereas on the other vdevs the size is around 20-30KB.
2. The number of writes operations to the disks in the second vdev is higher than the writes to the other vdevs.
3. All the disks in the second vdev are running on average at 90-100% busy. The other disks are running at around 40-50% busy.
4. There is not one specific disk in the 2nd vdev which is causing the issue.
5. We're running with sync set to always, and when I run zilstat, there are very few operations between 4-32kB being written to the slog from the clients.
6. The issue normally lasts for a few hours before it disappears and the performance becomes uniform across all the vdevs again.
My theory is that the small write size is making the disks in the vdev extremely busy, and causing the high latency, slowing down the pool performance. But where are the small IOs coming from, if they're not coming from the clients (I can't see them in zilstat)? And why are they isolated to the 2nd vdev? Shouldn't it be spread across the vdevs assuming equal usage across the pool? Could it be some kind of internal ZFS workload that I'm not aware of causing the issue?
Appreciate any thoughts / ideas from the community.