iSCSI multipath not working correctly

Status
Not open for further replies.

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Here's the image from Crystal Diskmark:
yVpdc7C.jpg


And here's the output from running dd on our linux vm
dd if=/dev/zero of=/mnt/test.dat bs=1024 count=500k
512000+0 records in
512000+0 records out
524288000 bytes (525 MB) copied, 2.60701 s, 201MB/s
That looks fine to me too. You can try increasing the number of threads in the test settings.
 

Dudde

Explorer
Joined
Oct 5, 2015
Messages
77
Update:
Since yesterday we have tried adding two extra disk to our Raid 10 setup, we have tried sending files between ssd (freenas) to an local ssd on esxi host.
We have tried with SLOG in all diffrent sizes.

Running dd on our linux box gives me 370MB/s in write performance, that's all good.

But our read performance is less then half that, about 150MB/s

All kind of file transfers in our windows box (windows 2012 server R2) peek at 150MB/s

How can we try to improve our read performance?
 

Dudde

Explorer
Joined
Oct 5, 2015
Messages
77
Update2: Okey after hours of testing and benchmarking im still not moving anywhere.
We have tried setting up our zpools in mirrors, stripes, with and without l2arc, with and without slog.
The strange part is that even when we use our ssd's our benchmark results doesn't change.
Current zpools setup:
HVG-data
mirror0
da1
da2
mirror1
da3
da4

HVG-sys
mirror0
da5
da6

See picture below. This is the maximum scores we can manage to get. And these results doesn't change no matter raid level, ssd or hdd setup.
Q5VvZva.png


Ok Sequential read and writes with a Queue Depth of 32 and 1 Threads max out our 3x1Gbit connection so we know Round Robin and MPIO works correctly.
Let's not focus too much on Crystal diskmark since i learned Seq writes and reads use their own QD settings.

Running SQLIO-Sim i get the following results:
3hjcsA8.png


8EkFRhi.png


Using Diskspd gives me the following results:
Jy5L0BH.png


Normal Filecopy using windows:
03xNLJg.png


DD write results:
r9ZG1wZ.png

DD read results:
LMYoKpU.png

Running DD locally on freenas (/mnt/iscsishare):
(dd if=/dev/zero of=test bs=1G count=10)
10+0 records in
10+0 records out
10737418240 bytes transferred in 33.079174 secs (324597532 bytes/sec)


Strange thing, running Crystal diskmark on my ESXI's local ssd datastore gives me the exact same result?
 

Dudde

Explorer
Joined
Oct 5, 2015
Messages
77
Update 3:

Seems like we're having problem with Jumbo frames.
Here's an output from our tcpdump while trying to ping from freenas to our esxi
(ping -d -s 8972 10.10.10.200)
tcpdump output:
10.10.10.100 > 10.10.10.200: ICMP echo request, id 51480, seq 6, length 1480
15:41:12.150650 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46296, offset 1480, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:12.150733 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46296, offset 2960, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:12.150762 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46296, offset 4440, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:12.150787 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46296, offset 5920, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:12.150811 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46296, offset 7400, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:12.150836 IP truncated-ip - 6 bytes missing! (tos 0x0, ttl 64, id 46296, offset 8880, flags [none], proto ICMP (1), length 88)
10.10.10.100 > 10.10.10.200: icmp
15:41:12.150861 IP truncated-ip - 8886 bytes missing! (tos 0x0, ttl 64, id 14630, offset 0, flags [none], proto ICMP (1), length 8968)
10.10.10.200 > 10.10.10.100: ICMP echo reply, id 51480, seq 6, length 8948
15:41:13.151597 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46360, offset 0, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: ICMP echo request, id 51480, seq 7, length 1480
15:41:13.151610 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46360, offset 1480, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:13.151927 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46360, offset 2960, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:13.151968 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46360, offset 4440, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:13.151997 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46360, offset 5920, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:13.152023 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46360, offset 7400, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:13.152047 IP truncated-ip - 6 bytes missing! (tos 0x0, ttl 64, id 46360, offset 8880, flags [none], proto ICMP (1), length 88)
10.10.10.100 > 10.10.10.200: icmp
15:41:13.152072 IP truncated-ip - 8886 bytes missing! (tos 0x0, ttl 64, id 14636, offset 0, flags [none], proto ICMP (1), length 8968)
10.10.10.200 > 10.10.10.100: ICMP echo reply, id 51480, seq 7, length 8948
15:41:13.193443 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 46364, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.53112: Flags [P.], ack 2675792010, win 16384, options [nop,nop,TS val 136765846 ecr 639080], length 48
15:41:13.193471 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 46366, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [P.], ack 4099515221, win 16384, options [nop,nop,TS val 4004329775 ecr 639080], length 48
15:41:13.193479 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 14639, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.53112 > 10.10.10.100.3260: Flags [P.], ack 48, win 514, options [nop,nop,TS val 639280 ecr 136765846], length 48
15:41:13.193484 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 14640, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.24221 > 10.10.10.100.3260: Flags [P.], ack 48, win 514, options [nop,nop,TS val 639280 ecr 4004329775], length 48
15:41:13.193817 IP (tos 0x0, ttl 64, id 46369, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.53112: Flags [.], cksum 0xea91 (correct), ack 49, win 16383, options [nop,nop,TS val 136765846 ecr 639280], length 0
15:41:13.193823 IP (tos 0x0, ttl 64, id 46370, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [.], cksum 0x99d5 (correct), ack 49, win 16383, options [nop,nop,TS val 4004329775 ecr 639280], length 0
15:41:13.194411 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 46371, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.29751: Flags [P.], ack 1072, win 16384, options [nop,nop,TS val 268115168 ecr 639163], length 48
15:41:13.194421 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 14642, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.29751 > 10.10.10.100.3260: Flags [P.], ack 97, win 514, options [nop,nop,TS val 639280 ecr 268115168], length 48
15:41:13.194563 IP (tos 0x0, ttl 64, id 46372, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.29751: Flags [.], cksum 0xa60d (correct), ack 1120, win 16383, options [nop,nop,TS val 268115168 ecr 639280], length 0
15:41:14.841652 IP truncated-ip - 4114 bytes missing! (tos 0x0, ttl 64, id 14646, offset 0, flags [DF], proto TCP (6), length 4196)
10.10.10.200.24221 > 10.10.10.100.3260: Flags [P.], ack 48, win 514, options [nop,nop,TS val 639445 ecr 4004329775], length 4144
15:41:14.841823 IP (tos 0x0, ttl 64, id 46574, offset 0, flags [DF], proto TCP (6), length 52)


tcpdump output (ping from esxi to freenas)
15:47:52.191659 IP (tos 0x0, ttl 64, id 4681, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.53112: Flags [.], cksum 0x2f4f (correct), ack 1120, win 16383, options [nop,nop,TS val 137164846 ecr 679179], length 0
15:47:52.191665 IP (tos 0x0, ttl 64, id 4682, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [.], cksum 0xabeb (correct), ack 49, win 16383, options [nop,nop,TS val 4004728775 ecr 679179], length 0
15:47:53.192494 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 4743, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.29751: Flags [P.], ack 49, win 16384, options [nop,nop,TS val 268515168 ecr 679079], length 48
15:47:53.192511 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 18207, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.29751 > 10.10.10.100.3260: Flags [P.], ack 96, win 514, options [nop,nop,TS val 679279 ecr 268515168], length 48
15:47:53.192650 IP (tos 0x0, ttl 64, id 4744, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.29751: Flags [.], cksum 0xe9ce (correct), ack 97, win 16383, options [nop,nop,TS val 268515168 ecr 679279], length 0
15:47:53.922676 IP truncated-ip - 1042 bytes missing! (tos 0x0, ttl 64, id 18215, offset 0, flags [DF], proto TCP (6), length 1124)
10.10.10.200.24221 > 10.10.10.100.3260: Flags [P.], ack 48, win 514, options [nop,nop,TS val 679352 ecr 4004728775], length 1072
15:47:53.922818 IP (tos 0x0, ttl 64, id 4775, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [.], cksum 0xa05b (correct), ack 1121, win 16367, options [nop,nop,TS val 4004730506 ecr 679352], length 0
15:47:53.922928 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 4776, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [P.], ack 1121, win 16384, options [nop,nop,TS val 4004730506 ecr 679352], length 48
15:47:54.037796 IP (tos 0x0, ttl 64, id 18216, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.200.24221 > 10.10.10.100.3260: Flags [.], cksum 0x2966 (incorrect -> 0xde0c), ack 96, win 514, options [nop,nop,TS val 679364 ecr 4004730506], length 0
15:47:54.191498 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 4781, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.53112: Flags [P.], ack 1120, win 16384, options [nop,nop,TS val 137166846 ecr 679179], length 48
15:47:54.191514 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 18221, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.53112 > 10.10.10.100.3260: Flags [P.], ack 145, win 514, options [nop,nop,TS val 679379 ecr 137166846], length 48
15:47:54.191657 IP (tos 0x0, ttl 64, id 4784, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.53112: Flags [.], cksum 0x2657 (correct), ack 1168, win 16383, options [nop,nop,TS val 137166846 ecr 679379], length 0
15:47:55.191467 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 4846, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [P.], ack 1121, win 16384, options [nop,nop,TS val 4004731775 ecr 679364], length 48
15:47:55.191484 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 18226, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.24221 > 10.10.10.100.3260: Flags [P.], ack 144, win 514, options [nop,nop,TS val 679479 ecr 4004731775], length 48
15:47:55.191624 IP (tos 0x0, ttl 64, id 4850, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [.], cksum 0x9a47 (correct), ack 1169, win 16383, options [nop,nop,TS val 4004731775 ecr 679479], length 0
15:47:55.192417 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 4851, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.29751: Flags [P.], ack 97, win 16384, options [nop,nop,TS val 268517168 ecr 679279], length 48
15:47:55.192450 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 18227, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.29751 > 10.10.10.100.3260: Flags [P.], ack 144, win 514, options [nop,nop,TS val 679479 ecr 268517168], length 48
15:47:55.192605 IP (tos 0x0, ttl 64, id 4852, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.29751: Flags [.], cksum 0xe0d6 (correct), ack 145, win 16383, options [nop,nop,TS val 268517168 ecr 679479], length 0
15:47:55.562767 IP truncated-ip - 8918 bytes missing! (tos 0x0, ttl 64, id 18230, offset 0, flags [DF], proto ICMP (1), length 9000)
10.10.10.200 > 10.10.10.100: ICMP echo request, id 52894, seq 0, length 8980
15:47:56.191480 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 4882, offset 0, flags [DF], proto TCP (6), length 100)
 

Dudde

Explorer
Joined
Oct 5, 2015
Messages
77
Well we got it working with jumbo frames and the system is up and running like it should.
 
Status
Not open for further replies.
Top