Update 3:
Seems like we're having problem with Jumbo frames.
Here's an output from our tcpdump while trying to ping from freenas to our esxi
(ping -d -s 8972 10.10.10.200)
tcpdump output:
10.10.10.100 > 10.10.10.200: ICMP echo request, id 51480, seq 6, length 1480
15:41:12.150650 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46296, offset 1480, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:12.150733 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46296, offset 2960, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:12.150762 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46296, offset 4440, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:12.150787 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46296, offset 5920, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:12.150811 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46296, offset 7400, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:12.150836 IP truncated-ip - 6 bytes missing! (tos 0x0, ttl 64, id 46296, offset 8880, flags [none], proto ICMP (1), length 88)
10.10.10.100 > 10.10.10.200: icmp
15:41:12.150861 IP truncated-ip - 8886 bytes missing! (tos 0x0, ttl 64, id 14630, offset 0, flags [none], proto ICMP (1), length 8968)
10.10.10.200 > 10.10.10.100: ICMP echo reply, id 51480, seq 6, length 8948
15:41:13.151597 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46360, offset 0, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: ICMP echo request, id 51480, seq 7, length 1480
15:41:13.151610 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46360, offset 1480, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:13.151927 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46360, offset 2960, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:13.151968 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46360, offset 4440, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:13.151997 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46360, offset 5920, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:13.152023 IP truncated-ip - 1418 bytes missing! (tos 0x0, ttl 64, id 46360, offset 7400, flags [+], proto ICMP (1), length 1500)
10.10.10.100 > 10.10.10.200: icmp
15:41:13.152047 IP truncated-ip - 6 bytes missing! (tos 0x0, ttl 64, id 46360, offset 8880, flags [none], proto ICMP (1), length 88)
10.10.10.100 > 10.10.10.200: icmp
15:41:13.152072 IP truncated-ip - 8886 bytes missing! (tos 0x0, ttl 64, id 14636, offset 0, flags [none], proto ICMP (1), length 8968)
10.10.10.200 > 10.10.10.100: ICMP echo reply, id 51480, seq 7, length 8948
15:41:13.193443 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 46364, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.53112: Flags [P.], ack 2675792010, win 16384, options [nop,nop,TS val 136765846 ecr 639080], length 48
15:41:13.193471 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 46366, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [P.], ack 4099515221, win 16384, options [nop,nop,TS val 4004329775 ecr 639080], length 48
15:41:13.193479 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 14639, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.53112 > 10.10.10.100.3260: Flags [P.], ack 48, win 514, options [nop,nop,TS val 639280 ecr 136765846], length 48
15:41:13.193484 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 14640, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.24221 > 10.10.10.100.3260: Flags [P.], ack 48, win 514, options [nop,nop,TS val 639280 ecr 4004329775], length 48
15:41:13.193817 IP (tos 0x0, ttl 64, id 46369, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.53112: Flags [.], cksum 0xea91 (correct), ack 49, win 16383, options [nop,nop,TS val 136765846 ecr 639280], length 0
15:41:13.193823 IP (tos 0x0, ttl 64, id 46370, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [.], cksum 0x99d5 (correct), ack 49, win 16383, options [nop,nop,TS val 4004329775 ecr 639280], length 0
15:41:13.194411 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 46371, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.29751: Flags [P.], ack 1072, win 16384, options [nop,nop,TS val 268115168 ecr 639163], length 48
15:41:13.194421 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 14642, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.29751 > 10.10.10.100.3260: Flags [P.], ack 97, win 514, options [nop,nop,TS val 639280 ecr 268115168], length 48
15:41:13.194563 IP (tos 0x0, ttl 64, id 46372, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.29751: Flags [.], cksum 0xa60d (correct), ack 1120, win 16383, options [nop,nop,TS val 268115168 ecr 639280], length 0
15:41:14.841652 IP truncated-ip - 4114 bytes missing! (tos 0x0, ttl 64, id 14646, offset 0, flags [DF], proto TCP (6), length 4196)
10.10.10.200.24221 > 10.10.10.100.3260: Flags [P.], ack 48, win 514, options [nop,nop,TS val 639445 ecr 4004329775], length 4144
15:41:14.841823 IP (tos 0x0, ttl 64, id 46574, offset 0, flags [DF], proto TCP (6), length 52)
tcpdump output (ping from esxi to freenas)
15:47:52.191659 IP (tos 0x0, ttl 64, id 4681, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.53112: Flags [.], cksum 0x2f4f (correct), ack 1120, win 16383, options [nop,nop,TS val 137164846 ecr 679179], length 0
15:47:52.191665 IP (tos 0x0, ttl 64, id 4682, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [.], cksum 0xabeb (correct), ack 49, win 16383, options [nop,nop,TS val 4004728775 ecr 679179], length 0
15:47:53.192494 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 4743, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.29751: Flags [P.], ack 49, win 16384, options [nop,nop,TS val 268515168 ecr 679079], length 48
15:47:53.192511 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 18207, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.29751 > 10.10.10.100.3260: Flags [P.], ack 96, win 514, options [nop,nop,TS val 679279 ecr 268515168], length 48
15:47:53.192650 IP (tos 0x0, ttl 64, id 4744, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.29751: Flags [.], cksum 0xe9ce (correct), ack 97, win 16383, options [nop,nop,TS val 268515168 ecr 679279], length 0
15:47:53.922676 IP truncated-ip - 1042 bytes missing! (tos 0x0, ttl 64, id 18215, offset 0, flags [DF], proto TCP (6), length 1124)
10.10.10.200.24221 > 10.10.10.100.3260: Flags [P.], ack 48, win 514, options [nop,nop,TS val 679352 ecr 4004728775], length 1072
15:47:53.922818 IP (tos 0x0, ttl 64, id 4775, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [.], cksum 0xa05b (correct), ack 1121, win 16367, options [nop,nop,TS val 4004730506 ecr 679352], length 0
15:47:53.922928 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 4776, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [P.], ack 1121, win 16384, options [nop,nop,TS val 4004730506 ecr 679352], length 48
15:47:54.037796 IP (tos 0x0, ttl 64, id 18216, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.200.24221 > 10.10.10.100.3260: Flags [.], cksum 0x2966 (incorrect -> 0xde0c), ack 96, win 514, options [nop,nop,TS val 679364 ecr 4004730506], length 0
15:47:54.191498 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 4781, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.53112: Flags [P.], ack 1120, win 16384, options [nop,nop,TS val 137166846 ecr 679179], length 48
15:47:54.191514 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 18221, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.53112 > 10.10.10.100.3260: Flags [P.], ack 145, win 514, options [nop,nop,TS val 679379 ecr 137166846], length 48
15:47:54.191657 IP (tos 0x0, ttl 64, id 4784, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.53112: Flags [.], cksum 0x2657 (correct), ack 1168, win 16383, options [nop,nop,TS val 137166846 ecr 679379], length 0
15:47:55.191467 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 4846, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [P.], ack 1121, win 16384, options [nop,nop,TS val 4004731775 ecr 679364], length 48
15:47:55.191484 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 18226, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.24221 > 10.10.10.100.3260: Flags [P.], ack 144, win 514, options [nop,nop,TS val 679479 ecr 4004731775], length 48
15:47:55.191624 IP (tos 0x0, ttl 64, id 4850, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.24221: Flags [.], cksum 0x9a47 (correct), ack 1169, win 16383, options [nop,nop,TS val 4004731775 ecr 679479], length 0
15:47:55.192417 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 4851, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.100.3260 > 10.10.10.200.29751: Flags [P.], ack 97, win 16384, options [nop,nop,TS val 268517168 ecr 679279], length 48
15:47:55.192450 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 18227, offset 0, flags [DF], proto TCP (6), length 100)
10.10.10.200.29751 > 10.10.10.100.3260: Flags [P.], ack 144, win 514, options [nop,nop,TS val 679479 ecr 268517168], length 48
15:47:55.192605 IP (tos 0x0, ttl 64, id 4852, offset 0, flags [DF], proto TCP (6), length 52)
10.10.10.100.3260 > 10.10.10.200.29751: Flags [.], cksum 0xe0d6 (correct), ack 145, win 16383, options [nop,nop,TS val 268517168 ecr 679479], length 0
15:47:55.562767 IP truncated-ip - 8918 bytes missing! (tos 0x0, ttl 64, id 18230, offset 0, flags [DF], proto ICMP (1), length 9000)
10.10.10.200 > 10.10.10.100: ICMP echo request, id 52894, seq 0, length 8980
15:47:56.191480 IP truncated-ip - 18 bytes missing! (tos 0x0, ttl 64, id 4882, offset 0, flags [DF], proto TCP (6), length 100)