Another *sigh* 10gb network speed thread

[ViciousXUSMC]

I have searched web all night and day, then crawled through this forum about 50 pages deep looking for information before I posed.
I am experiencing slower speeds that I should be using my new 10gb network setup, and it is isolated to Freenas so looking for help.

This is the most relevant environment information I can think of.

I have 2 10gb Hosts using Mellanox ConnectX-3 cards over Multi-Mode Fiber on a Aruba S2500 Layer 3 Switch.
Strictly Layer 2, no Layer 3 or VLANs no crazy MTU adjustments. Just a nice basic network setup.
ESXi just one vSwitch and all the VM's are in the same network and port-group.

Host #1 - Windows 10
Newest drivers on my Windows 10 machine from Mellanox
2600K CPU and 16GB RAM

Host #2 - ESXI 6.5
Has about 6 VM's on it, all of them with VMWare Tools and the VMX3 NIC.
Host has dual Xeon L5460's and 96GB of RAM.
FreeNAS is allocated 8vCPU's and 46GB of RAM, the high amount of CPU was for the Plex Jail
Had Freenas 11.1U6 but have upgraded now to 11.2
I also have Two Transmission Jails installed.
Jails were Warden, now migrated to IOCage using the migration script.

I see no contingency for resources on any machine during my testing.

Here is what I can say I know.

Testing using IPERF3 as it is a good way to see transparent network speeds without the overhead of storage subsystems or different tpc/ip protocols.

All of the VM's are using the same host/subnet/uplink

I can test from any of them and get about 9gb/s with Iperf3 using 5 parallel streams.

Freenas gives me about 3gb/s.
**Something is "UP" with jails. When I upgraded to 10gb it pretty much broke my Plex jail. It started to intermittently not want to respond to pings and got sluggish.
I could recreate or dissolve the issue easily with one of two solutions.

Change the ESXi uplink back to 1gb and the Plex jail started to behave again -or- inside of the jail, stop the Plex services and then the jail worked fine even on the 10gb uplink.

I have not noticed any issues with the Transmission jails, they work properly on 10gb.

**However** with all the testing I have been doing to try tunables, tweaks, reboots, etc. I did notice I get a pretty substantial boost in speed if I shut all the jails down. I go from about 3gb/s to 6gb/s if I turn the jails off and there is no way they are using 2 or 3gb/s of bandwidth. I see bursts up to 7gb/s but it seems like you get a high speed for a moment and it quickly drops down and you have to wait a while for some kind of buffer to refill before you can get those kinds of speeds again.

Non of the other machines have any issues and pull 9gb/s constantly.

Here is a video of the Plex Jail issue, and it helps show how my environment is setup: https://youtu.be/3WPrQUoMcIk

There has got to be some kind of magic setting here as the network has been verified 7 different ways from Sunday that all the equipment is functioning properly. I can isolate Freenas as the only slow part not the ESXi host or any other VM.

I guess I will just start with this and see where we go with it. Need more info or specific testing let me know and I'll try to oblige.

 

[Chris Moore]

Did you look through the guides that @Stux and @jgreco have done on FreeNAS inside a virtual machine? You might find some tips there:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.
https://forums.freenas.org/index.ph...ide-to-not-completely-losing-your-data.12714/

Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.
https://forums.freenas.org/index.ph...ide-to-not-completely-losing-your-data.12714/

Testing the benefits of SLOG
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561

SLOG benchmarking and finding the best SLOG
https://forums.freenas.org/index.ph...-and-finding-the-best-slog.63521/#post-454773

10 Gig Networking Primer
https://forums.freenas.org/index.php?resources/10-gig-networking-primer.42/

 

[ViciousXUSMC]

Yes I have seen all of those.
Most of them do not apply as it has to do with storage subsystems, and I am abstracting that layer from the equation.
I also am following all our best practices (HBA Passthru, etc)

 

[Spearfoot]

Since you've virtualized FreeNAS and are running on ESXi... why don't you install Plex and Transmission and such in ESXi-hosted virtual machines running the platform of choice for these apps (usually some flavor of Linux)? Then you can do away with jails and plug-ins and FreeNAS-hosted VMs, which seem to be a problem for many users.

I've found that using jumbo frames was the final piece of the puzzle I needed to get 10G working on my systems:

https://forums.freenas.org/index.ph...ceiving-network-data.53722/page-2#post-378891

I agree with the opinion espoused by some of our more experienced members that jumbo packets are a nuisance and ineffective on 1G networks. But 10G is a different story. YMMV.

Also, if you can do so without too much trouble, I suggest you revert to FreeNAS version 11.1-U6.3; it seems to be the most stable version -- 11.2 has caused a lot of problems for users.

And lastly... what motherboard are you using? The L5640 CPU is rated for 5.86 GT/s QPI, which should be enough 'ooomph' for your 10G NIC. But your motherboard selection affects your real-world transfer rate, and most importantly, you need to make sure you've plugged the NIC into a PCIe 3.0 x8 slot.

Good luck!

 

[jgreco]

So there is probably some combination of these things killing you (I'm guessing at least 3 out of 4):

1) You have tragic old Westmere CPU's, which has various implications for what's "well supported" under ESXi and to a much lesser extent FreeNAS.

2) You have way too many CPU's allocated to FreeNAS. Because of "but I need it for Plex", you're probably not interested in hearing what I'm saying, alas. Sorry, but there it is. There are complications with the ESXI scheduler and other factors here that are effectively unfixable. Do not allocate more vCPU to a VM than it actually has a demonstrated need for. Not my rule, it's a well-known best practice in the VMware world.

3) The use of jails probably involves enabling bridging and promiscuous mode on the interfaces (you can check this with ifconfig). With bridging/promiscuous, there can be a precipitous drop in network performance. This may couple neatly up with 2) by creating much heavier demand for run status in the ESXi scheduler because the VM has to process every packet it sees, which can present as an apparent and counterintuitive drop in performance.

4) If you absolutely must run this configuration, you can see if E1000 or VMXNETmumble works better.

I think the Westmere may preclude you ditching ESXi and running FreeNAS on the bare metal with bhyve to support your other VM's. That's too bad because that's really the best fix.

So the likely fix is probably to go back to something reasonable like two vCPU's for FreeNAS and create a separate VM for Plex, and mount the NAS via NFS. This reduces the charlie foxtrot effect from the ESXi scheduler trying to find a bunch of schedulable CPU's all at the same time. You might find that the Plex VM then becomes a problem though.

To the extent that 2) might be playing a role, you can possibly reserve cores for FreeNAS within ESXi to reduce the impact of scheduler issues. This makes them totally unavailable for other use though. A mostly-effective but wasteful fix.

Also see

https://blogs.vmware.com/vsphere/2014/02/overcommit-vcpupcpu-monster-vms.html

etc.

 

[jgreco]

I've found that using jumbo frames was the final piece of the puzzle I needed to get 10G working on my systems:

I agree with the opinion espoused by some of our more experienced members that jumbo packets are a nuisance and ineffective on 1G networks. But 10G is a different story. YMMV.

That doesn't make sense. If you're using the X520 then that has full offload.

Are you just using virtual ethernet interfaces to the ESXi vSwitch? Because if so yes that'd be a bottleneck especially on older hardware. You really need to set up SR-IOV and give your FreeNAS a VF. ESXi on older hardware isn't going to be able to drive 10G through a vSwitch easily.

 

[Spearfoot]

That doesn't make sense. If you're using the X520 then that has full offload.

Are you just using virtual ethernet interfaces to the ESXi vSwitch?

Yes...

Because if so yes that'd be a bottleneck especially on older hardware. You really need to set up SR-IOV and give your FreeNAS a VF. ESXi on older hardware isn't going to be able to drive 10G through a vSwitch easily.

Honestly, this never occurred to me. I'll give it a try! Thanks.

 

[Spearfoot]

Well, @jgreco -- turns out my free VMware license has restrictions: no SR-IOV support! Drat!

 

[ViciousXUSMC]

So there is probably some combination of these things killing you (I'm guessing at least 3 out of 4):

1) You have tragic old Westmere CPU's, which has various implications for what's "well supported" under ESXi and to a much lesser extent FreeNAS.

2) You have way too many CPU's allocated to FreeNAS. Because of "but I need it for Plex", you're probably not interested in hearing what I'm saying, alas. Sorry, but there it is. There are complications with the ESXI scheduler and other factors here that are effectively unfixable. Do not allocate more vCPU to a VM than it actually has a demonstrated need for. Not my rule, it's a well-known best practice in the VMware world.

3) The use of jails probably involves enabling bridging and promiscuous mode on the interfaces (you can check this with ifconfig). With bridging/promiscuous, there can be a precipitous drop in network performance. This may couple neatly up with 2) by creating much heavier demand for run status in the ESXi scheduler because the VM has to process every packet it sees, which can present as an apparent and counterintuitive drop in performance.

4) If you absolutely must run this configuration, you can see if E1000 or VMXNETmumble works better.

I think the Westmere may preclude you ditching ESXi and running FreeNAS on the bare metal with bhyve to support your other VM's. That's too bad because that's really the best fix.

So the likely fix is probably to go back to something reasonable like two vCPU's for FreeNAS and create a separate VM for Plex, and mount the NAS via NFS. This reduces the charlie foxtrot effect from the ESXi scheduler trying to find a bunch of schedulable CPU's all at the same time. You might find that the Plex VM then becomes a problem though.

To the extent that 2) might be playing a role, you can possibly reserve cores for FreeNAS within ESXi to reduce the impact of scheduler issues. This makes them totally unavailable for other use though. A mostly-effective but wasteful fix.

Also see

https://blogs.vmware.com/vsphere/2014/02/overcommit-vcpupcpu-monster-vms.html

etc.

1.) Westmere CPU is well supported under ESXi and mind you its running the other VM's with no issues. Could Freenas have an issue? Maybe but Freenas is not using the CPU its using ESXi and its resource scheduler so it really has no idea what is going on with the CPU as its abstracted from the guest OS.

2.) Your think of NUMA and how it is affected by L-CPU and P-CPU scheduling. My CPU Wait times are well within threadhold and ever since ESXi 6 they have a dynamic resource schedular to help off set over scheduling CPU it will not wait for all the requested processes it will dynamically adjust based on actual need. Also I have other VM's with the same amount of CPU doing even less and they have no issues here. I am quite interested in what you have to say, that is why I posted here but I am also a VMWare Certified Professional and do this for a living, so I know quite a lot about it. I have seen what under/over utlized CPU does and especially with a Paravirtulized network interface where CPU cycles = NIC IOPS and trust me that is not the issue here.

3.) Yes, this one holds true and that is why I mentioned those symptoms, the question is what to do about it? The jails have several different options for what interface to use, and so many settings that can be changed.

4.) I used to run baremetal and ran VM's but hated FreeNAS for resource alocation, there was limitations and for things such as my NVR software that needed a solid amount of cpu power the Bhyve environment was not able to cope, I tried all the tricks/hacks to manually give it the proper resources and it was a flop. Moving to ESXi was the best thing I ever did for my server, its doing more work, and handling everything like a champ.

I can stand up a FreeBSD VM and see what it does, this completely negates the freebsd being the issue and brings us back full circle with FreeNAS being the main issue.

For now I am pretty sure its the networking, especially the jails. I get CLOSE to the speeds I should see with the jails off, just not quite there.

You have to see that I did a full circle of test, each specifically targeted to narrow down the fault domain.
By testing various angles I have shown that the ESXi host is not the issue, that the switch, uplink, host, etc are not the issue as each of them have given me full speed with the same hardware/connection/software/etc.

I already narrowed it down to a FreeNAS issue before I posted. I have never seen so many "tweaks" for any system ever to make things work right, almost every 10gb thread in here has the tunables and such. Thats the kind of stuff that shows we know there are issues, and usually have to work around them.

I have done many more tests that just those I posted, but I left them out since I think I already provided enough information to focus on the issue rather than excuse it.

 

[Chris Moore]

I can stand up a FreeBSD VM and see what it does, this completely negates the freebsd being the issue and brings us back full circle with FreeNAS being the main issue.

For now I am pretty sure its the networking, especially the jails. I get CLOSE to the speeds I should see with the jails off, just not quite there.

I don't have the answer to this question but I can tell you this for certain, hoping that it helps someone figure out an answer.
My FreeNAS installation that I setup for testing, that has no jails, gets about twice the speed over my 10Gb network as my main FreeNAS syste that has my Plex jail, and they are both on bare metal.

 

[Spearfoot]

1.) Westmere CPU is well supported under ESXi and mind you its running the other VM's with no issues. Could Freenas have an issue? Maybe but Freenas is not using the CPU its using ESXi and its resource scheduler so it really has no idea what is going on with the CPU as its abstracted from the guest OS.

2.) Your think of NUMA and how it is affected by L-CPU and P-CPU scheduling. My CPU Wait times are well within threadhold and ever since ESXi 6 they have a dynamic resource schedular to help off set over scheduling CPU it will not wait for all the requested processes it will dynamically adjust based on actual need. Also I have other VM's with the same amount of CPU doing even less and they have no issues here. I am quite interested in what you have to say, that is why I posted here but I am also a VMWare Certified Professional and do this for a living, so I know quite a lot about it. I have seen what under/over utlized CPU does and especially with a Paravirtulized network interface where CPU cycles = NIC IOPS and trust me that is not the issue here.

3.) Yes, this one holds true and that is why I mentioned those symptoms, the question is what to do about it? The jails have several different options for what interface to use, and so many settings that can be changed.

4.) I used to run baremetal and ran VM's but hated FreeNAS for resource alocation, there was limitations and for things such as my NVR software that needed a solid amount of cpu power the Bhyve environment was not able to cope, I tried all the tricks/hacks to manually give it the proper resources and it was a flop. Moving to ESXi was the best thing I ever did for my server, its doing more work, and handling everything like a champ.

I can stand up a FreeBSD VM and see what it does, this completely negates the freebsd being the issue and brings us back full circle with FreeNAS being the main issue.

For now I am pretty sure its the networking, especially the jails. I get CLOSE to the speeds I should see with the jails off, just not quite there.

You have to see that I did a full circle of test, each specifically targeted to narrow down the fault domain.
By testing various angles I have shown that the ESXi host is not the issue, that the switch, uplink, host, etc are not the issue as each of them have given me full speed with the same hardware/connection/software/etc.

I already narrowed it down to a FreeNAS issue before I posted. I have never seen so many "tweaks" for any system ever to make things work right, almost every 10gb thread in here has the tunables and such. Thats the kind of stuff that shows we know there are issues, and usually have to work around them.

I have done many more tests that just those I posted, but I left them out since I think I already provided enough information to focus on the issue rather than excuse it.

FWIW, here are the tunables from my main FreeNAS-on-ESXi server, which gets ~9.6Gb/s running iperf. I put the original default values in the comment field:

 

[jgreco]

1.) Westmere CPU is well supported under ESXi and mind you its running the other VM's with no issues.

Then I suggest you quick run over to VMware HQ and tell them that they mistakenly deprecated your CPU in 6.7. Westmere was always a bit of a problem in a few different ways. VMware dumped Westmere, Nehalem, and a bunch of other older CPU's over CPU features that weren't present that they've been hacking around for years. Westmere's interesting because it's the point where they introduced VMX Unrestricted Guest and some other stuff that theoretically allowed some really great new virtualization capabilities, but in practice it was unreliable and buggy. We've been getting reports of issues with various Westmere platforms for *years* here when people try to virtualize FreeNAS on ESXi, problems that generally don't appear on Sandy or newer.

Could Freenas have an issue? Maybe but Freenas is not using the CPU its using ESXi and its resource scheduler so it really has no idea what is going on with the CPU as its abstracted from the guest OS.

That doesn't parse.

2.) Your think of NUMA and how it is affected by L-CPU and P-CPU scheduling. My CPU Wait times are well within threadhold and ever since ESXi 6 they have a dynamic resource schedular to help off set over scheduling CPU it will not wait for all the requested processes it will dynamically adjust based on actual need. Also I have other VM's with the same amount of CPU doing even less and they have no issues here. I am quite interested in what you have to say, that is why I posted here but I am also a VMWare Certified Professional and do this for a living, so I know quite a lot about it. I have seen what under/over utlized CPU does and especially with a Paravirtulized network interface where CPU cycles = NIC IOPS and trust me that is not the issue here.

3.) Yes, this one holds true and that is why I mentioned those symptoms, the question is what to do about it? The jails have several different options for what interface to use, and so many settings that can be changed.

4.) I used to run baremetal and ran VM's but hated FreeNAS for resource alocation, there was limitations and for things such as my NVR software that needed a solid amount of cpu power the Bhyve environment was not able to cope, I tried all the tricks/hacks to manually give it the proper resources and it was a flop. Moving to ESXi was the best thing I ever did for my server, its doing more work, and handling everything like a champ.

I can stand up a FreeBSD VM and see what it does, this completely negates the freebsd being the issue and brings us back full circle with FreeNAS being the main issue.

For now I am pretty sure its the networking, especially the jails. I get CLOSE to the speeds I should see with the jails off, just not quite there.

You have to see that I did a full circle of test, each specifically targeted to narrow down the fault domain.
By testing various angles I have shown that the ESXi host is not the issue, that the switch, uplink, host, etc are not the issue as each of them have given me full speed with the same hardware/connection/software/etc.

I already narrowed it down to a FreeNAS issue before I posted. I have never seen so many "tweaks" for any system ever to make things work right, almost every 10gb thread in here has the tunables and such. Thats the kind of stuff that shows we know there are issues, and usually have to work around them.

I have done many more tests that just those I posted, but I left them out since I think I already provided enough information to focus on the issue rather than excuse it.

Okay, have it your way. I'm not going to sit here and try to RE-explain the problem to a VMware Certified Professional. I won't make you listen to someone who literally wrote the guide on virtualizing FreeNAS and has been supporting it here in the forums for years. I won't make you listen to someone who's been working with the FreeBSD networking stack for a quarter of a century.

Good luck to you.

 

[kspare]

FWIW, here are the tunables from my main FreeNAS-on-ESXi server, which gets ~9.6Gb/s running iperf. I put the original default values in the comment field:
View attachment 27804

How much of this would change for 40gb? i've been using auto tune on my machines and we started to see that 10gb wasn't enough and upgaded to 40gb. when we do storage vmotions now we see the data transfer spike to 20-30gb and the data moves between 8-11gb. not bad!

 

[jgreco]

Well, @jgreco -- turns out my free VMware license has restrictions: no SR-IOV support! Drat!

View attachment 27800

Doh! I forgot about that.

If this is a home lab, you might want to consider whether paying ~$200 a year for VMUG Advantage and getting a legit fully licensed copy of vSphere for lab use would be an investment that would make sense.

 

[ViciousXUSMC]

Then I suggest you quick run over to VMware HQ and tell them that they mistakenly deprecated your CPU in 6.7. Westmere was always a bit of a problem in a few different ways. VMware dumped Westmere, Nehalem, and a bunch of other older CPU's over CPU features that weren't present that they've been hacking around for years. Westmere's interesting because it's the point where they introduced VMX Unrestricted Guest and some other stuff that theoretically allowed some really great new virtualization capabilities, but in practice it was unreliable and buggy. We've been getting reports of issues with various Westmere platforms for *years* here when people try to virtualize FreeNAS on ESXi, problems that generally don't appear on Sandy or newer.



That doesn't parse.



Okay, have it your way. I'm not going to sit here and try to RE-explain the problem to a VMware Certified Professional. I won't make you listen to someone who literally wrote the guide on virtualizing FreeNAS and has been supporting it here in the forums for years. I won't make you listen to someone who's been working with the FreeBSD networking stack for a quarter of a century.

Good luck to you.

Look no doubt I give you credit for knowing more about FreeBSD than me, but apparently you can't offer the same respect in reverse when it comes to virtualization. Yes I am VMWare certified, I also Cisco certified, and a ton of other crap that nobody cares about. I have been doing this for about 20 years for the government and I am not somebody who speaks before I read/test/verify and know what I am talking about. If your pointing blame at the hardware and its configuration the problem would carry across all other systems that share that same fault domain. However none of them have any issues.

CPU, Switch, Firewall, L2 Domain, ESXi, Server, RAM, ESXi cpu config, network config, port group, etc, etc all the same for all of my VM's and yet FreeNAS is the only VM with slow speeds. I checked CPU/RAM utilization already, I checked my cpu ready time, etc.

You seem to not know about how the cpu ready works and how that has been greatly changed in v6+ because you were very quick to directly blame that as an issue when it was not even possible given the configuration, and you know that my cpu is supported in 6.5 without any officially documented issues. Many enterprise run this setup with no issues on very large systems and deployments . I am not on 6.7 for that reason of removal of support (though it still works, its just not officially supported).

Also really lastly I must mention I didn't disrespect you, or have a negative tone and I do not mind a friendly and professional debate. I ask that we stick to just the facts so I would appreciate not needing to personalize how you feel about the situation or use passive aggressive sarcasm. Your a mod, and if you were not you probably wouldn't be allowed to talk to others that way in most professional environments, its just degrading and tends to take a conversation off track. I used to be a Mod on two different forums, I do not have time for that stuff anymore but I still understand the lay of the land and hope you can accept my direct addressing of the issue without the need to string it out further.

 

[jgreco]

My FreeNAS installation that I setup for testing, that has no jails, gets about twice the speed over my 10Gb network as my main FreeNAS syste that has my Plex jail, and they are both on bare metal.

You're very likely hitting bridging and promiscuous mode issues. When you do jails the way that FreeNAS does jails (or at least has done in the past), it turns on promiscuous mode, which forces the host to process all traffic received on the local network interface. This has the upside of allowing jails and virtual machines a much larger degree of flexibility in their network configurations. You can almost forget that it's a jail. Unfortunately, it means that there's a lot more packets to process. This is item 3) I described to our VCP gentleman above.

One of the things about networking is that once you get up to a certain speed, network performance is very sensitive to the responsiveness of the TCP stack of a host. A modest delay in sending packets in either direction can cause a significant fall-off in performance. There are a few things to combat this, including increasing the size of the sendbuf/recvbuf for TCP sockets, and switching to a different congestion control algorithm (as newreno was really designed for lower speed networks). However, you reach a point where the existence of extra packets in-between your desired packets, and the CPU overhead to process them, conspire to negatively impact performance. Observationally, this has always seemed to be of worse effect on older pre-Sandy Bridge hardware. Some users here have fought this by switching to jumbo, which covers up some of this by reducing the number of packets that must be processed (or looking at it the other way, increases the amount of useful data per desired packet). That's more of a kludge fix but it leverages the variables in play here and swings it more heavily in the end user's favor. (I still generally disapprove.)

Aside: This gets more complex when you get into passing data via a VMware vSwitch. You're introducing extra latency, which impacts performance negatively. Which is why Intel got so heavily into research on virtual functions and then started dumping this into their product offerings around a decade ago. The inclusion of a vSwitch is generally going to represent a performance loss. The vSwitch is extremely efficient, but not negligible unfortunately.

But you can also make this worse: Turn on promiscuous mode. Because promiscuous mode requires the guest OS to accept and process the packets, and at 10Gbps, if you assume wire speed in one direction that's 800Kpps, so with reply traffic it's much greater, so let's assume an optimistic 1Mpps ...

Well you know what I don't have time for this, this morning. If you want to get a better idea of the complexities of getting high performance networking running, I'd suggest looking at the numerous posts Cloudflare has written

https://blog.cloudflare.com/how-to-achieve-low-latency/
https://blog.cloudflare.com/how-to-receive-a-million-packets/

which may help give you an idea why this isn't really as simple or straightforward as we might wish.

 

[jgreco]

Look no doubt I give you credit for knowing more about FreeBSD than me, but apparently you can't offer the same respect in reverse when it comes to virtualization. Yes I am VMWare certified, I also Cisco certified, and a ton of other crap that nobody cares about. I have been doing this for about 20 years for the government and I am not somebody who speaks before I read/test/verify and know what I am talking about. If your pointing blame at the hardware and its configuration the problem would carry across all other systems that share that same fault domain. However none of them have any issues.

CPU, Switch, Firewall, L2 Domain, ESXi, Server, RAM, ESXi cpu config, network config, port group, etc, etc all the same for all of my VM's and yet FreeNAS is the only VM with slow speeds. I checked CPU/RAM utilization already, I checked my cpu ready time, etc.

You seem to not know about how the cpu ready works and how that has been greatly changed in v6+ because you were very quick to directly blame that as an issue when it was not even possible given the configuration, and you know that my cpu is supported in 6.5 without any officially documented issues. Many enterprise run this setup with no issues on very large systems and deployments . I am not on 6.7 for that reason of removal of support (though it still works, its just not officially supported).

Also really lastly I must mention I didn't disrespect you, or have a negative tone and I do not mind a friendly and professional debate. I ask that we stick to just the facts so I would appreciate not needing to personalize how you feel about the situation or use passive aggressive sarcasm. Your a mod, and if you were not you probably wouldn't be allowed to talk to others that way in most professional environments, its just degrading and tends to take a conversation off track. I used to be a Mod on two different forums, I do not have time for that stuff any more but I still understand the lay of the land and hope you can accept my direct addressing of the issue without the need to string it out further.

I tried to give you a big shove in the right direction into an area that requires a huge amount of experimentation and doesn't usually have easy answers. You didn't like it. I'm fine with that, but saying stuff like I don't know how the VMware scheduler works doesn't make that so, and isn't going to cause me to reveal some different secret answer that is more to your liking.

You don't believe that this is generally better on modern processors? *Try it*.

You don't believe that overallocation of vCPU's is an issue? I'll note that VMware disagrees with you. *Try it*.

You don't believe that bridging and promiscuous mode has a significant impact? *Try it*.

These things are all real issues. You can dismiss them offhand if you wish. Again, good luck to you.

 

[ViciousXUSMC]

You're very likely hitting bridging and promiscuous mode issues. When you do jails the way that FreeNAS does jails (or at least has done in the past), it turns on promiscuous mode, which forces the host to process all traffic received on the local network interface. This has the upside of allowing jails and virtual machines a much larger degree of flexibility in their network configurations. You can almost forget that it's a jail. Unfortunately, it means that there's a lot more packets to process. This is item 3) I described to our VCP gentleman above.

One of the things about networking is that once you get up to a certain speed, network performance is very sensitive to the responsiveness of the TCP stack of a host. A modest delay in sending packets in either direction can cause a significant fall-off in performance. There are a few things to combat this, including increasing the size of the sendbuf/recvbuf for TCP sockets, and switching to a different congestion control algorithm (as newreno was really designed for lower speed networks). However, you reach a point where the existence of extra packets in-between your desired packets, and the CPU overhead to process them, conspire to negatively impact performance. Observationally, this has always seemed to be of worse effect on older pre-Sandy Bridge hardware. Some users here have fought this by switching to jumbo, which covers up some of this by reducing the number of packets that must be processed (or looking at it the other way, increases the amount of useful data per desired packet). That's more of a kludge fix but it leverages the variables in play here and swings it more heavily in the end user's favor. (I still generally disapprove.)

Aside: This gets more complex when you get into passing data via a VMware vSwitch. You're introducing extra latency, which impacts performance negatively. Which is why Intel got so heavily into research on virtual functions and then started dumping this into their product offerings around a decade ago. The inclusion of a vSwitch is generally going to represent a performance loss. The vSwitch is extremely efficient, but not negligible unfortunately.

But you can also make this worse: Turn on promiscuous mode. Because promiscuous mode requires the guest OS to accept and process the packets, and at 10Gbps, if you assume wire speed in one direction that's 800Kpps, so with reply traffic it's much greater, so let's assume an optimistic 1Mpps ...

Well you know what I don't have time for this, this morning. If you want to get a better idea of the complexities of getting high performance networking running, I'd suggest looking at the numerous posts Cloudflare has written

https://blog.cloudflare.com/how-to-achieve-low-latency/
https://blog.cloudflare.com/how-to-receive-a-million-packets/

which may help give you an idea why this isn't really as simple or straightforward as we might wish.

I may go and setup Layer 3 switching soon so I can enable large MTU, It's more of a for the fun than a need. I do have promiscuous mode enabled to allow the multiple IP addresses allowed from one interface. I think what I will try to do is add additional virtual nics to the FreeNAS VM and then assign these directly to the jails so that promiscuous mode is no longer needed.

I am not sure the exact configuration yet until I sit and think on it, but that should be possible.

I have a feeling this will help the issue quite a bit.
But to the core of the issue now, its configuration and tweak needed not the hardware.
These are the kind of insightful answers I really wanted

thank you,

 

[Chris Moore]

Well you know what I don't have time for this, this morning. If you want to get a better idea of the complexities of getting high performance networking running, I'd suggest looking at the numerous posts Cloudflare has written

I sure appreceate your time. I have been fighting Windows permissions today and have spent more time reading than doing.

 

[Chris Moore]

I think what I will try to do is add additional virtual nics to the FreeNAS VM and then assign these directly to the jails so that promiscuous mode is no longer needed.

With iocage it may be different, but I tried physically separate interfaces under warden jails and it created some problems for me.