iSCSI multipath advise

[Fred974]

Hi,

I am building an xcp-ng cluster with 3 nodes and I will be using FreeNAS for the shared storage over iSCSI.
In FreeNAS, I created 2x 10GBGe Interfaces and set the MTU to 9000.
I am now trying to create the portal and I am not sure if I should click on the Add extra Portal IP option or configure multiple group IDs instead.
Which will give me better result/performance?

Thank you in advance for the advise

PS: is multipath design for 1Gb NIC? Will you do multipath on 10GbE?

 

[RickH]

For iSCSI multipathing the recommended way is to add an extra Portal IP. While it should be possible to add another portal group and then add the second portal to the target this creates a more complex config - neither way is going to alter performance in the least.

iSCSI multipathing definitely works over 10GbE connections - I've seen read/write performance over 1600 MiB/sec from a Windows 2012 host using the software iscsi initiator to an all flash FreeNAS server. Even if your disk array won't go that high, I still like multipathing in a production environment as it adds another layer of redundancy.

Remember that with multipathing, the initiator determines the load-balancing protocol - I don't have experience with xcp-ng; however I run quite a few ESXi hosts that use FreeNAS for the datastores. In ESXi the default behavior is to use a single path in an active/failover config - but you can configure it to use round-robin (and can further tune it to change paths every single IOP or every 10, 100, etc....). For best performance with 3 hosts (assuming your 10GbE adaptors are only providing the storage connection) you'll want to go with a round-robin config and set it up to swap paths every IOP.

Unless your drive array is something pretty exotic, you're wasting your time with the Jumbo frame MTU setting. In my experience getting Jumbo frames to work correctly across different OS's, different NIC vendors, and switches is seldom worth the payoff in a 1GbE environment, much less in a 10GbE one. My 1600 MiB/sec performance benchmark was done without the use of Jumbo Frames...

 

[Fred974]

@RickH
Thank you for your reply and sorry for the delay in replying :(
The reason I was asking is because in this FreeNAS iSCSI Configuration for MPIO they mention the following:

A quick look here at a step you don’t want to do. When I was initially configuring MPIO, I used the Add extra Portal IP option. However, this will place the two IP addresses on the same portal ID and won’t answer properly to the iSCSI MPIO requests. You want to configure multiple group IDs instead.

What is your tough on this if you don't mind sharing?

Thank you

 

[RickH]

@RickH
Thank you for your reply and sorry for the delay in replying :(
The reason I was asking is because in this FreeNAS iSCSI Configuration for MPIO they mention the following:


What is your tough on this if you don't mind sharing?

Thank you

That looks like I pretty good tutorial - but I'm not exactly clear on where he's getting that information on MPIO. It's always been my understanding that the initiator is responsible for handling all MPIO or Multipathing algorithms. I have 6 FreeNAS servers that server as back-end storage for 14 ESXi hosts that are all configured with a single portal with multiple IP's and iSCSI Multipathing works perfectly. I have also previously configured connections to Windows Server 2012 R2 machines using MPIO with the same FreeNAS configuration. In all cases the multipathing has worked fine...

That being said, there may definitely be a scenario that I'm unaware of where it would be preferable to have 2 separate portals. It's fairly easy to configure it this way, and if it makes you more comfortable with the set-up then go for it.

 

[Fred974]

@RickH At this stage I consider myself a newbie so I'll stay with the recommended way of doing it.

Thank you

 

[tvtue]

For iSCSI multipathing the recommended way is to add an extra Portal IP. While it should be possible to add another portal group and then add the second portal to the target this creates a more complex config - neither way is going to alter performance in the least.

Hi, thank you for your post. I know what an initiator and a target is, but find these terms "extra portal ip" and "portal group" puzzling. Do you have or know of a good doc where this is explained for dummies?

Cheers
Timo

 

[RickH]

Hi, thank you for your post. I know what an initiator and a target is, but find these terms "extra portal IP" and "portal group" puzzling. Do you have or know of a good doc where this is explained for dummies?

Cheers
Timo

The FreeNAS guide walks you through the basic ISCSI configuration and goes over exactly what these terms are and where they can be set: https://www.ixsystems.com/documentation/freenas/11.3-RELEASE/sharing.html#block-iscsi

To directly answer your terminology questions:

• Extra Portal IP - this refers to the IP addresses an ISCSI portal is listening on. It can be found/edited by going to Sharing -> Block Shares (ISCSI) -> Portals. The 'Listen' column shows the configured IP addresses for a particular portal.
• Portal Group - this is the numeric ID number that FreeNAS assigns each ISCSI portal that is created (starts at 1 and increments up with each new portal). This is simply the internal ID that is used when configuring the ISCSI Target

The guide is a little light on ISCSI multipathing (MPIO) so I'll go over the basics from the FreeNAS side of things.

Unlike a lot of other multipathing or load balancing protocols - ISCSI multipathing is completely controlled by the initiator. That means that whatever load-balancing/failover protocols are used are determined by the system connecting to the block storage - all we need to provide from FreeNAS is 2 (or more) paths to the same Target/Extent.

In order to accomplish this you'll need to configure the following from within FreeNAS:

• Networking - you'll need at least 2 IP addresses assigned to your FreeNAS system. Ideally, your ISCSI traffic should be on separate physical network adapters connected to dedicated storage network switches, but I understand that this isn't always possible. The ISCSI configuration doesn't check the validity of your networking configuration so it will allow you to assign 2 IP addresses that are backed by the same physical network interface (IE: IPs on 2 different VLANs on the same adapter) but this completely eliminates the entire point of multipathing.
  
  Networking best-practices for ISCSI
  • Use 2 or more separate physical adapters - assign IP's on different subnets that are dedicated to your storage environment. If possible, connect these 2 adapters to separate physical switches.
  • Avoid link aggregations - adding additional load-balancing in front of the ISCSI multipathing can decrease the effectiveness of the initiator's load-balancing protocols. Again, FreeNAS will allow you to assign LAGG-backed IP addresses to an ISCSI portal, and it will work, but the configuration is not ideal as the initiator no longer has full control over where the traffic is flowing.
    
• ISCSI Config - for the remainder of this, I'm assuming you already have a functioning ISCSI setup (if not, there are plenty of other tutorials on how to set this up) - this will only focus on the changes necessary to enable MPIO.
  
  We need to create 2 paths to the same target/extent - there are 2 different ways to do this from within FreeNAS:
  
  • Adding an additional IP to an existing portal - this is my preferred config and has worked well in a FreeNAS/ESXi environment for the past 4 years. Assuming you already have a functioning ISCSI environment, this method is much simpler as any existing portal Discovery and CHAP auth rules are applied to the new portal IP.
    • Go to your ISCSI portals (Sharing -> Block Shares (ISCSI) -> Portals), you'll see the list of configured portals on the table - the 'Listen' column shows the IP address that the portal is listening on. Click the edit button and add the appropriate ip address(es) to the portal.
  • Create a new portal and add it to the Target- the poster above indicated that he had read somewhere that this was the preferred method, but I haven't seen or found any evidence to support that. And my own experience with 16 ESXi hosts connected to 7 FreeNAS servers using the method above is proof enough for me that the additional IP method works perfectly fine - however, if you're interested, here it is:
    • Go to your ISCSI portals (Sharing -> Block Shares (ISCSI) -> Portals) - this time we're going to add an additional portal. Click the add button, select the new portal IP and assign any applicable Discovery rules. Save the portal and make a note of the Portal Group ID's (numeric value on the left of the table view) for the portals that you want to use
    • Go to your ISCSI Targets (Sharing -> Block Shares (ISCSI) -> Targets) and click the edit button for the Target you want to modify. On the edit screen, click the 'Add Extra ISCSI Group' button at the bottom of the page and select the newly created portal - add any applicable CHAP rules and save your changes.

That's it - at least from the FreeNAS side - the configuration tends to be much more complex from the initiator side. The concept is straightforward - on your initiator assign 2 IP's on the same subnets as the FreeNAS ISCSI portal IP's and create 2 connections to the FreeNAS ISCSI target (and associated extents). Then assign whatever Load-Balancing protocol you need (Fail-over, Round-Robin, etc...)

I have personally used FreeNAS as ISCSI back-end storage for Windows servers and VMware ESXi and have always used the additional portal IP method to make the multipath connections and have not had any stability or performance issues since the FreeNAS 9.1 days (that was a separate ISCSI issue not related to multipathing)

Good Luck!

 

[tvtue]

Dear RickH, thank you very much for that great elaborate answer. I think I will try out both methods and see which one fits me better.
Is your preferred method "Adding an additional IP to an existing portal" providing the initiator with all the ip addresses which are on the portal within one discovery?
Have a nice day
timo

 

[RickH]

Dear RickH, thank you very much for that great elaborate answer. I think I will try out both methods and see which one fits me better.
Is your preferred method "Adding an additional IP to an existing portal" providing the initiator with all the IP addresses which are on the portal within one discovery?
Have a nice day
timo

Yes, all the available IP/Target combos will be presented to the initiator during discovery. That being said, I don't actually use autodiscovery in my current environment (it really slows down the boot time of ESXi when it has to scan multiple servers for available ISCSI targets) - I specify static ip/target names in my ESXi configs. But discovery should work if you choose to use it.

 

[tvtue]

Dear RickH,
thank you for your reply. I've tried both methods and I get both ip addresses / portals with both methods. Thats really fine.

This is different to our Infortrend systems. You have to do a discovery against the two portals to get the targets. We are using rhv-h (Red Hat Virtualization hypervisors) and the multipath stuff is beeing managed from a web gui.

Now, what I get in multipath -ll on the initiator side is this:

Code:

36589cfc000000ca24bbef5c2c471a35b dm-21 FreeNAS ,iSCSI Disk
size=25T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 12:0:0:0 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 11:0:0:0 sdd 8:48 active ready running

AFAIK, thats a active/standby configuration. What I would like to have is a active/active setup. Maybe like so:

Code:

36589cfc000000ca24bbef5c2c471a35b dm-21 FreeNAS ,iSCSI Disk
size=25T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 12:0:0:0 sdc 8:32 active ready running
  `- 11:0:0:0 sdd 8:48 active ready running

I am not sure how to get to that and if that is beyond FreeNAS scope here. But I know from our Infortrend setups that active/active setups need to come from the same controller (in dual controller systems) or you have to do a special symmetric drive setup within the Infortrend management gui. But here with my FreeNAS system I only have a dual port 10G nic, so everything is on the same "controller" in some point of view.

However, I will continue to play around with iscsi settings, maybe adding the same extent to two different targets and each target to a own portal gives something else ... Also, I should search the forum specifically for that.

Thanks and have a nice day
timo

 

[tvtue]

Ok, path_grouping_policy multibus in respective device section of the multipath.conf is what I needed.
Now I am getting two active paths.

Code:

36589cfc000000ca24bbef5c2c471a35b dm-13 FreeNAS ,iSCSI Disk
size=25T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 2:0:0:0 sdc 8:32 active ready running
  `- 3:0:0:0 sdd 8:48 active ready running

Great! :)

 

[MasterYous]

Unless your drive array is something pretty exotic, you're wasting your time with the Jumbo frame MTU setting. In my experience getting Jumbo frames to work correctly across different OS's, different NIC vendors, and switches is seldom worth the payoff in a 1GbE environment, much less in a 10GbE one. My 1600 MiB/sec performance benchmark was done without the use of Jumbo Frames...

While I agree with everything else that has been recommended, let me just put in that in my experience with VMware and FreeNAS as an iSCSI SAN, jumbo frames are /absolutely/ worth it, if you’ve got a dedicated physical network or are isolating traffic via VLANs. And it’s especially beneficial if you are bandwidth-constrained.

It’s correct that the setup can be tough to get right and tough to validate.

You set up jumbo frames in the following 4 places. I’ll go inside-out:
1) on the freenas network interface settings - each interface. Use the max of 9000
2) on the physical switch connecting freenas to your VMware iSCSI network. Make sure the switch is set to pass the max size frame, which should be 9216, for each iSCSI port
3) on the distributed switch or vSwitch’s configuration settings. Set MTU to 9000.
4) on each VMkernel attached to the iSCSI hba, inside the host network config in VMware. Set MTU to 9000.

To test it, you will want to send pings to the FreeNAS iSCSI portal IPs, over the iSCSI network. Unfortunately, ESXi’s ping is not up to the task because you need the ping options to NOT fragment as well as to set packet size. Do this:
1. It’s different depending on your OS, so it’s time to make a VM of your favorite Mac, Linux, or Windows flavor.
2. attach it temporarily to the iSCSI network using ESXi’s web client or using vCenter. Assign it an IP address that does not conflict with FreeNAS or the VMware iSCSI initiators that is one one of your iSCSI subnets. I use /29 networks for my iSCSI networks because they have room for multiple ESXi hosts, VMs, and a FreeNAS portal IP per subnet.
3. Log into the VM and ping away. Follow this article to know which ping to use with which OS. https://blah.cloud/hardware/test-jumbo-frames-working/
4. If the ping works then you’re all set. Otherwise check your settings as something will be wrong in one of the 4 places above.
5. Repeat steps 2-4 for each unique iSCSI subnet
6. you can delete or shut off this VM when you’re done testing.

I have set up jumbo frames at MTU 9000 and on a MPIO’d 2x10 GbE network, it boosts my performance by 10-20% depending on the type of traffic. It’s most evident when synchronous writes are present, like vCenter operations, vCenter or other VM backups, or when a VM is mounting read-write NFS file systems on one or more other VMs. However it will boost performance on regular I/O between your VMs and SAN by at least a bit, and esp if you are maxing out your Ethernet bandwidth over iSCSI.

Warning: please /do not/ use jumbo frames if you cannot isolate iSCSI traffic to its own layer 2 or layer 3 domains (preferably with separate physical switches but VLANs all on the same switch works just as well).