How do you set line rate for 1Gb/10Gb SFP+ optics? Diagnosing SFP+ issues?

victorhooi · Apr 21, 2019

Hi,

I'm setting up a new FreeNAS system with 10Gbase-LR for network connectivity.

I'm using an Intel dual-rate SFP+ optic that does both 1Gb and 10Gb.

On one machine (using a Intel X520-DA2 card) it's coming up as a 1GB link. My question is - how do you configure this under FreeNAS as 10Gb line rate? (It's odd because normally it should default to 10Gb)

On the other machine, the link isn't coming up, and I'm trying to figure out why. In this case the SFP+ port is inbuilt to the motherboard.

It might be an optic compatibility thing - however, I'm not very familiar with FreeNAS/FreeBSD.

Are there specific commands I can use to diagnose why a SFP+ link isn't coming up? How do you query the optics/DOM?

Thanks,
Victor

SweetAndLow · Apr 22, 2019

Messages log file should give you a dump of info when you plug it in. You can also look at if config. What freenas version? What is the other is? Transceivers also are manufacture special and have to match correctly. So you need Intel transceivers, anything else could cause incompatibilities.

victorhooi · Apr 22, 2019

Both machines are running FreeNAS 11.2-U3.

OK, for the machine that isn't brining up the link at all - I checked /var/log/messages, and the output of dmesg - nothing is outputted when I remove/re-insert the SFP+ module.

The card is the in-built Intel PRO 10GbE card, and the SFP+ modules are dual-rate Intel optics, so there shouldn't be incompatibilities there.

There is one small odd thing - when I removed the SFP+ light, the little green arrow next to the port started blinking, but when I re-insert the SFP+ module it stops.

However, the interface still doesn't come up. I even tried swapping the cables with the other machine, still nothing. I'm pretty stumped as to why.

For the machine that is doing 1Gb instead of 10Gb, here is the output of ifconfig:

Code:

root@freenas[~]# ifconfig ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether b4:96:91:21:68:80
    hwaddr b4:96:91:21:68:80
    inet 10.5.1.31 netmask 0xffffff00 broadcast 10.5.1.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseLX <full-duplex>)
    status: active

and pciconf -lv:

Code:

root@freenas[~]# pciconf -lv
...
ix0@pci0:2:0:0:    class=0x020000 card=0x7b118086 chip=0x154d8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet 10G 2P X520 Adapter'
    class      = network
    subclass   = ethernet

SweetAndLow · Apr 22, 2019

Are these direct connected or is there a switch? Your sure it's all sfp+ and not just sfp?

Mlovelace · Apr 22, 2019

victorhooi said:
Both machines are running FreeNAS 11.2-U3.

OK, for the machine that isn't brining up the link at all - I checked /var/log/messages, and the output of dmesg - nothing is outputted when I remove/re-insert the SFP+ module.

The card is the in-built Intel PRO 10GbE card, and the SFP+ modules are dual-rate Intel optics, so there shouldn't be incompatibilities there.

There is one small odd thing - when I removed the SFP+ light, the little green arrow next to the port started blinking, but when I re-insert the SFP+ module it stops.

However, the interface still doesn't come up. I even tried swapping the cables with the other machine, still nothing. I'm pretty stumped as to why.

For the machine that is doing 1Gb instead of 10Gb, here is the output of ifconfig:

Code:
root@freenas[~]# ifconfig ix0 ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether b4:96:91:21:68:80 hwaddr b4:96:91:21:68:80 inet 10.5.1.31 netmask 0xffffff00 broadcast 10.5.1.255 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseLX <full-duplex>) status: active

and pciconf -lv:

Code:
root@freenas[~]# pciconf -lv ... ix0@pci0:2:0:0: class=0x020000 card=0x7b118086 chip=0x154d8086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet 10G 2P X520 Adapter' class = network subclass = ethernet

media: Ethernet autoselect (1000baseLX <full-duplex>)

You have a 1G long range optic installed, of course it's not going to link at 10Gbe...

You'll need a compatible 10Gbe transceiver for the X520-DA2.

jgreco · Apr 22, 2019

Mlovelace said:
media: Ethernet autoselect (1000baseLX <full-duplex>)

You have a 1G long range optic installed, of course it's not going to link at 10Gbe...

You'll need a compatible 10Gbe transceiver for the X520-DA2.

I think he's talking about a dual speed optic.

https://www.newegg.com/Product/Product.aspx?Item=N82E16833106066

The first step would be to loop it and see what happens, or just plug in two optics and see if they link happily or not. I'd expect it's quite possible you have to set a desired link speed manually.

jgreco · Apr 22, 2019

victorhooi said:
The card is the in-built Intel PRO 10GbE card, and the SFP+ modules are dual-rate Intel optics, so there shouldn't be incompatibilities there.

There is one small odd thing - when I removed the SFP+ light, the little green arrow next to the port started blinking, but when I re-insert the SFP+ module it stops.

Are these both supported by the ix driver?

In any case, run "ifconfig -m ix0" (or other appropriate interface) to see what supported media types are. If need be you can manually set them as additional arguments to ifconfig.

victorhooi · Apr 22, 2019

On the machine that was negotiating at 1Gb - I tried checking the media types:

Code:

root@freenas[~]# ifconfig -m ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    capabilities=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether b4:96:91:21:68:80
    hwaddr b4:96:91:21:68:80
    inet 10.5.1.31 netmask 0xffffff00 broadcast 10.5.1.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseLX <full-duplex>)
    status: active
    supported media:
        media autoselect
        media 1000baseLX
        media 10Gbase-LR

I tried setting it as so:

Code:

root@freenas[~]# ifconfig ix0 media 10Gbase-LR

Aftewrads, I ran ifconfig again - the media line shows both 10Gbase-LR and 1000baseLX?

Code:

root@freenas[~]# ifconfig ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether b4:96:91:21:68:80
    hwaddr b4:96:91:21:68:80
    inet 10.5.1.31 netmask 0xffffff00 broadcast 10.5.1.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet 10Gbase-LR (1000baseLX <full-duplex>)
    status: active

Even after I bought interface down/up, it still went back to 1000baseLX.

However, on the switch side (Ruckus ICX 7150) - I realised for some reason it was defaulting to 1Gb. So I set it there:

Code:

SSH@ICX7150-C12 Router#conf t
SSH@ICX7150-C12 Router(config)#interface ethernet 1/3/1
SSH@ICX7150-C12 Router(config-if-e10000-1/3/1)#speed-duplex 10g-full
SSH@ICX7150-C12 Router(config-if-e10000-1/3/1)#write memory

And now, on the FreeNAS host:

Code:

root@freenas[~]# ifconfig ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether b4:96:91:21:68:80
    hwaddr b4:96:91:21:68:80
    inet 10.5.1.31 netmask 0xffffff00 broadcast 10.5.1.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-LR <full-duplex,rxpause,txpause>)
    status: active

It now shows 10Gbase-LR under media. Yay! (I think).

Not sure why the switch defaults SFP+ ports to 1Gb.

On the FreeNAS host with the inbuilt SFP+ ports, I'm still not getting anything on any of the ports, and nothing in dmesg or /var/log/messages.

Any advice on what else to check there?

victorhooi · Apr 22, 2019

OK, after setting the port on the switch side to 10Gb, I rebooted the FreeNAS host and it now is coming up on the network. So I guess on the FreeNAS side, this motherboard/system didn't like negotiating at 1Gb...lol. (Same optics in both FreeNAS machines, and in the switch - Intel dual-rate ones).

But then the network seems to go down again - couldn't ping the gateway from the machine. I had to use ifconfig to bring down the interface, and bring it back up to get it working again.

It's very odd. I did check /var/log/messages, nothing obvious I can see there:

Code:

Apr 22 11:43:40 freenas kernel: ix3: link state changed to UP
Apr 22 11:43:40 freenas kernel: ix3: link state changed to UP
Apr 22 11:43:40 freenas ums0 on uhub1
Apr 22 11:43:40 freenas ums0: <NXP SEMICOND USB KM, class 0/0, rev 1.10/1.00, addr 2> on usbus0
Apr 22 11:43:40 freenas ums0: 3 buttons and [Z] coordinates ID=0
Apr 22 11:43:40 freenas ums1 on uhub1
Apr 22 11:43:40 freenas ums1: <NXP SEMICOND USB KM, class 0/0, rev 1.10/1.00, addr 2> on usbus0
Apr 22 11:43:40 freenas ums1: 5 buttons and [XYZ] coordinates ID=0
Apr 22 11:43:40 freenas ums2 on uhub3
Apr 22 11:43:40 freenas ums2: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 5> on usbus0
Apr 22 11:43:40 freenas ums2: 3 buttons and [Z] coordinates ID=0
Apr 22 11:43:42 freenas ntpd[2693]: ntpd 4.2.8p11-a (1): Starting
Apr 22 12:02:37 freenas dhclient[1658]: connection closed
Apr 22 12:02:37 freenas dhclient[1658]: exiting.
Apr 22 12:02:37 freenas kernel: ix3: link state changed to DOWN
Apr 22 12:02:37 freenas kernel: ix3: link state changed to DOWN
Apr 22 12:02:42 freenas kernel: ix3: link state changed to UP
Apr 22 12:02:42 freenas kernel: ix3: link state changed to UP
Apr 22 12:02:49 freenas dhclient: New IP Address (ix3): 10.5.1.39
Apr 22 12:02:49 freenas dhclient: New Subnet Mask (ix3): 255.255.255.0
Apr 22 12:02:49 freenas dhclient: New Broadcast Address (ix3): 10.5.1.255
Apr 22 12:02:49 freenas dhclient: New Routers (ix3): 10.5.1.1

I guess I'll keep an eye on things?

jgreco · Apr 22, 2019

victorhooi said:

No, it's telling you that it's configured to try for 10Gbase-LR. On many/most interfaces, that will be the forced setting, but this is really just signalling the driver what you have set. The thing inside the parens is what the driver has determined are the actual applied settings. Normally this is just a way for the driver to report back when you have autoselect enabled

media: Ethernet autoselect (1000baseLX <full-duplex>)

but occasionally it shows other interesting and/or unexpected things.

Not sure why the switch defaults SFP+ ports to 1Gb.

Media negotiation failure of some sort.

On the FreeNAS host with the inbuilt SFP+ ports, I'm still not getting anything on any of the ports, and nothing in dmesg or /var/log/messages.

Any advice on what else to check there?

Not offhand. If they're supported, it'll be the same sort of thing.

victorhooi said:
OK, after setting the port on the switch side to 10Gb, I rebooted the FreeNAS host and it now is coming up on the network. So I guess on the FreeNAS side, this motherboard/system didn't like negotiating at 1Gb...lol. (Same optics in both FreeNAS machines, and in the switch - Intel dual-rate ones).

But then the network seems to go down again - couldn't ping the gateway from the machine. I had to use ifconfig to bring down the interface, and bring it back up to get it working again.

If that's a built-in chipset like the X552/X557, I've seen those be a bit funky. I strongly suggest trying the standard LR optic instead of trying the mixed speed one. Absolutely no promises there, this is just my frequently-correct intuition that it's more likely to work without the drama. Because so much of 1G/10G ethernet is no longer really doing autonegotiation in adverse conditions, I see a large amount of room for bugs when ambiguity enters the picture.

Mlovelace · Apr 22, 2019

victorhooi said:
Not sure why the switch defaults SFP+ ports to 1Gb.

I suspect this is a side effect of the dual-rate optics. Most switches do not support auto-negotiation for 10Gbe, but do for 1Gbe. So if the ports were set for auto, they probably defaulted to 1Gbe as the optics support it.

blanchet · Apr 22, 2019

According to the specs the Intel X520-DA2 supports only SFP+ Direct Attached Twin Axial Cabling up to 10m.
The specs says that it supports both 1 GbE and 10 GbE.

jgreco · Apr 22, 2019

blanchet said:
According to the specs the Intel X520-DA2 supports only SFP+ Direct Attached Twin Axial Cabling up to 10m.

The specs are full of CRAP.

victorhooi · Apr 22, 2019

OK, so the first machine (that previously negotiated at 1Gb) is working, after manually setting the switch port to 10Gb.

The second FreeNAS machine which periodically drops off the network is still dropping off. The system is based around the SuperMicro A2SDi-H-TP4F, using the Intel C3000's inbuilt network controller.

Also, for some reason, the IPMI is now timing out when I try to access it as well (although maybe that's a separate thing).

jgreco · Apr 23, 2019

Got nothing for you there. I'm not a super-big fan of the all-in-one boards like that although we do have some of the X10SDV-something's here.

Important Announcement for the TrueNAS Community.

How do you set line rate for 1Gb/10Gb SFP+ optics? Diagnosing SFP+ issues?

victorhooi

Contributor

SweetAndLow

Sweet'NASty

victorhooi

Contributor

SweetAndLow

Sweet'NASty

Mlovelace

Guru

jgreco

Resident Grinch

jgreco

Resident Grinch

victorhooi

Contributor

victorhooi

Contributor

jgreco

Resident Grinch

Mlovelace

Guru

blanchet

Guru

jgreco

Resident Grinch

victorhooi

Contributor

jgreco

Resident Grinch

Similar threads