Can a "daisy-chain" a 2nd server at 10Gbps off my FreeNAS server?

[pclausen]

So I'm upgrading to 10Gbps on my home network.

The switch I got is an Ubiquiti UniFi 48 port with 2 SFP+ uplink ports. This guy:

https://www.ubnt.com/unifi-switching-routing/unifi-switch/

I also picked up an Intel X520-DA2 that will go in my FreeNAS server and connect to one of the SFP+ ports on the switch.

I have a 2nd server (Emby Media Server) that I'd like to connect at 10Gbps as well. However, I'd like to avoid using the 2nd SFP+ port on the switch since I want to connect a workstation to it for large file transfers to the FreeNAS server.

So, since the X520-DA2 is a dual SFP+ card, would it be possible for me to just connect the Emby Media server directly to the 2nd port? The Emby server will need to be accessible from clients connected to the switch, as well as remotely from the Internet.

In case it matters, my router is also a Ubiquiti unit, namely this one:

https://www.ubnt.com/edgemax/edgerouter-pro/

It is connected to the switch via a 1 Gbps SFP link.

So ideally, I'd like to be able to talk at 10 Gbps between each of the following:

FreeNas <-> Emby
FreeNas <-> Workstation
Emby <-> Workstation

To do that, the local traffic can't go through the router, since that would limit the traffic to 1 Gbps.

So can I configure FreeNAS to route that local traffic?

 

[depasseg]

It's not supported, but it's possible. Look into bridging. I think @jgreco had a couple posts about it a little while back. There might have even been a discussion in the 10GbE Networking Primer thread.

 

[jgreco]

It's not supported, but it's possible. Look into bridging. I think @jgreco had a couple posts about it a little while back. There might have even been a discussion in the 10GbE Networking Primer thread.

Totally possible. Just a little configuration and shell scripting. The only real risk is that FreeNAS hasn't been designed with this in mind, so things like reconfiguring the network stack could trainwreck if not thoroughly tested.

# ifconfig bridge0 create
# ifconfig bridge0 addm em0 addm em1 addm em2 (may need to toss "up" in here too, I forget)

 

[pclausen]

Thanks guys. I'll try setting up bridging. First step will be to install the 10Gig card and disable the 2 on-board NICs. And just get everything running like it does now. I never did figure out how to disable the 2nd one in BIOS in the first place, but this time around I'll have another go at it rather than leaving both on-board NICs enabled. For some reason, on these X10 mobo's it doesn't appear obvious how to do so,unlike just about every other mobo I have dealt with over the years.

 

[jgreco]

Why would you care if it is enabled??

 

[pclausen]

I guess it doesn't really matter, but I like to disable all features on my motherboards that I don't use, like sound, Serial ports, etc.

I notice during boot that FreeNAS tries to get a DHCP address for Eth1 and times out after like 15 seconds or so. Disabling the onboard NICs would shave a little off the boot up time I would imagine.

 

[jgreco]

So would configuring it down... but that's mostly personal preference.

 

[pclausen]

Ok, I got the X520 card installed and connected to my switch. The switch shows I have a 10Gbps link.

ifconfig gives the following

Code:

[root@freenas] ~# ifconfig
ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
        ether 00:1b:21:b8:bc:34
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: no carrier
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
        ether 00:1b:21:b8:bc:35
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: no carrier
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 0c:c4:7a:33:11:b4
        inet 10.0.1.50 netmask 0xffffff00 broadcast 10.0.1.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 0c:c4:7a:33:11:b5
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: no carrier

So what do I need to do to get it up and running? I would have expected either ix0 or ix1 (whichever the "top" one on the card is) to show active, but maybe I need to run some commands first?

Here's a view from the switch side showing the details for the SFP+ port:

So the link definitely appears to be up.

 

[pclausen]

Well, once I assigned an IP to the interface (via the GUI) it came up!

Code:

[root@freenas] ~# ifconfig
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
        ether 00:1b:21:b8:bc:34
        inet 10.0.1.59 netmask 0xffffff00 broadcast 10.0.1.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)
        status: active

And I can ping that interface from other clients connected to the switch. So far so good. Next step will be to disable the current 1GigE connection to force all traffic through the new 10GigE link.

 

[jgreco]

You need to configure the port as "up" before most ethernet cards will show as link up.

Configure your second card EXCLUSIVELY as up; no DHCP, no IPv4 or v6.

Then

ifconfig bridge0 create
ifconfig bridge0 addm ix0 addm ix1

and that second command may need "up" tacked on the end, too lazy to check.

 

[pclausen]

Thanks, I'll give that a shot once I bridge a 2nd server to the primary one.

As far as switching over to using only the 10Gig link, it was a simple matter of just unplugging the GigE cable and change the primary interface from the console, the my DHCP the new MAC mapping to the same IP as before and then rebooting the server.

So during boot, I now 3 interface where FreeNAS attempts to pick up DHCP assigned IPs (which never happens since the interfaces are not physically connected to anything).

So how do I "configure them down" permanently? I need to do this for ix1, igb0 and igb1.

 

[jgreco]

I'd suggest going to the interfaces list and either deleting the spares or at least editing the options to turn DHCP off.

 

[pclausen]

You mean from the GUI? If so, I'm not seeing any interfaces listed:

I do see ix0 under Network Summary:

I'm guessing the GUI only reflects me deleting the original igb0 interface from the console, but not me adding ix0?

 

[jgreco]

I think that's broken somehow. I'm not exactly sure what to suggest other than to "try stuff".

You can probably fix the original issue of DHCP by adding the unused interfaces under the Interfaces tab and then configuring DHCP off.

But the fact that you're not seeing any interfaces shown under Interfaces seems very broken. What happens if you go and add ix0 from the GUI? (Warning: might totally hose things, have a config backup.)

 

[pclausen]

So after backing up my config, I added the missing ix0 via the GUI. It took without any issues. I then proceeded to add the remaining 3 missing NICs are well, and didn't check DHCP and didn't assign IPs. That went without a hitch as well. Now all my interfaces are showing under Interfaces as seen here:

Next time I reboot, I'll keep an eye on it to see if it still tries to pick up IPs for those 3 unused interfaces.

 

[jgreco]

Think that'll do it for you.