Configuring 10GB VLAN's to be routable

vroger

Dabbler
Joined
Sep 13, 2019
Messages
12
Hello,


I have built a freenas system built off of a HP DL380 G7, with a mellenox connectx-2.

With a fresh install or default's restored it grabs a DHCP address of 172.168.1.x my vlan 1 address range. But I want to use vlan 70 and 80 for iSCSI.

But soon as I create the vlan interfaces and configure them, I loose connectivity to the original configuration. I tried setting static routes without any luck.

all my 10GB ports on my arsita are trunked to all vlans (I have many other devices although non unix ) working correctly with the same vlans.

Thanks
 

vroger

Dabbler
Joined
Sep 13, 2019
Messages
12
ok, I added
gateway_enable="YES" to etc/rc.conf
then rebooted and added my default route to the router 172.168.1.1 and it seems to be working, this correct?
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Well, if it's working, it's probably correct to some degree. We'd need to see the network configuration from your Arista switch to tell for sure.

From what I understand, you have three VLANs - default VLAN 1, VLAN 70, and VLAN 80 - and a two-port ConnectX-2. Is using one of the DL380's onboard Broadcom NICs an option for management? This would let you just assign VLAN70/VLAN80 directly as access ports to the two interfaces on your ConnectX-2.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Please don't use 172.168. That's someone else's network (AOL to be specific). You seem to be confusing 192.168.0.0/16 and 172.16.0.0/12.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
And why exactly do you need iSCSI networks to be routable? You can do that but it won't work well, or right, or maybe at all.
 

vroger

Dabbler
Joined
Sep 13, 2019
Messages
12
And why exactly do you need iSCSI networks to be routable? You can do that but it won't work well, or right, or maybe at all.

point taken, i'll consider breaking off nfs and iSCSI separately
 

vroger

Dabbler
Joined
Sep 13, 2019
Messages
12
Please don't use 172.168. That's someone else's network (AOL to be specific). You seem to be confusing 192.168.0.0/16 and 172.16.0.0/12.

Thats what happens when you setup your lab in the wee hours of the morning and make a typo..
 

vroger

Dabbler
Joined
Sep 13, 2019
Messages
12
Well, if it's working, it's probably correct to some degree. We'd need to see the network configuration from your Arista switch to tell for sure.

From what I understand, you have three VLANs - default VLAN 1, VLAN 70, and VLAN 80 - and a two-port ConnectX-2. Is using one of the DL380's onboard Broadcom NICs an option for management? This would let you just assign VLAN70/VLAN80 directly as access ports to the two interfaces on your ConnectX-2.

A single connectx-2 card. vlan 1 the default is what the card grabs by default.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
A single connectx-2 card. vlan 1 the default is what the card grabs by default.

Unlikely. The card is likely configured untagged by default (i.e. no VLAN support). Your switch might place untagged traffic on VLAN 1 but that doesn't mean the card is configured for VLAN 1. If you configure your switchport as access VLAN 100, that untagged traffic will go to VLAN 100.
 

vroger

Dabbler
Joined
Sep 13, 2019
Messages
12
Unlikely. The card is likely configured untagged by default (i.e. no VLAN support). Your switch might place untagged traffic on VLAN 1 but that doesn't mean the card is configured for VLAN 1. If you configure your switchport as access VLAN 100, that untagged traffic will go to VLAN 100.

the switchports are setup all trunked, allowing all vlans. Lyaer 3 svi's are on the arsita as well.

Either way, I got it working , and moved back to NFS.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
the switchports are setup all trunked, allowing all vlans. Lyaer 3 svi's are on the arsita as well.

Either way, I got it working , and moved back to NFS.
At 10Gbps the difference is mostly academic, unless you can leverage the VAAI primitives over iSCSI.

@jgreco is right as usual when it comes to the networking though. Your switch may accept the untagged traffic and assign a global native VLAN of 1, that doesn't mean that you can't push a different VLAN in access/trunk mode.

We need the equivalent of a show run interface on the switchports connected to your FreeNAS unit as well as the interface config from FreeNAS. There's no reason it can't handle VLAN 1+70 on one port and 1+80 on another, if trunked/native VLANs are set probably.

Still a fan of dedicating the two 10Gbps to access mode on each of VLAN 70/80, and using a 1Gbps aggregation group to keep management traffic out of band.
 

vroger

Dabbler
Joined
Sep 13, 2019
Messages
12
The below is the switch config, i'd have to run to the colo to see what port it's at.


Command: show running-config
! device: Arista7148 (DCS-7148S, EOS-4.13.10M)
!
! boot system flash:/EOS-4.13.10M.swi
!
errdisable recovery cause bpduguard
!
transceiver qsfp default-mode 4x10G
!
queue-monitor length
!
hostname Arista7148
ip domain-name foo.com
!
spanning-tree mode none
!
no aaa root
!
username redacted
!
clock timezone America/New_York
!
vlan 3,5-6,8,10,20,30,40
!
vlan 50
name VDI
!
vlan 60,70,80,90,100
!
interface Ethernet1
mtu 9214
switchport mode trunk
!
interface Ethernet2
mtu 9214
switchport mode trunk
--More--
interface Ethernet3
mtu 9214
switchport mode trunk
!
interface Ethernet4
mtu 9214
switchport mode trunk
!
interface Ethernet5
mtu 9214
switchport mode trunk
!
interface Ethernet6
mtu 9214
switchport mode trunk
!
interface Ethernet7
mtu 9214
switchport mode trunk
!
interface Ethernet8
mtu 9214
switchport mode trunk
!
interface Ethernet9
mtu 9214
switchport mode trunk
!
interface Ethernet10
mtu 9214
switchport mode trunk
!
interface Ethernet11
mtu 9214
switchport mode trunk
--More--
!
interface Ethernet12
mtu 9214
switchport mode trunk
!
interface Ethernet13
mtu 9214
switchport mode trunk
!
interface Ethernet14
mtu 9214
switchport mode trunk
!
interface Ethernet15
mtu 9214
switchport mode trunk
!
interface Ethernet16
mtu 9214
switchport mode trunk
!
interface Ethernet17
mtu 9214
switchport mode trunk
!
interface Ethernet18
mtu 9214
switchport mode trunk
!
interface Ethernet19
mtu 9214
switchport mode trunk
!
interface Ethernet20
mtu 9214
switchport mode trunk
--More--

!
interface Ethernet21
mtu 9214
switchport mode trunk
!
interface Ethernet22
mtu 9214
switchport mode trunk
!
interface Ethernet23
mtu 9214
switchport mode trunk
!
interface Ethernet24
mtu 9214
switchport mode trunk
!
interface Ethernet25
mtu 9214
switchport mode trunk
!
interface Ethernet26
mtu 9214
switchport mode trunk
!
interface Ethernet27
mtu 9214
switchport mode trunk
!
interface Ethernet28
mtu 9214
switchport mode trunk
!
interface Ethernet29
mtu 9214
switchport mode trunk
--More--

interface Ethernet30
mtu 9214
switchport mode trunk
!
interface Ethernet31
mtu 9214
switchport mode trunk
!
interface Ethernet32
mtu 9214
switchport mode trunk
!
interface Ethernet33
mtu 9214
switchport mode trunk
!
interface Ethernet34
mtu 9214
switchport mode trunk
!
interface Ethernet35
mtu 9214
switchport mode trunk
!
interface Ethernet36
mtu 9214
switchport mode trunk
!
interface Ethernet37
mtu 9214
switchport mode trunk
!
interface Ethernet38
mtu 9214
switchport mode trunk
--More--
!
interface Ethernet39
mtu 9214
switchport mode trunk
!
interface Ethernet40
mtu 9214
switchport mode trunk
!
interface Ethernet41
mtu 9214
switchport mode trunk
!
interface Ethernet42
mtu 9214
switchport mode trunk
!
interface Ethernet43
mtu 9214
switchport mode trunk
!
interface Ethernet44
mtu 9214
switchport mode trunk
!
interface Ethernet45
mtu 9214
switchport mode trunk
!
interface Ethernet46
mtu 9214
switchport mode trunk
!
interface Ethernet47
mtu 9214
switchport mode trunk
--More--

!
interface Ethernet48
mtu 9214
!
interface Management1
ip address 172.168.1.252/24
!
interface Management2
!
interface Vlan10
ip address 172.16.10.1/24
!
interface Vlan20
ip address 172.16.20.1/24
!
interface Vlan30
ip address 172.16.30.1/24
!
interface Vlan40
ip address 172.16.40.1/24
!
interface Vlan50
description VDI
ip address 172.16.50.1/24
!
interface Vlan60
ip address 172.16.60.1/24
!
interface Vlan70
ip address 172.16.70.1/24
!
interface Vlan80
ip address 172.16.80.1/24
!
interface Vlan90
ip address 172.16.90.1/24
--More--
interface Vlan100
!
ip routing
!
management ssh
authentication mode password
!
!
end


---


also I made a post in storage about some different storage configs, love some input.
 
Last edited by a moderator:

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
Why would you have to go on site to check port numbers? You should be able to use lldp or even just check the MAC table.
 
Top