Please help - 10gb-10gb-1gb setup without 10gb switch

ysnk

Explorer
Joined
Dec 1, 2014
Messages
59
jgreco , Thanks again! I got it to work (with just a couple of revisions).
It survived few test restarts / shutdowns - comes back to life as expected. I'm planning to upgrade to 9.3 tonight, will see how it will go.

If someone wants to try it, below is the postinit script to use.
In my case em0 - adapter connected to the switch / rest of the network. ix0 - adapter connected directly to the workstation. Just substitute your interface names and it will work.
Capture1.JPG
 

vikingboy

Explorer
Joined
Aug 3, 2014
Messages
71
Interesting thread, thanks. I'd be interested in seeing what sort of perf numbers you can push over this config.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
You can probably redo that somewhat more clearly as:

ifconfig ix0 up; ifconfig bridge0 create; ifconfig bridge0 addm em0 addm ix0 up

I should never offer advice while on a mobile device. I always forget something. ;-) My concern was whether or not the "command" option would let you run several commands separated by semicolons, since this is ordering-dependent. The command I just gave is probably the cleanest, shortest way. But I applaud the effort you took to make this work, I very much appreciate anyone willing to put forth effort rather than just hope for an answer on a silver platter.

I have also created a new thread with some more general 10G discussion:

https://forums.freenas.org/index.php?threads/10-gig-networking-primer.25749/

I've suggested to some people the Dell 5524 switch with dual 10G SFP+ as a way to get inexpensive 10G switching, and I'm told it works well. We use the 7048 here as an edge switch with 10G uplinks into the switching core. The 7048 is basically a beefier L3 model of that device, and that also works very well. While I do not believe that the use of bridging within FreeNAS is bad/evil/etc., you are of course cautioned that it comes with some caveats, like your 10G workstation loses access to the network when you reboot FreeNAS.
 

ysnk

Explorer
Joined
Dec 1, 2014
Messages
59
Great, thanks - I'll try that command.
I realize workstation will loose the network if freenas is off, but that's fine - all my work files are going to be on Freenas anyway. In case of freenas failure I will just hook up the workstation to the switch and use backup storage - hopefully won't be any more than a few minutes downtime.

I just found one little thing though... 10G workstation cant resolve the "freenas" name anymore. If I try to ping freenas I get time out, or if I try to login by typing //freenas I get nothing. If I use IP address everything works.
I suspect it has something to do with host file, or name server, but not sure there to start. Other computers can connect using name or ip address. I have a simple windows workgroup setup here, not a domain if that makes any difference.
Any suggestions on how to resolve it?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I think bridging is such a basic thing in any OS that setting it up on freenas is not really making it "non-standard", it is a FreeBSD after all. And having just one cat7 cable between workstation and "IT closet" makes everything neat :)

That's a very bad assumption to make. FreeNAS' code isn't making assumption like "the user might create a bridged network" so lots of things may break either now or in the future as a result of this non-standard configuration. Yes, it seems simple, but when static scripts are running everything, doing stuff from the CLI won't necessarily end well. The WebGUI assume you don't do much of anything from the CLI except look around. You're doing *much* more than "looking around" when you create a bridge. ;)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Great, thanks - I'll try that command.
I realize workstation will loose the network if freenas is off, but that's fine - all my work files are going to be on Freenas anyway. In case of freenas failure I will just hook up the workstation to the switch and use backup storage - hopefully won't be any more than a few minutes downtime.

I just found one little thing though... 10G workstation cant resolve the "freenas" name anymore. If I try to ping freenas I get time out, or if I try to login by typing //freenas I get nothing. If I use IP address everything works.
I suspect it has something to do with host file, or name server, but not sure there to start. Other computers can connect using name or ip address. I have a simple windows workgroup setup here, not a domain if that makes any difference.
Any suggestions on how to resolve it?

Very likely the 10G workstation has mysteriously decided to prefer the 1G interface for Windows authentication. Look in the network preferences on the workstation for the 1G and 10G interfaces and twiddle appropriately so it treats the 10G interface as being part of the workgroup. I don't remember the specifics, try Network Connections -> 10G Ethernet -> Properties -> Client for Microsoft Network (must be checked) and related cruft underneath that. Disable the 1G ethernet card on the workstation to help it figure it all out. Windows networking is all very depressing.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
That's a very bad assumption to make. FreeNAS' code isn't making assumption like "the user might create a bridged network" so lots of things may break either now or in the future as a result of this non-standard configuration. Yes, it seems simple, but when static scripts are running everything, doing stuff from the CLI won't necessarily end well. The WebGUI assume you don't do much of anything from the CLI except look around. You're doing *much* more than "looking around" when you create a bridge. ;)

Dear Mister Pessimistic,

There is a reason that FreeNAS was based on a powerful open source operating system like FreeBSD. It brings with it lots of interesting possibilities such as jails and vlans and other stuff that many other NAS devices can't or won't do.

I don't think it's a bad assumption to make and I've been doing this networking stuff a Really Long Time, since, like, long-before-FreeBSD. Bridging is reasonably simple. It's fine to make note that the device firmware might stomp all over it if you try to do something like configure jails alongside this. That's even a correctable problem, but at this point when you're saying things like "lots of things may break" you're just spreading FUD. (Name half a dozen things that'll actually break and I'll be impressed and even apologize.)

I've spent many years making FreeBSD boxes do amazing networking tricks. This is straightforward. It's simple. It's easy to debug. Some of us bridge frickin' vlans over TAP devices to create complex VPN servers servicing numerous types of applications, the more complex stuff here automatically performs instantiation and destroying of bridge devices after the last member tap closes. I have some "small" experience with the bridging system and complex network topologies. What is suggested in this thread is child's play. This Is Quite Simply Not A Big Deal, and if/when it were to break, it also isn't hard to fix.

Yours Truly,

Grinchy
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I don't think you get what I'm saying. If the code is tracking what it creates for bridges (and it obviously uses bridges for jails), creating a bridge might confuse the code. The code might expect bridge0 to be for jails and might be hard coded that way. If you do bridge0 how is that going to work with the jails? Will the code realize that you've already created a bridge and create bridge1 instead? Will it even work? Will it work correctly? Will it work correctly in the future? These are all questions that none of us can answer with a high degree of certainty.

I'm not saying that the bridging feature isn't trivial. It is. I'm saying that using it when FreeNAS expects full control and for you to not do anything spooky behind it's back is probably a bad place to assume.

And you might call it "easy to debug", but I bet 99% of the forum wouldn't know where to start if there was a problem with bridges.
 

rogerh

Guru
Joined
Apr 18, 2014
Messages
1,111
The bridge is intellectually cleaner than a separate subnet and saves one bit of wire (running parallel with existing wires). You potentially lose up to 1Gb of your 10Gb connection to other network traffic (but only if both machines are very fast). Doesn't seem that clearcut to me.

The simplest Windows network solution is a hosts file on the workstation either way - the only problem is remembering to change it if necessary.
 

rogerh

Guru
Joined
Apr 18, 2014
Messages
1,111
The bridge is intellectually cleaner than a separate subnet and saves one bit of wire (running parallel with existing wires). You potentially lose up to 1Gb of your 10Gb connection to other network traffic (but only if both machines are very fast). Doesn't seem that clearcut to me.

The simplest Windows network solution is a hosts file on the workstation either way - the only problem is remembering to change it if necessary.


Edit: what is the problem with another subnet if a) it has no gateway and b) all machines are on the main subnet?

Edit2: I meant to edit that rather than reply to it - sorry.
 

ysnk

Explorer
Joined
Dec 1, 2014
Messages
59
Thanks guys for all the help - will try tomorrow.
My freenas is still in testing stage so I'm experimenting with it.

What is suggested in this thread is child's play.
I thought the same thing starting this thread. Who knew it will take 50 replies to sort it out! :)
 

ysnk

Explorer
Joined
Dec 1, 2014
Messages
59
I don't think you get what I'm saying. If the code is tracking what it creates for bridges (and it obviously uses bridges for jails), creating a bridge might confuse the code. The code might expect bridge0 to be for jails and might be hard coded that way. If you do bridge0 how is that going to work with the jails? Will the code realize that you've already created a bridge and create bridge1 instead? Will it even work? Will it work correctly? Will it work correctly in the future? These are all questions that none of us can answer with a high degree of certainty.

I'm not saying that the bridging feature isn't trivial. It is. I'm saying that using it when FreeNAS expects full control and for you to not do anything spooky behind it's back is probably a bad place to assume.

And you might call it "easy to debug", but I bet 99% of the forum wouldn't know where to start if there was a problem with bridges.

I see your point. But I hope it will work fine with my simple configuration. All I need it to be is a file storage for 3-4 users, without any jails, pluging, media servers, nothing. I want to maximize the performance for one of the users, that's why I need 10gb. That's all. I might want to use owncloud, but not sure yet, maybe i'll skip it to keep everything as simple as possible.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I don't think you get what I'm saying. If the code is tracking what it creates for bridges (and it obviously uses bridges for jails), creating a bridge might confuse the code. The code might expect bridge0 to be for jails and might be hard coded that way. If you do bridge0 how is that going to work with the jails?

Since the jail code appears to go to some trouble to be able to keep track of the active bridges, I'm *guessing* it'll be just fine. You can go check out the code over in /usr/local/share/warden/scripts/backend/functions.sh etc yourself if you wish.

Generally speaking, there are some of us that spent years finding the edge cases, talking to subsystem developers, filing bug reports, and helping guide the development of FreeBSD's high performance network stack. That's why things like "ifconfig bridge create" go to the trouble of dynamically allocating a new bridge and putting the new device name on standard output, and why a lot of the rc code to implement the network stack is so complicated.

Will the code realize that you've already created a bridge and create bridge1 instead?

Appears to. We can work around it if it really doesn't;

1) s/bridge0/bridge999/g

2) A more thorough solution that would address the more likely issue of the postinit script running after jails starting and the bridge0 already existing: use

ifconfig ix0 up; ifconfig `ifconfig bridge create` addm em0 addm ix0 up

but I deemed backquoting to be less likely to work in a command so I didn't suggest it - keeping it simple and understandable is somewhat better anyways.

Will it even work?

Try it instead of spreading FUD.

Will it work correctly? Will it work correctly in the future? These are all questions that none of us can answer with a high degree of certainty.

Maybe a question that you can't answer with a high degree of certainty. I'm familiar enough with them that I feel pretty certain. Just a matter of getting the details right.

I'm not saying that the bridging feature isn't trivial. It is. I'm saying that using it when FreeNAS expects full control and for you to not do anything spooky behind it's back is probably a bad place to assume.

And you might call it "easy to debug", but I bet 99% of the forum wouldn't know where to start if there was a problem with bridges.

A new user in this very thread figured it out from a push I gave in the general right direction. I think any admin smart enough to ask the question can be safely given the information. Which brings us back to the original post in this thread.

Now, rather than unproductively rattle on about how dangerous and risky this all is, a more productive course of action would be to advocate to create a feature to support interface bridging. It's similar in complexity to vlans or lacp in that the technology is well-understood and baked into the system, and just requires a little configuring.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Edit: what is the problem with another subnet if a) it has no gateway and b) all machines are on the main subnet?

Edit2: I meant to edit that rather than reply to it - sorry.

The usual problem with more subnets and Windows machines are that it'll invariably pick the wrong network for SOMETHING, and if it doesn't, it's more often dumb luck that it didn't, so that when something changes in the future, you get random breakage.

Given sufficient Windows-fu, it can be all made to work correctly and reliably, but to me, a UNIX guy, I'd rather just get a switch with 10Gbps uplinks and forget about it, or there's a lot of attractiveness in creating the single subnet solution we're discussing with bridging. Most Windows admins are not experienced in administering a Windows system with multiple network interfaces, so the keep-it-simple principle suggests that finding a way to create a single subnet is the best bet.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Actually Jordan doesn't like the "me too" stuff in tickets. :P
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Jordan's a big boy and he can tell me that himself if he wants. Historically we've been encouraged to help devs gauge interest in a feature in this manner.
 

eldo

Explorer
Joined
Dec 18, 2014
Messages
99
I'm glad I found this thread, my new installation will be similar to ysnk's.
Thanks very much jgreco for pointing in the correct direction to make a simple bridge for certain usage scenarios.
 

Dave Genton

Contributor
Joined
Feb 27, 2014
Messages
133
WOW, That was an entertaining thread for an aging network architect/engineer to read during a bout of insomnia, and I'm even still awake ! One aspect that was not mentioned was the fact that nearly 100% of packet/frame processing is done in hardware today. In the past the CPU was punted every packet, then through evolution only broadcasts and multicasts were cpu punted with the NIC processing all unicast traffic. Then broadcasts and multicasts known to the NIC were even prevented from being cpu punted and today very few packets/frames require a cpu interrupt for processing. Most NICs even offload tcp/udp processing, encryption, etc all onto the NIC relieving the CPU of this duty. With DMA and other improvements even this "software" bridge should require minimal if any CPU horsepower as one NIC places a frame into memory buffer and the other NIC retrieves it. The bridge at this point is not much more than a buffer space allocated by the OS via the CPU. Beyond its basic construction, maintenance, and teardown by the OS all packets in and out should be cpu interrupt free, provided of course you have a decent NIC and chipset, cannot say all NICs can do this...

Aside from using a point to point 10G connection which I understand at 10G prices for home setups I cannot really see any benefit when you can indeed spend under $30 and get a 5 port gigabit ethernet switch with a non-head of line blocking design. For $29 bucks I got a 5 port, silent (fanless) unmanaged switch that does wirespeed gigabit ethernet. 5 gigabit ports with a nearly 10Gb backplane, so with no head of line blocking all 5 ports can do wirespeed with room to breathe, $29 I would pay to get back the time spent reading the banter and negativity in this thread ! :) Jgreco does however have a new fan :)
 
Top