Direct conenction via secondary NIC ?

[HolyK]

Hi all,

I have small network with Mikrotik, WorkStation (=WS), NAS, AP, and several more devices which are not important. All three of them are connected to router (blue) so any communication is going through it. There is also direct connection (red) between Workstation and NAS IPMI interface.

I have free ethernet port on NAS and also on Workstation so what i am thinking to do is to create another direct connection (green) between them to lower the load and bandwidth going through Mikrotik. Speed will not increase dramaticaly since NAS and WS ports are on HW switch on mikrotik, but it could lower the load of the interfaces on both sides since most data transfers are between these two, so standard connectivity on WS will be not affected when transferring large amount of data from/to NAS (considering the fact that WS has more HDDs).

NAS is running FreeNAS 9.2.0 and WS is Windows 7. There is CIFS, FTP running and also several services inside several Jails

The question is HOW would i do that? On router i am able to block any communication between NAS and WS so there is no problem redundancy conflict. Thing is how to achieve that all running services will be available on BOTH interfaces/IPs.

See the diagram, i guess it's self-explainable... Blue and Red lines already exists. GREEN one i'd like to achieve. IPs are simplified, just consider that everything is in the same subnet and no isolation or VLANs exists.

Thank you in advance for any tips :)

HolyK

 

[cyberjock]

Umm.. why not just connect eth4(or eth2 or eth5) to a switch, then connect your NAS and other devices to the switch. Any internal traffic would then go through the switch and not the router....

The switch is exactly how I handle all of my traffic at Gb speeds despite my router being a pfsense box with an Atom and can only handle about 300Mb/sec of traffic.

 

[HolyK]

Well, like i said, eth4 and eth2 on Mikrotik are on "switch" so dataflow between NAS and WS is not routed/bridged at all but going thru switch chip and then straight to the target, so router router chipset is not impacted at all an i am getting nearly Gbit network speed. Thing is that i have periodical snapshots of several partitions/directories and synchronizations scheduled from the WS and once this data transfer starts, all another usual activities have horrible responses (any download/upload drop to near zero, webpage loading is slow, Mumble/TS lagging, etc) even the WAN resorces are free. The one interface just get fully utilized. Similar situation is happening on NAS side. I have Transmission running (data are being stored/loaded on/from another physical disks/pool) and ZNC bouncer. Transmission speeds are dropping which i don't like much since i am one of the seeders of another open-project sources and mu mIRC starts to have high pings towards IRCbouncer which is on the same LAN (lol).

Yea i could probably limit the speed/queue size for transfers between these two machines or set some prioritization but before i do that i'd like to at least try to achieve the situation mentioned above, because if thats possible (and i really believe it is) it would keep the pure Gbit speed between NAS and WS while keeping the normal connections (blue one) free :]


EDIT: Huh, i found this which is basically the same situation what i'd like to achieve. I am a bit surprised that it could be THAT easy (just to set up static IP's on both sides). I thought that even if the connection will work, services (CIFS, FTP, ...) will be unreachable... i'll test it tomorrow ^^

 

[cyberjock]

I'm confused. Here's why....

There's two possible causes for your problems. Either you are saturating your network speed and that's slowing down other traffic on your LAN or your pool is being taxed with the snapshot/replication causing all other functions that use the pool to slow down.

If my pfsense box had 2 LAN ports, all traffic coming in 1 port *would* unequivocally be processed by the pfsense box before being dumped back on the LAN. If you are doing eth2 and eth4 on the eth3 switch, you may be in switch mode, but that won't negate the processing requirements. That "switch" still must look at every packet to determine if a packet coming in on eth2 is destined for eth4, eth5, or your WAN.

In my case, since I'm using an Intel Atom which is limited to about 300Mb/sec per my own testing, if I were to have 2 LAN ports on my pfsense box and I went in one LAN port and out the other, I can expect to be limited to 300Mb/sec. That's obviously really crappy and not want I want since I'm all 1Gb here at my house and I want that fill 1Gb dammit! Also, as a consequence of my pfsense box being taxed excessively, any machine I have on my LAN that is trying to do anything with the internet will have much slower throughput because I've taxed my router to the extreme. Your router box should only be "seeing" data that is intended to come into your LAN from your WAN or go out from your LAN to your WAN. Anything that's LAN->LAN should not be forced to go through a router ever. You don't *want* that kind of traffic going through your router. Most routers are low power and aren't designed to pass 1Gb/sec of traffic through it. Granted, if you have a $10000+ router this is a mute point. But for most home and small businesses users, these rules follow. ;)

 

[HolyK]

Backups, snapshots are on one pool (RAIDZ2 / 6 drives), services and jails are on another pool (mirror / two drives), so there no bottleneck with pool.

Like i said, dataflow between WS1 and NAS are flying with 1Gbps speed where router CPU will raise by 1% (yes, 1% so from 2 to 3% ^^). Thing is that Mikrotik has a separated Atheros switch chip which is handling all dataflow "inside" switch making them "wire-speed" so it looks like there is a real switch inside that box which is connected to the router :) . (In case of bridged settings, it would be like you said ... router overloaded, low speeds, whole network performance in hell). Good thing is that it's up to the admin which ports are in switch "mode" , if any. Bad thing is that for using this you need to sacrifice one physical port for the "masterport" purpose. Right now i don't need multiport swich (as separated device), that time will come when i will move to my own place and make my own structural network with a nice and shiny rack in the closet/basement ^^

It looks like this: (Its similar to my case, just ether2 in the diagram is my eth5 and ether5 in the diagram is my eth2)

So basically it's like having one router with 3 ports and then one switch with 4 ports, connected together...

Source: http://wiki.mikrotik.com/wiki/Manual:Switch_Chip_Features

So basically by buying switch (i mean physically separated device) will not help me since the i will still be using one physical NIC on both machines.

I will try to set the direct connection today but it will take some time since i am not able to kick my ass from the bed :/

 

[cyberjock]

Ah. Yeah, I wasn't convinced it worked that way internally. The stuff I looked at explained it more of how I had discussed it above.

You could do a direct link between the two as you have proposed. If you set up your direct connect with a different subnet than the others they should direct-connect.

 

[HolyK]

Mwhahaha, it is working and the network bandwidth is amazing :D

Code:

[156] local xxx.xxx.xxx.xxx port 50626 connected with yyy.yyy.yyy.yyy port 5001
[ ID] Interval      Transfer    Bandwidth
[156]  0.0- 1.0 sec  118 MBytes  990 Mbits/sec
[156]  1.0- 2.0 sec  118 MBytes  991 Mbits/sec
[156]  2.0- 3.0 sec  118 MBytes  991 Mbits/sec
[156]  3.0- 4.0 sec  118 MBytes  991 Mbits/sec
[156]  4.0- 5.0 sec  118 MBytes  991 Mbits/sec
[156]  5.0- 6.0 sec  118 MBytes  991 Mbits/sec
[156]  6.0- 7.0 sec  118 MBytes  991 Mbits/sec
[156]  7.0- 8.0 sec  118 MBytes  991 Mbits/sec
[156]  8.0- 9.0 sec  118 MBytes  991 Mbits/sec
[156]  9.0-10.0 sec  118 MBytes  991 Mbits/sec
[156]  0.0-10.0 sec  1181 MBytes  990 Mbits/sec
Done.

 

[cyberjock]

Very nice!

This is virtually identical to how I have 10Gb and 1Gb on my desktop and server. Both are connected to the 1Gb LAN, but my desktop and server are direct connected via a 10Gb fiber. ;)