friolator
Explorer
- Joined
- Jun 9, 2016
- Messages
- 80
We've got a 40GbE Chelsio T580-SO-CR card installed in our FreeNAS box, and have just begun serious testing with it today. It's able to run for less than a minute, before the connection is dropped. Here's what the 40Gb switch's log says:
Sep 22 14:49:16 192.168.1.209 INFO web server: System log cleared by user admin.
Sep 22 14:49:37 192.168.1.209 ALERT stg: STG 1, topology change detected
Sep 22 14:49:47 192.168.1.209 NOTICE link: link down on port 61
Sep 22 14:50:03 192.168.1.209 NOTICE link: link up on port 61
Sep 22 14:50:33 192.168.1.209 ALERT stg: STG 1, topology change detected
Sep 22 14:51:33 192.168.1.209 NOTICE link: link down on port 61
Sep 22 14:51:49 192.168.1.209 NOTICE link: link up on port 61
Sep 22 14:52:19 192.168.1.209 ALERT stg: STG 1, topology change detected
Sep 22 14:59:28 192.168.1.209 NOTICE link: link down on port 61
Sep 22 14:59:29 192.168.1.209 WARNING system: 1m QDAC removed at port 61
[At this point, I moved the cable to port 49 to see if the problem was with the port itself]
Sep 22 14:59:37 192.168.1.209 NOTICE system: 1m QDAC inserted at port 49 is Accepted
Sep 22 14:59:52 192.168.1.209 NOTICE link: link up on port 49
Sep 22 15:00:22 192.168.1.209 ALERT stg: STG 1, topology change detected
Sep 22 15:16:04 192.168.1.209 NOTICE link: link down on port 49
Sep 22 15:16:21 192.168.1.209 NOTICE link: link up on port 49
Sep 22 15:49:30 192.168.1.209 ALERT stg: STG 1, topology change detected
Sep 22 16:01:08 192.168.1.209 NOTICE link: link down on port 49
[Here I decided to test both another port *and* another cable, just to rule that out.]
Sep 22 16:01:20 192.168.1.209 WARNING system: 1m QDAC removed at port 49
Sep 22 16:01:31 192.168.1.209 NOTICE system: 2m QDAC inserted at port 9 is Accepted
Sep 22 16:01:46 192.168.1.209 NOTICE link: link up on port 9
Sep 22 16:02:16 192.168.1.209 ALERT stg: STG 1, topology change detected
Sep 22 16:02:39 192.168.1.209 NOTICE link: link down on port 9
Sep 22 16:02:55 192.168.1.209 NOTICE link: link up on port 9
Sep 22 16:03:24 192.168.1.209 ALERT stg: STG 1, topology change detected
The dropped connections are only happening when actual data is flowing through the switch. We're not seeing this on the Windows workstation (Mellanox ConnectX3-VPI card) that we're doing the tests with. Our tests involve reading/writing test files on a shared volume on the FreeNAS box, so both the Windows and FreeNAS boxes are handling the same amount of data on the network.
If the switch is left alone for a while, you don't get any warnings. But every time you try to move data, you only get about 1 minute (or usually less) before the connection craps out.
Any suggestions on where we should start with this?
Sep 22 14:49:16 192.168.1.209 INFO web server: System log cleared by user admin.
Sep 22 14:49:37 192.168.1.209 ALERT stg: STG 1, topology change detected
Sep 22 14:49:47 192.168.1.209 NOTICE link: link down on port 61
Sep 22 14:50:03 192.168.1.209 NOTICE link: link up on port 61
Sep 22 14:50:33 192.168.1.209 ALERT stg: STG 1, topology change detected
Sep 22 14:51:33 192.168.1.209 NOTICE link: link down on port 61
Sep 22 14:51:49 192.168.1.209 NOTICE link: link up on port 61
Sep 22 14:52:19 192.168.1.209 ALERT stg: STG 1, topology change detected
Sep 22 14:59:28 192.168.1.209 NOTICE link: link down on port 61
Sep 22 14:59:29 192.168.1.209 WARNING system: 1m QDAC removed at port 61
[At this point, I moved the cable to port 49 to see if the problem was with the port itself]
Sep 22 14:59:37 192.168.1.209 NOTICE system: 1m QDAC inserted at port 49 is Accepted
Sep 22 14:59:52 192.168.1.209 NOTICE link: link up on port 49
Sep 22 15:00:22 192.168.1.209 ALERT stg: STG 1, topology change detected
Sep 22 15:16:04 192.168.1.209 NOTICE link: link down on port 49
Sep 22 15:16:21 192.168.1.209 NOTICE link: link up on port 49
Sep 22 15:49:30 192.168.1.209 ALERT stg: STG 1, topology change detected
Sep 22 16:01:08 192.168.1.209 NOTICE link: link down on port 49
[Here I decided to test both another port *and* another cable, just to rule that out.]
Sep 22 16:01:20 192.168.1.209 WARNING system: 1m QDAC removed at port 49
Sep 22 16:01:31 192.168.1.209 NOTICE system: 2m QDAC inserted at port 9 is Accepted
Sep 22 16:01:46 192.168.1.209 NOTICE link: link up on port 9
Sep 22 16:02:16 192.168.1.209 ALERT stg: STG 1, topology change detected
Sep 22 16:02:39 192.168.1.209 NOTICE link: link down on port 9
Sep 22 16:02:55 192.168.1.209 NOTICE link: link up on port 9
Sep 22 16:03:24 192.168.1.209 ALERT stg: STG 1, topology change detected
The dropped connections are only happening when actual data is flowing through the switch. We're not seeing this on the Windows workstation (Mellanox ConnectX3-VPI card) that we're doing the tests with. Our tests involve reading/writing test files on a shared volume on the FreeNAS box, so both the Windows and FreeNAS boxes are handling the same amount of data on the network.
If the switch is left alone for a while, you don't get any warnings. But every time you try to move data, you only get about 1 minute (or usually less) before the connection craps out.
Any suggestions on where we should start with this?
Last edited: