Howdy. I hear someone asked for the iSCSI guy. :)
@Elliot Dierksen is on the money here - iSCSI has its own load-balancing protocol called MPIO (MultiPath Input/Output). The short answer here is that LACP and MPIO don't mix. LACP takes several interfaces, bundles them together and presents a single IP address. MPIO takes those several interfaces, keeps them distinct with their own IP addresses, and shotguns the traffic across all of them.
For a long answer, I'll have to set up an MPIO resource similar to this thread.
"Yes" to all of it. The
ciss
driver used for the SmartArray P4x0 series is not particularly good under FreeBSD, ZFS doesn't like hardware RAID and vice versa due to the way ZFS throws large volumes of traffic in "transaction groups" and often makes RAID controllers barf, and having personally worked with the P410 in the HP G6 series, it's a pile of crap in my opinion. Thankfully a "proper HBA" in the form of the Dell PERC H200/H310, IBM M1015, or HP H220 is very affordable online. Swapping in-place though is going to be impossible though because your SmartArray has set each drive up as a RAID0. Sorry.
The good news is that if you can get it swapped you should get a decent bit of performance improvement. (I've got some other hardware suggestions, but rather than clutter this thread, maybe a new one. Throw me a DM if you do, the @ tag functionality seems to be more miss than hit these days.)
In the case of iSCSI specifically adding "more lanes" does change the limit if done with MPIO. To extend the metaphor, LAGG/LACP widen the highway. If you do it with MPIO, you'll not only widen the highway, but add a bigger on-ramp to get more cars on at a time.
The 8x1.2T SAS drives should throw a decent (for spinning disk) amount of IOPS around, but the P410 is likely bottlenecking you, as would a choice of using anything other than mirror vdevs in the ZFS config. I suspect you might have a RAIDZ-something here. Re: the iSCSI drops, check to see if your switch is reporting any high error/retransmit counts, and whether or not your client side is seeing the connection actually drop.