To my knowledge, it has never been on. Is there a way to check this from the command line? The GUI shows "zfs deduplication" as inheret (off) everywhere.
If you can type "zpool list" at the CLI and see "1.00x" in all columns, let's consider that sufficient as a sanity check.
I will delete the tunable and reboot. Once I figured out that the crash was memory related, I watched top with high update rates (top -s1), to see what was happening as the system was stressed with the iSCSI write. Swapping appears to be a hard wall... and when it happens, freenas stops serving iSCSI, NFS, CIFS... and every machine on the network that talks to freenas comes to a screeching halt. And if I don't catch it and stop the write within a split second of it starting to swap, it will lock up hard, dropping its network connections, etc.
It shouldn't do that, obviously. The swapping isn't a great thing but it is fairly normal for some modest amount of swapout to occur over time. The ~4GB that unused bits of the FreeNAS middleware seems to like to occupy is the usual target. This is because there's a lot of stuff on a NAS that isn't used by your *particular* configuration.
Is there any chance that when it "crashes", it recovers over time? It's possible that you're running into some variation of the issues in bug 1531 relating to transaction group writes, which are supposed to be addressed by the new write throttle mechanism, but if you're maybe catching it before it is able to measure and adjust, it's very possible you could create a situation where the system might go catatonic for ... I'm just going to guess at 30-180 seconds. In such a case, what's actually happening is that one transaction group is being flushed to disk and another full transaction group has been created in the meantime. At that point, ZFS *must* pause, because it isn't committing to disk quickly enough.
You can test that by running "iostat 1" on some of your disks, or gstat, on the console and then causing this to happen. If that keeps running, and shows your disks very busy, .... bingo.
That can be at least partially addressed by reducing the transaction group window to one second. But you're still basically doing what Stanley does to this kid:
to your pool. Seriously, do not just sit your ZFS pool in front of the data firehose and then suddenly turn it on full blast. While the system warms up (after any reboot or pool import), try some smaller heavy bursts of traffic to help ZFS learn the characteristics of your pool, at which point it is likely to smarten up a good bit and not be soaking up totally stupid sized transaction groups. Once you warm up the pool and ZFS has learned its performance, it's fairly good about not totally misjudging transaction group sizes. But it can, and does, when cold.
The primary iSCSI target was default block size. I created a secondary target with block size = 512 to experiment to see if write speeds would improve. Writing to this target with bs=512 seems to lock up the system very quickly. The target with bs=16k seemed more resilent, but would still lock things up if freenas had other NFS activity going on at the same time. I was going to confirm this tonight - I created 2 identical bare-bones linux VMs on the ESXi. One on the bs=16k iSCSI datastore, and the other on the bs=512 iSCSI datastore.
K.
The SSDs are simply another RAID mirror on the system with data served over NFS. The data on these disks is a cache that we wanted the fastest possible random read access to, so we put it on SSDs. I simply haven't been talking about them because they didn't seem pertinent to the discussion. The iSCSI datastore is on the 4x4TB HDDs.
How refreshing. Usually my cynical crystal ball is right on the money. This is suddenly a bunch less tedious.
At the end of the day, if what I'm trying to do is beyond the scope of what freenas can do on this platform with 32GB of RAM - that's fine. I'm happy to reconfigure things... but there is time and money involved with changing it, so I felt I needed to understand what might be happening. It didn't feel to me that I could be overwhelming the disk I/O or RAM on the system with just 1Gb/s NICs. It seems so lightly loaded moving NFS traffic while flooding the NIC, that serving 1TB over iSCSI with a dedicated NIC link didn't strike me as something that would require a huge jump in compute power and storage.
Well, it's a bit perverse but experience suggests that you just need ... more ... for iSCSI. Whether or not it will work for your scenario is best left to testing. Things here worked okay with 64GB of RAM but I pushed it out to 128 anyways.