I certainly never plan on editing the base FreeNAS install using up remaining USB space.. Unless you wanted to accelerate the disintegration of your pool/nas never install apps/plugins/whatever on the install USB.. Reading more of the DenyHosts FAQ I think this is possible to run in a jail and have full functionality (except maybe email).. I know DenyHosts can be run from cron or as a daemon or called from command line so it's versatile..
I'm going to assume that if I place this in a jail that the jail's hosts.allow would be edited by DenyHosts assuming SSH was running in the jail and bad connection attempts were logged appropriately - everything would work but isolated.. Aslong as the appropriate FreeNAS base logs could be parsed by the denyhosts daemon it should be able to update the jail's hosts.allow and/or create an naughty hosts file to use..
This user had success except for email which shouldn't be an issue. FreeNAS already handles this log (auth?) in it's daily email so if successful a second email shouldn't be necessary.. Not sure if they placed in a jail as user indicates 8.0.2 release version (could be 8.2.0?)..
http://forums.freenas.org/threads/denyhosts-and-sending-email.4539/
If I point the appropriate logs into the jail then configure DenyHosts properly. Afterwards point the output "naughty hosts file" back to base freenas and/or configure sshd to check this "naughty hosts file" within the jail before allowing all other connections I think this would work as intended.. DenyHosts doesn't watch the socket/port it uses the logs if I understand it properly.. So if run via cron should update file every X minutes? Seems interesting to say the least..
Basically I would probably take this route at first..
1. Create jail as desired.. Install DenyHosts & python dependency..
2. Using jail storage send the appropriate logs to the jail previously created for DenyHosts to read..
3. Configure DenyHosts to read appropriate logs (now linked in the jail).. Configure DenyHosts to create NaughtyHostsFile within the jail..
4. Either configure SSH to read NaughtyHostsFile within the jail or configure FreeNAS jail storage to send NaughtyHostsFile outside of the jail (back to base FreeNAS install/pool)..
5. Finalize the SSH setup depending on step #4.
Maybe would work? Anything I'm missing? This should shorten up the security email considerably for anybody with SSH on the outside..
I'm becoming more impressed with FreeBSD&Jails/FreeNAS everyday in terms of what it can do.. The rewrite was definitely worth it.. I was skeptical at first as it seemed Freenas 0.7X had all of the features (torrents etc) implemented albeit probably not in the safest way.. I don't want to advocate poor security practices.. Not using keys and pointing SSH onto the WAN is a risk and your asking for big trouble without editing root password..