(Bad) experience with .system on USB

[solarisguy]

I'm building a new x64 system with 9.2.1.3, for home use.

I do not like the idea of .system files competing with my real storage. Also if anything goes wrong with my real storage, it would be nice to be able go to .system/syslog/log/ and not be left in the dark.

My Intel server motherboard only has 6 SATA ports and all are to be taken by my RAID-Z2. Since I am booting from USB flash memory, my first thought was to use another USB flash memory. Decided against it. But just for testing purposes, played with a SATA drive in USB enclosure.

It worked with the following limitations:

• The system would not complete a reboot, possibly after shutting down the USB services FreeNAS still tried to write something to .system residing on USB.
• To simulate a failure, I have unplugged the USB cable. FreeNAS noticed it and, as expected, Samba stopped working. Plugging the USB cable back was a no-op (however, that could be specific to my hardware). I could not initiate a reboot from the GUI, also mistakenly I did exit the console and the console menu became unresponsive (it looked like it was trying to read or write GUI's IP address from .system). From my SSH session, I was still able to execute shutdown -r now, but it looks like I should have tried reboot -l instead. The reset button came to my rescue. And the system would not complete the boot sequence... Tried another USB memory with fresh 9.2.1.3. The same... Power button helped. I guess something needs to be reset within the USB controller (likely such a behaviour is specific to my hardware).
• I did not try the GUI and did zpool import on a command line. However, to get FreeNAS writing to it again required a reboot (messing with GUI did nothing). On import, there were the following errors &&&&
  
  Code:

  [root@freenas] ~# zpool import Log
  cannot mount '/Log': failed to create mountpoint
  cannot mount '/Log/.system': failed to create mountpoint
  cannot mount '/Log/.system/cores': failed to create mountpoint
  cannot mount '/Log/.system/samba4': failed to create mountpoint
  cannot mount '/Log/.system/syslog': failed to create mountpoint
  [root@freenas] ~#

For now, I bought a PCI card with SATA and a small SSD, but it looks that if I want .system to be separate, that pool should be mirrored (kind of obvious...). Tomorrow will bring me the news whether the newly acquired SATA controller is supported by FreeNAS, but that is another story :)

&&&& Added on 2014-05-02: I made a common mistake. Today I know that ZFS tried to mount on / and I should have told it to mount on /mnt

 

[cyberjock]

LOL. So much work for so little gains. The .system doesn't run hardly any load at all. so I find it interesting you'd put so much effort in getting rid of the .system dataset for such a small gain. But hey, worst case it's a learning experience.

 

[fracai]

Bigger issue I would think is that it seems like your main (only?) pool is on an external usb drive.

 

[solarisguy]

An update.

I ended up with two mirrored WD Blue 1TB disks for .system, jails/plugins and random small files.

A comment from https://bugs.freenas.org/issues/4912 that “Mixing system data with storage data is in my opinion illogical” could have been mine...

Especially, that with 9.2.1.5 there are already the following datasets:
.system
.system/cores
.system/samba4
.system/syslog
and likely with 9.2.1.6 there will be also:
.system/collectd

 

[cyberjock]

And why is it illogical? Should we extend our system requirements to "you require two pools"? At some point you have to make concessions. There's no getting around some problems at all.

If you've read any tickets I've wrote, I've been pretty active and aggressive with arguing over things if I think there's a better way. Sorry, but saying it's illogical and then not providing a viable solution isn't helpful at all.

Not to mention that the ticket mentioned *exactly* how you can achieve what you want "separate system data with storage data". So what's the problem if you can set it that way yourself? I just see no justifiable reason to force people to have 2 pools(one for data and one for system data) when it's not necessary. To me you're adding complexity that doesn't offset the additional risk of a more complex setup.

 

[solarisguy]

It is not my comment in https://bugs.freenas.org/issues/4912

I had explained my logic behind two separate pools, when I started this thread, and then proceeded to implement two pools for myself without promoting it anywhere else. As I have to 100% agree with you that suddenly mandating two pools from everybody would add unnecessary complexity. To sum up: I will continue to implement two pools for myself, but I will continue not asking that everybody does the same.

I made design decisions and performed customizations that are probably not for an average user, but I will try to add some explanation to my deployment steps and share the result on the forum.

Ultimately, it is likely that in the future some changes to FreeNAS are made that would allow seamless use of a USB something for .system

 

[cyberjock]

For the record.. my Mini(which is the only machine I have running 9.2.1.x) does have 2 pools. My second pool is a 32GB SSD with SLC, and there's no chance I could wear out that drive in my lifetime even if I deliberately tried. So guess where my system dataset is? ;)

I didn't think you posted that comment. But you sounded like you agreed with it when you said "could have been mine". I'm all about giving opinions about things that suck. I'm also all about giving a solution that is actually viable if you're going to say something sucks. There are exceptions. I don't generally go into detail about problems I've discussed to death because I'm not here to say the same thing 5000000 times for the rest of my life. I put it in my guide and warn against it, but that's about as far as I go for the common problems.