FreeNAS only sees a single disk from 4-disk USB enclosure

[Zugschlus]

Hi,
I have temporarily connected a Raidsonic IB-3640SU3 to my FreeNAS 11.0-U4 via USB3. The Raidsonic box is a four-disk external USB enclosure without its own RAID capabilities and exports the four disks as four disks. The host system is an HPE Microserver Gen8 which has 4 SATA disks of its own.

On this hardware, Linux (grml) sees the disks from the Raidsonic box just fine and can use them. I have used this system to zero out the disks completely.

The Raidsonic box has the annoying habit of exporting all four disks with the same serial number:

Code:

da2: Serial Number 5D0FFFFFFFFF
da3: Serial Number 5D0FFFFFFFFF
da4: Serial Number 5D0FFFFFFFFF
da5: Serial Number 5D0FFFFFFFFF

This caused FreeNAS to act up. The (old) web interface just offers da2 in "View Disks", and the syslog is full of multipath messages. This can be googled for FreeBSD and the recommendation is to manually disable the geom_multipath kernel module in this case. I did that.

In dmesg, I see da2 to da5 just fine, but the web interface still only shows da2. I am also a bit confused that dmesg is full of unretryable errors regarding da0, which is one of the USB sticks the system runs from, without the webinterface showing obvious troubles with the system "disks".

I do not have too much clue about how FreeBSD handles its disks, and the FreeBSD people in my personal bubble don't either, so I'm asking here. I know my way around Linux and somer other Unices, and need some FreeBSD cookies to debug this. I'll happily give information that might be helpful. Attaching dmesg for starters.

 

[Arwen]

Well, it would appear that the enclosure has a SATA Port Multiplier in the mix. Then that goes to the USB port. SATA Port Multipliers are not a good thing for any server, (regardless of OS, and this does apply to Linux). They also are limited to 5 disks per SATA host port.

Note that SAS Expanders are different. They support multiple host ports, (like up to 8). And the downstream device ports can be greater than 32. Plus, SAS supports 12Gbps when using HBA and SAS Expander chip that support SAS 3. (SAS 4 is coming out soon, with 22.5Gbps transfer rate, but double the speed due to reduced over-head).

Basically USB attached data disks in FreeNAS, (except for backups), is discouraged for a good reason. Adding in a SATA Port Multiplier, (if that is what is used), makes it worse.

 

[Zugschlus]

That's why I said "temporarily". Having the disks connected locally is not any worse than connecting them to a different machine and then doing the transfer over the network.

I can do that after acknowledging that this is just one more thing I tried and didn't work at all with FreeNAS/BSD (four of three, two of them probably caused by the picky hardware, one by ZFS and now, one by FreeNAS).

 

[Arwen]

Sorry, I don't have any suggestions to over come your issue.

For completeness, (in case someone else has an idea), what is your use case?

 

[Zugschlus]

The use case is using the disks in the USB enclosure as a temporary storage place for the contents on the internal disks because I wasn't aware that ZFS is unable to expand a three-disk set ("RAID-Z1"?) to a four-disk set ("RAID-Z2"?) without data loss. When setting up the box, I only had three equally-sized disks on hand and obtained the fourth one two weeks later, after the three-disk ZFS set was already in use since I expected to be able to convert the three-disk set to a four-disk set by just adding the new disk to the set.

When I queried the FreeBSD/ZFS people in my filter bubble how to do this, I got told that ZFS is a datacenter file system and adding single disks to an existing disk set is not a datacenter use case. The USB enclosure is already my Plan B, and it looks like that I'll have to establish plan C, hauling the data over the network twice.

 

[wblock]

Is it impractical to remove the four new drives from the USB enclosure and attach them directly to the FreeNAS system? Create a volume on that, replicate the original data to it, detach the original volume, recreate it as RAIDZ2, then replicate the data back. Yes, this requires moving the data twice, it's essentially a backup and restore.

got told that ZFS is a datacenter file system and adding single disks to an existing disk set is not a datacenter use case

Really, it's just that this is a limitation of ZFS. It's not that it's not desired, it's just that the design of ZFS makes it difficult to do. I'm pretty sure that technically it could be done, but there has not been a high enough demand for it yet.

 

[danb35]

When I queried the FreeBSD/ZFS people in my filter bubble how to do this, I got told that ZFS is a datacenter file system and adding single disks to an existing disk set is not a datacenter use case.

That's probably a generally-accurate statement of why the feature isn't there yet. However, it appears it's coming: https://forums.freenas.org/index.php?threads/raidz-expansion-its-happening.58575/

Edit: To be clear, the project is intended to allow you to expand a vdev while staying at the same RAID level (e.g., turn a three-disk RAIDZ1 into a four-disk RAIDZ1). Changing the RAID level (e.g., RAIDZ1 to RAIDZ2) is not in scope. There are hack-ish ways to do something like this (involving sparse files, CLI pool creation, and intentionally degraded pools), but if you have to be told what they are, you probably shouldn't be messing with them, at least for important data.

 

[Zugschlus]

Is it impractical to remove the four new drives from the USB enclosure and attach them directly to the FreeNAS system? Create a volume on that, replicate the original data to it, detach the original volume, recreate it as RAIDZ2, then replicate the data back. Yes, this requires moving the data twice, it's essentially a backup and restore.

The machine is a HPE MicroServer Gen 8, which has only four drive bays.