NetApp DS2246 SSDs - no multipath, cannot sg_format

[Yuka]

I'm working on building out a system for my home lab. I'll admit the use of a shelf as a JBOD via HBA is new to me as is the NetApp in general. Please forgive any naivety on my part - this is intended to be a learning experience for myself. I've been leveraging the best of my google-fu since Wednesday and here is where I'm at:

The hardware specs are as follows:

Server blade:
Dell Poweredge R610
2x Xeon E5620
192 GB RAM
2x136 HDDs in JBOD (FreeNAS-11.2-U7 is installed here and manages the two as RAIDz1)
Dell H200E HBA

NetApp DS2246:
20x 1.6 TB NetApp HDD (da4-da23;da28-da47)
4x ~600 GB NetApp SSD (da0-da3;da24-da27)
2x FAS2552 controllers

1. The FAS2552 controllers are connected to the Dell 200E HBA via a QSFP+ to SFF-8088 cable.
2. The Dell has no problem seeing the disks in the NetApp.
3. I was able to run sg_format to resize the HDDs in the NetApp and create a RAIDz2 pool on the 20x HDDs.
4. FreeNAS sees /dev/multipath/diskX for the HDDs but not the SSDs. gmultipath status output shows the same.
5. The output of sg_format -v --format --size=512 --six /dev/da0 shows:

Code:

root@nfs01[~]# sg_format -v --format --size=512 --six /dev/da0
    NETAPP    X438_S1635400ATD    NA01    peripheral_type: disk [0x0]
        PROTECT=1
        << supports protection information>>
        Unit serial number: S2GUNAAH802160
        LU name: 5002538a0674ca70
    mode sense (6) cdb: 1a 00 01 00 fc 00
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks 781422768 [0x2e9390b0]
  Block size=520 [0x208]
    mode select (6) cdb: 15 11 00 00 18 00
MODE SELECT command: Reservation conflict SCSI status

camcontrol shows the following UUID pairings between the following pairs:
da0,da24
da1,da25
da2,da26
da3,da27

Attempting to run gmultipath label -v disk21 /dev/da0 /dev/da24 results in the following output:

Code:

gmultipath: cannot store metadata on /dev/da0: Input/output error

dd attempts with if=/dev/zero result in an Invalid argument error being thrown. /dev/null does not throw an error.

All of the above commands fail in the same way with any of the SSD /dev/daX's.

I'm out of ideas at this point and I have exhausted all references available to me at this point. Please let me know if you require any additional information and I'll be happy to provide it.

 

[Yuka]

FWIW: My goal here is to resize the sector size on the SSDs to 512 (as I did with the HDDs) and add them to the pool as L2ARC.

 

[Yuka]

I'm not entirely sure what caused this but the SSDs are taking an sg_format now. In order to make this happen I shut down the Dell blade and then shut down the netapp and then disconnected both from power for 5 minutes (probably an excessive amount of time but I had other tasks I was working on) and then brought up the Netapp and then the dell blade. At this point I was able to push sg_format -v --format --size=512 --six /dev/da0 which started successfully. I ran the remaining 3 SSDs this way via their own individual tmux sessions and they all completed successfully.

I gave FreeNAS a reboot. I had to manually label the multipath devices. I did so like this:
gmultipath label -v ssd1 /dev/da0 /dev/da24
... and so on for each pair.

At this point I was able to add the ssdX multipath devices to my existing pool via the extend feature as cache disks. Seems to be working well. I'll give it a reboot after a current large file operation is completed (2-3 days).

This complete cold shut down was the first time I had performed such a shutdown since bringing these two devices up in their current hardware configuration. I know these sorts of solutions aren't exactly the greatest due to their unexplained nature but I'm hoping that my documentation of my solution may help someone in the future who has the same problem.

Moving forward I've got two open ears to anyone who has any advice for me in regards to this situation or my FreeNAS configuration. Thank you for your attention and I hope this post helps someone out in the future.

 

[HoneyBadger]

Happy this worked out for you, doubly happy that you documented the solution in such detail. Performing reformats that change sector size or T10 protection level often seem to necessitate a full power cut to the device afterwards, but I suppose there might be situations like yours where it needs to be done beforehand as well.

With regards to advice; what is the general planned use of the server itself? Will it be used for household file sharing, or in a business environment? Are you planning to run virtual machines from this, store databases or other I/O sensitive applications? Or is it just a matter of having "big, reliable storage" and speed is a secondary consequence?

This will help determine the best disk layout for your HDDs, and use of your SSDs.

 

[Yuka]

With regards to advice; what is the general planned use of the server itself? Will it be used for household file sharing, or in a business environment? Are you planning to run virtual machines from this, store databases or other I/O sensitive applications? Or is it just a matter of having "big, reliable storage" and speed is a secondary consequence?

This will help determine the best disk layout for your HDDs, and use of your SSDs.

It's the primary storage for my home lab. I work in the cyber security field and this will be the core storage device for whatever it is that I'm working on. Nothing is business critical on it so to speak. It's already running as a file server via both NFS and SMB. I may spin up some of my other blades to act as VMWare hosts which would use this intermittently as storage - iSCSI or NFS - likely NFS for better interoperability with traditional NFS deployments/SMB. I've been thinking of hosting a few jails or even VMs on this for small services that may need to run 24/7 in the lab such as a DNS/DHCP server or perhaps an nginx webserver for hosting some static content - nothing that could be considered heavy lifting.

All that considered, I've currently enabled lz4 and dedup. I'm considering dropping dedup as I've noticed slow performance when handling large duplicate file-laden directory trees for a project I'm working on. (~50,000 directories with ~1 million files dispersed.) The lost time is starting to approach outweighing the virtue of saved disk space. Any guidance on the use of dedup, performance tuning of it, and best practices on disabling it would be greatly appreciated.

To elaborate a bit on the large directory trees I mentioned:
In this project I create a template directory tree with template files. Most of the template files contain the same content but have different file names. I'm using this as a file-based database for the project as I can very easily extract a particular piece of data via its file through a variety of easy and fast methods. I may go to a proper database in the future and am considering elasticsearch.
This template directory is copied to a new branch location and the new branch runs, collects the data and writes any changes from the template baseline (there are a lot of changes) to its respective branch off the template directory. The "completed" branch typically runs about 90 GB of data in full.

 

[HoneyBadger]

I'd consider using your SSDs as a small dedicated pool (2x2-way mirror) for your intermittent/high-performance VMware storage. No matter how much cache you throw at it there's no way to make magnetic storage sing and dance the way NAND can.

For the HDD stuff, start with setting atime=off and xattr=sa on the dataset with the huge number of files. I'd strongly advise killing deduplication entirely as it's not only a RAM hog, but also causes a large amount of metadata updates (vdev I/O) even if it's not overflowing your ARC with the DDT.

Throw zpool status -D yourpoolnamehere into an SSH session and see what the total end result is between Allocated and Referenced, as well as the line that starts with "dedup: DDT entries X, size on disk Y, size in core Z" - X * Z is the approximate amount of RAM used by your deduplication table.

For turning it off, that doesn't actively un-dedup deduped data. You'll need to delete or rewrite it. The same zpool status -D line will let you monitor if you have any deduped records still tracked.

 

[Yuka]

I'd consider using your SSDs as a small dedicated pool (2x2-way mirror) for your intermittent/high-performance VMware storage. No matter how much cache you throw at it there's no way to make magnetic storage sing and dance the way NAND can.

I'll keep this in mind if I get heavy on VM use. The L2ARC seems to have a great effect on my research processes and accessing the data... a huge performance boost... about 20x faster than my last solution (USB drive attached to a Dell Optiplex thin client running Ubuntu sharing the USB disk via NFS to my blades).

For the HDD stuff, start with setting atime=off and xattr=sa on the dataset with the huge number of files. I'd strongly advise killing deduplication entirely as it's not only a RAM hog, but also causes a large amount of metadata updates (vdev I/O) even if it's not overflowing your ARC with the DDT.

I set atime=off for all of my datasets. It makes sense and I really don't care about last accessed for any of my data. I'll reenable on a dataset-by-dataset basis if it breaks something and I can't find a way to fix it but I honestly doubt it.

Oddly enough, I cannot seem to set xattr=sa on the pool itself. I tried on both the pool and datasets.. all attempts resulted in cannot set property for 'ds01': 'xattr' must be one of 'on | off'. The command I used to receive that message was zfs set xattr=sa ds01 Subsequent attempts on pools gave the same result.

Throw zpool status -D yourpoolnamehere into an SSH session and see what the total end result is between Allocated and Referenced, as well as the line that starts with "dedup: DDT entries X, size on disk Y, size in core Z" - X * Z is the approximate amount of RAM used by your deduplication table.

For turning it off, that doesn't actively un-dedup deduped data. You'll need to delete or rewrite it. The same zpool status -D line will let you monitor if you have any deduped records still tracked.

I ended up just disabling dedup across the board. The system is running MUCH faster with much lower load averages. I've got more than enough space so I'm honestly not worried about consuming extra disk space going forward.

Thanks for all your advice and insight. I'll continue digging into why I cannot set xattr=sa and hopefully find a solution/reason.

 

[photomatix]

Not sure if this is directly related to my issue, but it seems pretty similar in some regards. I'm having issues adding/formatting some new SAS HDDs to a pool. See thread: NetApp DS4486 SAS Drive Issue - FreeNAS/TrueNAS | TrueNAS Community

HoneyBadger has already seen it, and I'm waiting to hear his response, but Yuka, if you have any input, I'm opened to ideas.