NVMe support?

Status
Not open for further replies.

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I don't see the required entry in loader.conf, so I'm thinking "no".
 

airflow

Contributor
Joined
May 29, 2014
Messages
102
I'm hoping to eval a few units next week or so.
Hi aufalien, did you really have the chance to eval those units as you said this summer? Please share your experience. I would like to hear of somebody having some success about this.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I've got a Samsung XP941 that I'll be playing with in the next month or two. It just arrived but I'm still waiting on the PCIe card and a new server to try it in.

The form factor on this M.2 stuff is just absolutely amazing! Look at this:

Photo-2014-12-31-07-07-50_4297.JPG

That's an old thumb drive and an SFP module next to the new module, still in-bag. It's also amazingly thin...
 

airflow

Contributor
Joined
May 29, 2014
Messages
102

zambanini

Patron
Joined
Sep 11, 2013
Messages
479
XP941 is AHCI based, not nvme.

m.2 is just the form factor.

right now I know only about intel nvme based ssd.

nvme should have less latency then ahci. slog :)
http://www.supermicro.com/products/nfo/NVMe.cfm with a nice compare table about speed and latency on nvme, sas and sata.
ZeusRAM maybe slower
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
XP941 is AHCI based, not nvme.

Good point, I'm just looking for a high performance device that can sit in an available PCIe slot in a 2U server instead of requiring all the plumbing of SAS/SATA. Hope to make it work as L2ARC.

I have a different strategy for SLOG that involves abusing a RAID controller's CacheVault-protected write cache and a decent SAS drive. This provides low latency to the controller, full data protection, nearly unlimited write endurance, etc. but does also limit sustained throughput to the speed of the SAS drive. The cost to implementation is similar to the P3700, but the P3700 could be a more interesting choice. The endurance of the 400GB unit is rated at 4TB per day and I know we're not pushing that much at storage, so it's possible I might retask some existing hardware and try a P3700 once things settle down here.
 

zambanini

Patron
Joined
Sep 11, 2013
Messages
479
meanwhile at the LSI hq, screaming voices: jgreco is raping our product!
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
You can fondle my bits, baby. That santa outfit and green skin EXCITES me!
 

airflow

Contributor
Joined
May 29, 2014
Messages
102
I got the Intel P3600 today and tried it together with FreeNAS v9.3-STABLE-201412090314. Hardware is otherwise ASUS H87I-Plus, Intel Core i3-4130T, 2x 2.90GHz, Kingston ValueRAM DIMM Kit 16GB (ECC capable). I will exchange the mainboard soon to ASRock E3C226D2I to leverage ECC.

During boot, the NVMe-SSD is identified:

nvd0: <INTEL SSDPEDME400G4> NVMe namespace
nvd0: 381554MB (781422768 512 byte sectors)

In the GUI of FreeNAS, the device is shown in the "View Disks"-table with the correct size.
View_Disks.PNG


When trying to create a simple striped volume with just the nvd0-device, the Volume Manager hangs. From then on the GUI is not responsive any more, and I have to restart the box from command-line to let it recover.

Volume-Mgr-creation.PNG

Volume-Mgr-dead.PNG
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
That's cause you got desktop hardware! Ok, I'm only partly kidding. I'd expect desktop hardware to have good thorough support, although it may not work in FreeBSD. I'm curious to see what happens with the AsRock board.
 

airflow

Contributor
Joined
May 29, 2014
Messages
102
Thanks to the rapid feedback of FreeNAS-developer Xin Li my problem with the NVME-SSD is already solved. Obviously waiting for half an hour for the creation of the volume was not enough, as the initial TRIM-process takes such a long time.

I set tunable vfs.zfs.trim.enabled to 0.

After that I retried to create the new volume just like before - and it worked right away.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Interesting tidbit! Thanks for the followup.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I've got a Samsung XP941 that I'll be playing with in the next month or two. It just arrived but I'm still waiting on the PCIe card and a new server to try it in.

You can fondle my bits, baby.

"That was fast..." (to which I have a friend who'd invariably respond, "That's what she said!")

So I got around to installing the XP941 in a test box, using an adapter card, where it proceeded to do about:

725MB/sec writes (of /dev/zero) at 98% idle (dd reporting ~10% WCPU, gstat reporting 5500 ops/s)

1085MB/sec reads at 99% idle (dd reporting ~4% WCPU, gstat reporting 8400 ops/s).

I'd call that a nice L2ARC device. Not too expensive, great performance.

xp941.PNG
 

AgentZero

Dabbler
Joined
Jan 7, 2013
Messages
24
Thanks to the rapid feedback of FreeNAS-developer Xin Li my problem with the NVME-SSD is already solved. Obviously waiting for half an hour for the creation of the volume was not enough, as the initial TRIM-process takes such a long time.

I set tunable vfs.zfs.trim.enabled to 0.

After that I retried to create the new volume just like before - and it worked right away.

Just wondering - which form factor p3600 were you using - the actual PCIe card, or the 2.5 drive?
 

airflow

Contributor
Joined
May 29, 2014
Messages
102
Just wondering - which form factor p3600 were you using - the actual PCIe card, or the 2.5 drive?

The actual PCIe card. For the record, it didn't initially fit in my case and couldn't be correctly installed, as the device comes with a half-height bracket attached. My case only takes full-height PCIe cards. My local PC dealer just gave me a correct bracket for free, he had tons of it. It can be easily swapped by unscrewing two screws.

Another important piece of information about the performance of the drive. I did test that and couldn't achieve the promised speeds, by far I have to say. I researched and found that NVMe-support in FreeBSD is there and working but yet to be improved performance-wise. Read this thread to find out more about the issues, it contains info about nvme-tools, benchmarking-methods, and tuning-tips. The responsible developer of the FreeBSD-driver is also participating. I expect this to be sorted out in newer releases of FreeBSD/drivers - the technology is still quite new at the moment.
 

gork

Dabbler
Joined
Sep 16, 2015
Messages
12
I have encountered a related problem in testing; vfs.zfs.vdev.trim_on_init=0 does solve the problem of fast pool creation, however configuring nvme as a L2ARC (a job at which it ought to excel) requires that the entire device be cleared with a TRIM on pool import, and this is a blocking operation. It would be nice if L2ARC is trimmed in the background or some other type of optimization is made either in zfs or nvme drivers to optimize or background/queue this type of operation. Other than "dont use nvme for l2arc" are there any thoughts on best practices in this situation?
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
AFAIK situation with TRIMs on NVMe was improved/fixed in FreeBSD couple months ago (SVN rev 285767). It just needs to be merged to FreeNAS.
 
Status
Not open for further replies.
Top