M.2 PCIe adapters and X10SL7-F

[superazamigo]

I HAVE
Addonics AD3M2SPX4 M.2 PCIe (NGFF)
Supermicro X10SL7-F motherboard
Samsung M.2 PCIe x4 SSD SM951 256GB AHCI version

PROBLEM
The Samsung SSD with the Addonics M.2 adapter does not initialize correctly to be seen by any OS or the BIOS. I have tried several settings on the Supermicro BIOS to initialize it but nothing seems to allow it to be set up correctly. Fortunately the Addonics M.2 adapter has LED indicators for the status of the drives plugged into it (it also has a M.2 SATA SSD slots on the other side). The indicator flashes red consistently on normal bootup.

NARROWING DOWN A SOLUTION
In the course of trying to access the drive from either the EFI shell or a OS, I came across something that does at least partially solve the problem - the initialization routine from Parted Magic. Apparently this has a Linux initialization that scans the PCI bus and initializes the Samsung M.2 PCIe SSD correctly (error indication on Addonics card goes solid green like the other M.2 SATA SSD I have on the card). If I don't power cycle after that, the M.2. SATA SSD will remain accessible to any OS thereafter.

OTHER COMMENTS/OBSERVATIONS
Have tried removing all drives but the M.2 PCIe SSD, reset several different ways (different BIOS configurations), tried the x8 slot instead of the x16. None of these have the correct initialization for the SM951 to operate correctly. Tried contacting Supermicro, but nothing is returned from them (email support) - sure they say go to your reseller - but what if their "certified' reseller is Newegg, who doesn't have any support functions? Basically they seem to not want to hear about it unless you're buying a bulk amount of their product. On the flipside, Addonics wasn't really able to help because the cards themselves depend on the motherboards to be seen - the BIOS does the heavy lifting typically. In my case, I didn't expect the Supermicro motherboard to do anything but allow access to the M.2 PCIe SSD (no boot from drive option).

I also tried the other Addonics card, AD2M2S-PX4 to see if there was any difference - same problem, so it's related to the motherboard's initialization, not the card itself (the cards are pretty straightforward PCIe to mini-PCIe adapters anyhow).

Here's the cards that Addonics supplies for the M.2 NGFF adaption:
http://www.addonics.com/category/ngff.php

 

[Ericloewe]

Do you have the 3.0 BIOS installed?

 

[superazamigo]

Do you have the 3.0 BIOS installed?

Yep - latest version 3.0 - known as x10sl75_424.zip on the Supermicro website. My board came with it since I purchased it recently.

 

[joeschmuck]

I haven't read a lot on these PCI-e M2 devices but some are not very compatible I guess since they are fairly new technology.

I don't think you can state it's the MB based off the fact that you used a different adapter card. You need to put the adapter and M2 SSD into a different computer and it must work in order to make that claim. But the Parted Magic does lend a bit of credence to the Supermicro MB BIOS being the issue here but without having this in my hands, I couldn't say for certain.

If you believe you have narrowed this down to be a Supermicro issue then you should visit their forums and troubleshoot the problem there as this is not a FreeNAS issue. Or if you feel it is a FreeNAS issue, then look to the FreeBSD forums but ensure you can prove other OS's do not fail like Ubuntu or Fedora or Suse and even maybe Windoze.

I'd like to see the result of your work and I wish you good luck on this journey.

 

[Ericloewe]

PCI-e devices are generally expected to provide their own extension ROMs that hook into BIOS interrupts.

Honestly, I have no idea what the expected situation is with PCI-e SSDs. Is the BIOS supposed to look for and initialize AHCI and/or NVMe or are drives expected to provide extension ROMs?

 

[superazamigo]

Based on my research, it's the BIOS that does the lifting. I don't have a Z9x board to try it out on - but these boards typically support M.2 PCIe devices in BIOS. The BIOS even has boot functionality in many cases - giving full initialization for the M.2 PCIe device. Hence the BIOS is probably the cause of the non-initialization of the M.2 PCIe SSD.

I have also dug into the Linux kernel concerning the PCIe bus, and the initialization that's being performed by Parted Magic is standard for the OS. Apparently a similar initialization could be set up by the BIOS - but it is missing. The X10SL7 is a 2013 era board, so this is at least a year before PCIe SSDs were introduced - hence the lack of support.

I'll see what I can get on the Supermicro forum, I was discussing it here because we deal with a lot of storage systems and use the exact same hardware. In the long haul, this should be a feature request to Supermicro that should be added to the next BIOS update. SuperMicro unfortunately is very unwilling to deal with us common users, just the big guys. I haven't even heard back on my first technical support request.

I am sure though that more of us will be using M.2 PCIe adapters on our Supermicro (and other) motherboards. It's unfortunate that the M.2 PCIe adapter cards don't have a built-in ROMs extensions that do the BIOS interrupt hooks. This would solve the interconnect issue without having to update every MB BIOS - greatly improving compatibility. I'll also throw Addonics a question on that one..

 

[cyberjock]

I am sure though that more of us will be using M.2 PCIe adapters on our Supermicro (and other) motherboards.

I don't agree. Using those adapters is, in my opinion, lame. You gain very little by moving from SATA controller to the PCIe bus. Sure, you gain a little latency (at least, in theory), but the reason to go with PCIe is for the larger bandwidth. You can't get that from an SSD that is still SATA 6Gb/sec. So the main advantage is basically gone. :(

The other reason...

It's unfortunate that the M.2 PCIe adapter cards don't have a built-in ROMs extensions that do the BIOS interrupt hooks. This would solve the interconnect issue without having to update every MB BIOS - greatly improving compatibility. I'll also throw Addonics a question on that one..

BINGO! That's going to be a killer for many people. So even if you ignore my above comment about there not being an advantage, the fact that you basically have to spend the money on the hardware just to find out if it will work is a bit of a let down.

That's like when Nancy Pelosi said "You have to pass it to find out what is in it!" Sorry, people like to know what they are getting into before money is spent. Buying it only to find out it isn't compatible is pretty much a buzz kill for most of us. I surely went shopping for FreeNAS compatible hardware so I didn't have to build a half-dozen systems to get one that worked.

 

[Ericloewe]

The adapter cards cannot have the ROM, as they are passive adapters. I'm not sure bridges can even supply boot ROMs and still act like PCI bridges.

 

[superazamigo]

I don't agree. Using those adapters is, in my opinion, lame. You gain very little by moving from SATA controller to the PCIe bus. Sure, you gain a little latency (at least, in theory), but the reason to go with PCIe is for the larger bandwidth. You can't get that from an SSD that is still SATA 6Gb/sec. So the main advantage is basically gone. :(

The other reason...



BINGO! That's going to be a killer for many people. So even if you ignore my above comment about there not being an advantage, the fact that you basically have to spend the money on the hardware just to find out if it will work is a bit of a let down.

That's like when Nancy Pelosi said "You have to pass it to find out what is in it!" Sorry, people like to know what they are getting into before money is spent. Buying it only to find out it isn't compatible is pretty much a buzz kill for most of us. I surely went shopping for FreeNAS compatible hardware so I didn't have to build a half-dozen systems to get one that worked.

Actually there's two reasons to use these types of cards
The first being space savings - you can now stick a SSD on a card (or slot on the motherboard) and have full solid state transfers without using a 2.5" space. PCIe adapters like the one I have 3 slots for SSDs, 1 for the PCIe x4 interface, and 2 for SSDs that get power from the PCIe slot. This is 3 available drive spaces that normally would have taken up 3 2.5" slots. The second part is speed - I'm getting >2GB/s reads and just over 1GB writes on the SM951. Sure IOPs are not as good as an Intel 750, but I'm getting similar read speeds so it doesn't matter (I don't need the IOP performance so much) at about 1/2 the cost.

I also have my workaround which lets this adapter card work normally... having to seek compatibility is part of using cutting edge hardware. If it's demanded, then manufacturers will comply eventually. I have heard that the next series of Supermicro boards will have PCIe x4 slots standard. It's just becoming a standard interface with most motherboards.

 

[superazamigo]

The adapter cards cannot have the ROM, as they are passive adapters. I'm not sure bridges can even supply boot ROMs and still act like PCI bridges.

After explaining the situation completely, customer support at Addonics has said they'll pass it on to engineering for consideration in a future edition of their cards (adding a ROM so that the card can work with BIOS/potentially boot). Apparently I'm not the only one concerned with power-on compatibility with these cards - there's several others on other forums working the same issue. The 2nd generation should be much more compatible with a wider range of boards that do not directly have BIOS support for PCIe SSDs.

Of course, on the other hand, I found that the Kingston Hyper-X PCIe SSDs addressed the bootrom issue directly - by putting it on their board. It may be that the M.2 PCIe SSD manufacturers address it before the adapter card makers do - either way, it's a win for the user.

So there is a few other options to use even if the SM951 isn't perfect. I just used the Samsung SM951 because it had the best speed ratings of any available SSD right now. Of course, Samsung is going one up on that with the 950PRO later this month.. It is V-NAND, so it'll sustain hundreds of TBs of data.. they back that with a 5 year warranty too.

 

[Thinkcat]

Would this one work? It achieves only SATA speeds, but it does have a BIOS for booting.

http://www.ebay.com/itm/SINTECH-M-2...963270?hash=item33abeffa86:g:V34AAOSw5ZBWNJLB

 

[Ericloewe]

Would this one work? It achieves only SATA speeds, but it does have a BIOS for booting.

http://www.ebay.com/itm/SINTECH-M-2...963270?hash=item33abeffa86:g:V34AAOSw5ZBWNJLB

Well, that's a crummy SATA controller card, with the odd feature that it exposes one port as an M.2 slot. Obviously, it will only work with M.2 SATA cards, not PCI-e M.2 cards.
Most importantly though, it's a horrible SATA controller. Avoid it.

 

[jgreco]

Using those adapters is, in my opinion, lame. You gain very little by moving from SATA controller to the PCIe bus.

Not lame at all. Quite frankly, when you're doing this stuff professionally, you're always fighting this insipid battle of insufficient space in the chassis, or, more often, imbalanced space. You might only need a slot or two of your PCIe capacity on a filer. Now look at a platform like the X10SRL with an E5-1650 in a SC216BE16 chassis. You put in boot on redundant SATA-DOM, a HBA to the expander and to the two rear facing bays. You're left with five PCIe slots empty and eight mainboard SATA ports available. So ... which is lame: do you ignore the potential to do more with this system, or do you proceed to fill four of those slots with triple-M.2 adapters and be able to cram a dozen more SSD storage devices into the filer at minimal additional cost?

Actually there's two reasons to use these types of cards
The first being space savings - you can now stick a SSD on a card (or slot on the motherboard) and have full solid state transfers without using a 2.5" space. PCIe adapters like the one I have 3 slots for SSDs, 1 for the PCIe x4 interface, and 2 for SSDs that get power from the PCIe slot. This is 3 available drive spaces that normally would have taken up 3 2.5" slots. The second part is speed - I'm getting >2GB/s reads and just over 1GB writes on the SM951. Sure IOPs are not as good as an Intel 750, but I'm getting similar read speeds so it doesn't matter (I don't need the IOP performance so much) at about 1/2 the cost.

Yup. It's been a nice place to stick the 941 I've got as an L2ARC on the VM filer right now.

 

[BERKUT]

I'm using this adapter "HyperX Predator PCIe SSD" and it work perfect, but you can use any other analogue.
Not forget in SUPERMICRO bios setup PCI-e line (x8x8 or x4x4x4x4) if you use more than 1 device on 1 PCI-e line. Default in BIOS enabled x16.

 

[Algwyn]

Has this issue been resolved? Can we expect to have this issue with other SuperMicro motherboard (X11SSH-F or X11SSM)
Jgreco you seem to have a good experience with this adapter.
With which motherboard did you test it?

 

[Stux]

PCI-e devices are generally expected to provide their own extension ROMs that hook into BIOS interrupts.

Honestly, I have no idea what the expected situation is with PCI-e SSDs. Is the BIOS supposed to look for and initialize AHCI and/or NVMe or are drives expected to provide extension ROMs?

NVMe drives need BIOS support. They don't have extension roms. Most PCIe Adapters should just be glorified pass-through cards.

 

[Algwyn]

NVMe drives need BIOS support. They don't have extension roms. Most PCIe Adapters should just be glorified pass-through cards.

Should we understand this statement as having the following consequences:

• on older motherboards, without NMVe support, the adapter will not work, unless an updated BIOS with NMVe support has been released by SuperMicro
  
• on newer motherboards (e.g. X11SSH-F, X10SRM-F) with a M2 expansion slot, the native BIOS support NVMe, adding another M2 slot through the adapter should not raise compatibility issues
• variants of the above M2 motherboards, may also support the adapter if they share the same BIOS (e.g. X11SSM vs X11SSH)

Is it possible to confirm this?

 

[Ericloewe]

on older motherboards, without NMVe support, the adapter will not work, unless an updated BIOS with NMVe support has been released by SuperMicro

For UEFI boot, you might be able to work around this by loading an EFI executable that provides an NVMe driver and proceeds to boot NVMe devices.

on newer motherboards (e.g. X11SSH-F, X10SRM-F) with a M2 expansion slot, the native BIOS support NVMe, adding another M2 slot through the adapter should not raise compatibility issues

The physical interface is irrelevant. U.2, M.2, PCI-e slot are all alike, as far as the BIOS is concerned (M.2 using PCI-e instead of SATA, obviously).

variants of the above M2 motherboards, may also support the adapter if they share the same BIOS (e.g. X11SSM vs X11SSH)

All Skylake boards should have NVMe support. Most Haswell/Broadwell boards should support NVMe with the latest updates, too.

 

[Stux]

on older motherboards, without NMVe support, the adapter will not work, unless an updated BIOS with NMVe support has been released by SuperMicro

NVMe BIOS support means being able to boot from an NVMe drive.

As far as I know. NVMe drives should work as data drives in any system with PCIe slots as long as the OS has an NVMe driver. NVMe is very new, and I think FreeNAS 9.10+ has the driver.

 

[Ericloewe]

NVMe drives should work as data drives in any system with PCIe slots as long as the OS has an NVMe driver.

Right, they're just regular PCI-e devices. Any OS with PCI support and an NVMe driver will work.

NVMe is very new, and I think FreeNAS 9.10+ has the driver.

It's been included since 9.3.1 at least, IIRC. Possibly some later 9.3 releases.