just a heads up, finally ASUS w680 available.

[Ericloewe]

Yeah, Intel's desktop platforms do not support x4/x4/x4/x4 - that's a Xeon E5/Xeon Scalable feature.

 

[a.dresner]

More specifically about those last 2 PCIe slots, too bad they are x4, not x8 - I would take PCIe 2 or 3 with 8 lanes vs 4 lanes

At the end of the day, I don’t see this as my forever board, and it could be repurposed around my life if a server platform comes out in the next few years that has the quicksync and expansion many of us are looking for. Or maybe this board does it all and is stable? The very good advise here by the mods is all accurate and I have an old x9 E5 on my bench if this board becomes an issue.

Started working on my case yesterday- fans installed, both CPU and memory are shipped, so have the drives. I hope to be up and running by months end :) if this board sells enough there might be a decent user base out there to support each other

Which PSU did you guys end up going with?

 

[a.dresner]

Can any of you tell which Intel NIC Chip Sets are in use on our board? I am still a few weeks before I can power it up

Okay, I can answer my own question... I went into the driver section of the board. GOOD news, they are i226.

 

[Kthwaits]

Hi All, I've been lurking on this forum and following along with this thread as I wait for my Pro WS W680-ACE IPMI to arrive (I'm actually an UnRAID user as opposed to TrueNAS, but I imagine my use case is similar to most of yours here).

Forgive me if this is a silly question but I'm wondering if someone could help validate my understanding of the PCIe slots/lanes on this board and how they're allotted. I've gone as far as reading through the user manual I've found online as part of my research but I'm still not 100% confident I've understood correctly. Could someone help me clarify if this PCIe configuration would work?

Slot Name (as designated in MB manual):	Utilized By:	Speed & Number of Lanes Used:
PCIEx1(G3)	IPMI Card (included with the MB)	PCIe Gen 3.0 x1
PCIEx16(G5)_1	ASUS HYPER M.2 X16 GEN 4 CARD (driving two Samsung 980 Pro NVME M.2 2TB)	PCIe Gen 4.0 x8
PCIEx16(G5)_2	LSI SAS 9300-8i	PCIe Gen 3.0 x8
PCIEx16(G3)_1	?	PCIe Gen 3.0 x4
PCIEx16(G3)_2	?	PCIe Gen 3.0 x4
M.2_1	Samsung 970 EVO Plus NVMe M.2 1TB	PCIe Gen 3.0 x4
M.2_2	Samsung 970 EVO Plus NVMe M.2 2TB	PCIe Gen 3.0 x4
M.2_3	Samsung 970 EVO Plus NVMe M.2 2TB	PCIe Gen 3.0 x4

I have built with several other boards in the past where the use of certain PCIe slots disabled one or more of the M.2 slots (or vise-versa), but I wasn't able to see anything about such limitations in the user manual for the Pro WS W680-ACE. Does this mean this that I could successfully take advantage of all the slots/lanes as shown in the table? And if those two currently unused slots are still available, could I theoretically populate them with additional M.2s using two individual PCIe 3.0 x4 adapters?

Any potential gotchas with such a configuration?

 

[omeganot]

The top M.2 slot connects directly to the CPU. Alder Lake CPUs have 4x PCIe 4.0 lanes for this purpose, where the 16x PCIe 5.0 lanes connect to the PCIe 5.0 slots on the board. The other two M.2 slots are going through the W680 chipset, connecting over the DMI link. The manual states this on pages vii and viii. Note that all of those non-PCIe 5.0 slots, the second and third M.2 slot, the SATA ports, and the SlimSAS port all share that DMI link, so that's 8x PCIe 4.0 bandwidth.

Whether using them disables another slot on the board, though, that's something I do not know. You're right in that the manual seems woefully silent on the subject.

 

[Etorix]

Why would you share lanes, and potentially disable slots, when there's enough I/O available?
Shame on Asus for not providing a block diagram of the motherboard tough.

And if those two currently unused slots are still available, could I theoretically populate them with additional M.2s using two individual PCIe 3.0 x4 adapters?

Yes. And don't forget the SlimSAS connector for an additional PCIe 4.0x4.
If you're absolutely desperate for even more drives, you could try this
Linkreal Quad M.2 NVMe SSD PCIe x16 Adapter with PLX 8747
or even that
Linkreal 8 Port U.2 to PCIe x16 SFF-8643
These have PLX PCIe switches, so they should work with less than x16 lanes and still making all devices available—with some oversubscribing, of course.

But if you really want to fit in as many NVMe drives as possible, maybe you should have gone for EPYC or Xeon Scalable to begin with…

 

[Kthwaits]

Thank you both for helping to clarify this for me. I'm still unsure how the chipset-provided lanes are allotted to the different slots but I'm clearly lacking some basic understanding of how that all works so I'll do some additional research to try to educate myself before asking any more questions.

The reason I went with a W680 board is because I wanted support for both Intel integrated graphics (for Plex) as well as DDR5 ECC. So this board is one of very few choices I had that meets those requirements. The goal isn't so much about fitting as many NVMe drives as I can, I was mostly just hoping to make the best use of all the available PCIe lanes as possible

 

[tsm37]

Hi, I want to add a couple more comments since I have this W680 board.

I think it's a good decision that you plan to put the LSI SAS 9300-8i on PCIEx16(G5)_2 because this x16 slot pulls 8 pcie lanes from the cpu. As one member mentioned, the W680 chipset limiting factor is the DMI, which is at pcie4 x8 bandwidth, but whether it's gonna be a bottleneck, it will depends on your use case and typical workload.

Here is the implication: the mobo has four sata ports and one slimSAS port. The slimSAS port can be put into SATA mode and support four more SATA drives. In total, the chipset allows eight SATA drives. On top of it, M.2_2 and M.2_3 pull pcie4 x4 lanes EACH along with other components like built-in NICs, USB ports, etc. - all of them need lanes/bandwidth on the W680 chipset. The LSI SAS 9300-8i on PCIEx16(G5)_2 will bypass the chipset and talks directly with the CPU. In my case, I like mirrors. I have one hdd/sdd in the mirror connected to a SATA port via the chipset and have the second hdd/sdd in the same mirror connected to a LSI 9207-8i on PCIEx16(G5)_2. This way, I feel the load will be better distributed and probably doesn't saturate the DMI. Something for you to consider.

I also looked into the Asus M2 hyper card, but unfortunately, you can only get 2 out of 4 slots to work on this motherboard because it cannot do 4x/4x/4x/4x bifurcation. As you probably know, asus has a webpage that documents how many slots it will work on this mobo. Also, in my opinion, the design of this hyper card is bad - it's way too long. There is no need to be this long.

For PCIEx16(G3)_1 and PCIEx16(G3)_2 where you have a question mark, perhaps you can consider a 10Gbps NIC card down the road, which needs x4 lanes.

If I remember correctly, W680 has a total of 28 lanes. Please refer to the chipset block diagram attached.

 

[a.dresner]

Hi, I want to add a couple more comments since I have this W680 board.

Thank you for those comments. I'm also living with this board, but PSU shipment is late so I can't power it up yet.

Can you help further clarify:
2 x PCIe 5.0 x16 slots are on the CPU. Right place for HDD controllers or a GPU.

2 x PCIe 3.0 x16 slots are on the W680 chipset.. It says x16, does that mean the slot supports x16, x8, and x4 lanes? Manual says supports x4, wondering if that means it only supports x4 or x16 and x4.

 

[tsm37]

Thank you for those comments. I'm also living with this board, but PSU shipment is late so I can't power it up yet.

Can you help further clarify:
2 x PCIe 5.0 x16 slots are on the CPU. Right place for HDD controllers or a GPU.

2 x PCIe 3.0 x16 slots are on the W680 chipset.. It says x16, does that mean the slot supports x16, x8, and x4 lanes? Manual says supports x4, wondering if that means it only supports x4 or x16 and x4.

Yes, sure.

PCIEx16(G5)_1 and PCIEx16(G5)_2 both obtain pcie lanes from the CPU. PCIEx16(G5)_1 is typically the slot where people would install a GPU.

Scenario #1:
If you install, let say, a RTX 3000 series GPU to PCIEx16(G5)_1 and leave PCIEx16(G5)_2 unused, then the GPU will receive x16 lanes from the CPU.

Scenario #2.
If you install a RTX3000 series GPU to PCIEx16(G5)_1 and for example, an HBA card like LSI9207-8I or LSI 9300-8I or equivalent to PCIEx16(G5)_2, then the CPU will give x8 lanes to PCIEx16(G5)_1 for the GPU and also x8 lanes to the HBA card on PCIEx16(G5)_2. This is called 8x/8x bifurcation in which the W680 is capable of. 8x lanes at pcie4 is really fast - 16 gigabytes per second. PCIEx16(G5)_1 and 2 are even capable of pcie5. 8x at pcie5 is 32 gigabytes per second - extremely fast. Please keep in mind that your limitation will be the HBA card itself because, for example, 9207-8i or 9300-8i is pcie3 x8 lanes.

In reality, you can do scenario #2 and this is because it's likely people will just use a GPU for the decoder under truenas scale, for decoding, x8 pcie lanes are more than enough in my opinion.

To your second question, PCIEx16(G3)_1 and PCIEx16(G3)_2 are physically x16 slots but each slot is only x4 speed. This kinda makes sense. these two slots obtain pcie lanes from the chipset, which has a maximum of 28 lanes (i believe). the chipset has to do a lot of things: sata, usb, two NICs, two nvme m2 slots, IPMI, etc. you only have so much lanes to work with on the chipset.

 

[tsm37]

Lastly, I have been using Asus mobo's for years and am familiar with their manual setup. For this mobo, I don't think using the other two nvme m2 slots will disable the SATA ports vice versa. Please keep in mind that this is a workstation mobo and not a gaming mobo. Gaming mobos usually have this kind of disabling and asus manual is often very clear on this.

 

[a.dresner]

Great response..thank you.

I also have not seen any notes about using 1 port disabling another, I have seen that in other manuals as I was shopping around.

I wonder what happens when I start to pass devices through... someone mentioned the NIC card stopped working? Not sure his setup...

Slot Name (as designated in MB manual):	Utilized By:	Speed & Number of Lanes Used:
PCIEx1(G3)	IPMI Card (included with the MB)	PCIe Gen 3.0 x1
PCIEx16(G5)_1	No plan yet, maybe a GPU passed through for a PC	?
PCIEx16(G5)_2	LSI SAS 9207-8i	PCIe Gen 3.0 x8
PCIEx16(G3)_1	? Maybe NIC
PCIEx16(G3)_2	?
M.2_1	Gen 4 NVME (Pass through for VM)	PCIe Gen 3.0 x4
M.2_2	Gen 3 NVME (Cache)	PCIe Gen 3.0 x4
M.2_3	Gen 3 NVME (Cache)	PCIe Gen 3.0 x4

Still unsure if I will go with UNRAID, TRUENAS, or Proxmox with TRUENAS virtualized or some combo I haven't thought about yet. I seem to go in circles daily... my PSU is out for delivery so I get to power up my rig and start testing shortly.

There is another option to send that iGPU into VMs for decoding and graphics, you can create virtual iGPUs. https://3os.org/infrastructure/proxmox/gpu-passthrough/igpu-passthrough-to-vm/ Hoping to test this out.. send 1 vGPU to Plex, 1 to Windows VM.

 

[Etorix]

I also have not seen any notes about using 1 port disabling another, I have seen that in other manuals as I was shopping around.

That depends how lanes are allocated.
Here the W680 chipset is exposing 2*4 + 1 as PCIe slots, 2*4 as M.2 and 4 as SlimSAS, plus 2*1 to the i226 NIC and 1 to CNVi slot; total 24 lanes (if I have not overlooked anything), which is within the capacity so there's no need to share lanes.

You may compare with the AsRockRack W480D4U-2L2T/G5 (a true W680 server board). Here, as shown by the block diagram on page 21 of the manual, 4 lanes are exposed either as x4 PCIe slot or through an OCuLink connector.

Still unsure if I will go with UNRAID, TRUENAS, or Proxmox with TRUENAS virtualized or some combo I haven't thought about yet. I seem to go in circles daily... my PSU is out for delivery so I get to power up my rig and start testing shortly.

If your data is important, running TrueNAS bare metal is the easiest and safest option.

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

 

[zettiness]

Hi all, been following this thread for a while.

I am putting together a workstation based on this motherboard that would also serve as NAS running TrueNAS in a VM in Windows (or Windows under TrueNAS or Ubuntu -- something that's probably more appropriate to discuss in a different thread).

I have ordered Kingston DDR5 Unbuffered ECC 32GB [KSM48E40BD8KM-32HM] x2 as it's listed on Kingston site as compatible (and here in the UK just had a big price cut @ £127 per module).

My question is what WIFI modules are suited for the the dedicated M.2 slot (Key E)? Is it CNVio2 compatible? Motherboard manual doesn't say and Asus support has been such a time waste so far..

Also, any idea on what expansion cards would work with the Thunderbolt header? Maybe I could run external Thunderbolt enclosure for the NAS array instead of keeping all the disks inside the case (nowadays there aren't many cases that will factor in more than 2 HDDs).

 

[Javafanboy]

Anybody that has tried running this board with four 32GB ECC sticks (128GB) and in that case what RAM did you use and what speed/settings did they work reliably with?

 

[tsm37]

Anybody that has tried running this board with four 32GB ECC sticks (128GB) and in that case what RAM did you use and what speed/settings did they work reliably with?

I'm still on 2 x 32GB ECC DDR5, but I plan to add two more sticks in the coming months. I feel good about it and don't think I will run into any issues. This is a NAS box, so there is really no need for any memory overclock whatsoever. I'm using Crucial (Micron)'s MTC20C2085S1EC58BA1R 4800 DDR5 CL40. Yes, it's slow and latency is high, but DDR5 ECC is still relatively new on the market and again it's not a gaming pc. Those are very low speeds in the DDR5 world that should be widely supported, but it's best to check the Asus website.

As for this mobo, it can do 4 x DIMM, max 128GB DDR5 4800 ECC memory as per their manual. Not sure if you can go higher (i.e. the speed)

 

[Javafanboy]

I'm still on 2 x 32GB ECC DDR5, but I plan to add two more sticks in the coming months. I feel good about it and don't think I will run into any issues. This is a NAS box, so there is really no need for any memory overclock whatsoever. I'm using Crucial (Micron)'s MTC20C2085S1EC58BA1R 4800 DDR5 CL40. Yes, it's slow and latency is high, but DDR5 ECC is still relatively new on the market and again it's not a gaming pc. Those are very low speeds in the DDR5 world that should be widely supported, but it's best to check the Asus website.

As for this mobo, it can do 4 x DIMM, max 128GB DDR5 4800 ECC memory as per their manual. Not sure if you can go higher (i.e. the speed)

I have similar view of the speed - just wanted to confirm if anybody managed to run 4 at 4800 and not having to downgrade further. I think ASUS information about supported ECC memory is very poor - extremely few sticks reviewed (when I looked only some Innodisk that is more or less impossible to source in Europe) and no info about if even that one worked with 1, 2 or 4 sticks...
One would have imagined that they should test more generally available modules that now exists from Crusial/Micron, Kingston (this one is supposedly compatible according to vendors site) etc...

 

[Jamberry]

I am facing the same problem. First wanted to go with a W680 board but I hear @jgreco and don't wanna beta test hardware.
Still can't decide between the A2SDi-8C-HLN4F and the X13SAE + i3-12500. I don't trust the E cores yet, this one only has P cores. I wanna use 8 HDDs and two SSDs for metadata and small files. Maybe I wanna add 10Gbit or SLOG later on.

• Both are not available to purchase
• Both are similar priced
• The X13 would have to start with 32GB and get more DIMMS later on, because DDR5 is crazy expensive right now (140$ for 32GB)
• X13 would offer 3x NVME instead of SATA
• X13 would offer more than just one PCI 3.0 x4 slot for later upgrades

 

[Etorix]

A2SDi is perfect for a pure, low-power, file-serving server and takes cheap DDR4 RDIMM. But it cannot accommodate more than two NVMe SSDs (one onboard, one in PCIe slot) and you must decide at purchase if you want 10 GbE onboard.

X13SA_ (SAE is a workstation board) would be more appropriate if the server is to run apps/VMs on top of NAS duties. Consider previous X12 or X11 generations instead of X13.

 

[Jamberry]

Thanks for the hint, hopefully the X11SDV-4C-TP8F will be available again.
That would offer 12 SATA Ports and 10Gbit SFP+ for 800$

My worry is, that because these motherboards are "old", I am not sure if they will ever become available again.