How to add blk dev (like sd*) to vm in Truenas scale ?

[Amudhan]

Hi,

I am using TrueNAS-SCALE-22.12.3.2, have created ubuntu VM. Now, I am looking to add physical disk block device (assuming /dev/sdk) in host machine to VM.
I couldn't find a way to add it in documentation, is it doable in Truenas scale ?

 

[morganL]

Generally, you add a zvol .. more flexible, manageable and scalable than a physical drive.

 

[NickF]

If you insist on block device access, check out the Datacenter in a box link in my signature.
The TLDR; Is the only way you can do what you are trying to do is to pass through a pcie addon card like an LSI HBA or an m.2 NVME drive.

 

[Amudhan]

Generally, you add a zvol .. more flexible, manageable and scalable than a physical drive.

Yeah, you are right. But I want to use attach disks in my VM.
I read somewhere we can do it qemu or virsh but when run command `virsh list` I get error "error: failed to connect to the hypervisor".
is there any other way to add disk to VM in TrueNAS scale ?

 

[sretalla]

is there any other way to add disk to VM in TrueNAS scale ?

PCI passthrough of a disk controller which is connecting the disk(s) you want in the VM.

 

[samarium]

Even if you could do so, /dev/sdk is a recipe for problems, you should use /dev/disk/by-id/... something more descriptive and not subject to a name change on the next reboot.

As for accessing the device, it is probably possible to pirate the qemu call, prep the device, rewrite the arguement list, then recall qemu. None of it will be easy, safe or supported, and will be fragile and probably break in the future. I thought about it for different purposes for a while, and ultimately decided it was not worth it for me to chase that down the rabbit hole.

 

[Amudhan]

PCI passthrough of a disk controller which is connecting the disk(s) you want in the VM.

I cannot use PCI passthrough to expose HBA to VM because zfs pool is created using some of the (12) disks attached in HBA.
If I attach HBA to VM my zfs pool is lost. That's why I just wanted to try to attach disk directly using by-id or disk serial number.

 

[Amudhan]

Even if you could do so, /dev/sdk is a recipe for problems, you should use /dev/disk/by-id/... something more descriptive and not subject to a name change on the next reboot.

As for accessing the device, it is probably possible to pirate the qemu call, prep the device, rewrite the arguement list, then recall qemu. None of it will be easy, safe or supported, and will be fragile and probably break in the future. I thought about it for different purposes for a while, and ultimately decided it was not worth it for me to chase that down the rabbit hole.

Yeah, that's a good point to use by id I was planning to use it. Just for reference posted drive /dev/sdk.

I wanted to try and test how the performance will be when disk are attached directly in VM vs zvol.

 

[samarium]

Yeah, that's a good point to use by id I was planning to use it. Just for reference posted drive /dev/sdk.

I wanted to try and test how the performance will be when disk are attached directly in VM vs zvol.

Add a cheap HBA to the system, so you can then pci pass thru that device.

 

[NickF]

Yes, unfortunately there is no good, safe option to do what you are trying to do without sending a PCIE device through to the guest.

 

[jyavenard]

Yes, unfortunately there is no good, safe option

what is your definition of safe?

I've been running VMs for over a decade in combination with physical disks. It's the easiest way to virtualise existing machines: take the existing disk.
It's been a way of setting VMs by VMWare for as long as they've been offering VM products (and all others). TrueNAS scale underlying KVM allows it, TrueNAS core's bhyve allows it.

Everyone is doing it, but somehow because truenas GUI isn't offering the possibility (still doable manually luckily) it must be because it's unsafe?

Having to use a dedicated HBA for each VM is a rather surprising choice.

 

[sretalla]

It's the easiest way to virtualise existing machines: take the existing disk.

It sounds more to me like the easiest way to run out of capacity on your hypervisor and get no benefit from features that allow aggregation of capacity or performance from multiple disks.

But those are your disks to waste as you please. I get the point about P2V migration.

 

[morganL]

what is your definition of safe?

I've been running VMs for over a decade in combination with physical disks. It's the easiest way to virtualise existing machines: take the existing disk.
It's been a way of setting VMs by VMWare for as long as they've been offering VM products (and all others). TrueNAS scale underlying KVM allows it, TrueNAS core's bhyve allows it.

Everyone is doing it, but somehow because truenas GUI isn't offering the possibility (still doable manually luckily) it must be because it's unsafe?

Having to use a dedicated HBA for each VM is a rather surprising choice.

TrueNAS is mostly focussed on using ZFS to virtualize storage. Easy to create and protect virtual disks.
The dedicated HBA is possible, but not commonly used.

 

[NickF]

I've been running VMs for over a decade in combination with physical disks. It's the easiest way to virtualise existing machines: take the existing disk.
It's been a way of setting VMs by VMWare for as long as they've been offering VM products (and all others). TrueNAS scale underlying KVM allows it, TrueNAS core's bhyve allows it.

Just dd the physical disk to a zvol? I totally hear you here, but @sretalla is right, he said it nicer but - thats kind of a lazy way to do it.

I've done both of those things with random old HVAC servers sitting in closets for 15 years. I've even used Windows/VMWare P2V agents. There are plenty of tools that exist to solve this problem.

Also, If you want to unsafely give a full disk to a VM, you can make a pool of one disk with one large ZVOL.