ZFS: vdev state changed

[Goose]

I get these, occasionally.

Nov 28 03:45:00 mother-goose ZFS: vdev state changed, pool_guid=4753035133703495120 vdev_guid=15343288276930064957
Nov 28 03:45:00 mother-goose ZFS: vdev state changed, pool_guid=4753035133703495120 vdev_guid=17738397629967930008

I read on a few threads that VMs might, perhaps, be the cause of this but most of the posts mention that it could be to do with disks that are set to powersave. Mine aren't but I am trialling a Windows 10 VM and maybe that is the link. I also have a couple of iSCSI drives also running off physical machines and a VM running on VBox on a laptop I had lying around.

No errors on any of the host OSes.

I'm happy to switch off the FreeNAS VM to see if the error goes but that isn't a good test as the errors are that infrequent.

Is there a way to see what vdev was changed and what it was being used by at the time?

Cheers,

Goose

 

[droeders]

I get these, occasionally.

Nov 28 03:45:00 mother-goose ZFS: vdev state changed, pool_guid=4753035133703495120 vdev_guid=15343288276930064957
Nov 28 03:45:00 mother-goose ZFS: vdev state changed, pool_guid=4753035133703495120 vdev_guid=17738397629967930008

I've seen similar messages in my FreeBSD machines during pool scrubs. Can you look at your scrub schedule and see if the timestamps match up?

 

[Goose]

I was just about to reply "no" as I my data scrub happens on the 1st and 15th. The I thought "I wonder about the boot thumb drives". Sure enough "Last Scrub Run on: Tue Nov 28 03:47:08 2017"

Next question is should they happen during a scrub?

 

[SweetAndLow]

Hardware specs and freenas version? How are your disks connected?

 

[Goose]

Supermicro Motherboard X10SLL-F with 32GB of EEC RAM with a pair of SanDisk Cruzer 8GB USB Flash Drives to boot from. FreeNas 11.0-U4

 

[droeders]

It appears these showed up in FreeBSD 11.0 when some log levels were changed. I think this is expected behavior and nothing to be concerned with. See here for some discussion:

https://lists.freebsd.org/pipermail/freebsd-fs/2017-April/024679.html

 

[Goose]

Just as a confirmation my main data pool scrubbed this morning and I have this:-

Dec 1 02:00:00 mother-goose ZFS: vdev state changed, pool_guid=8043956164769476261 vdev_guid=2482936001401079796
Dec 1 02:00:00 mother-goose ZFS: vdev state changed, pool_guid=8043956164769476261 vdev_guid=2244538289072489857
Dec 1 02:00:00 mother-goose ZFS: vdev state changed, pool_guid=8043956164769476261 vdev_guid=3013460537430833814
Dec 1 02:00:00 mother-goose ZFS: vdev state changed, pool_guid=8043956164769476261 vdev_guid=9407425694745527375

Interestingly there are 4 entries, I have 4 disks. When I posted originally it was my boot pool which has 2 "disks" and there were 2 entries. It would be logical to assume then that part of the scrub process is to close and reopen each device which leads to the state change.

 

[nojohnny101]

I am receiving these as well. I have a 6 drive pool with one raidz2 vdev.

Code:

Dec  1 03:00:01 tank ZFS: vdev state changed, pool_guid=7242467664024359546 vdev_guid=14736167918695098525
Dec  1 03:00:01 tank ZFS: vdev state changed, pool_guid=7242467664024359546 vdev_guid=12167762758278325982
Dec  1 03:00:01 tank ZFS: vdev state changed, pool_guid=7242467664024359546 vdev_guid=7740896019147825754
Dec  1 03:00:01 tank ZFS: vdev state changed, pool_guid=7242467664024359546 vdev_guid=18019470686639070827
Dec  1 03:00:01 tank ZFS: vdev state changed, pool_guid=7242467664024359546 vdev_guid=3612702732018427685
Dec  1 03:00:01 tank ZFS: vdev state changed, pool_guid=7242467664024359546 vdev_guid=16834649809563987411

It corresponds to the time a scrub was initiated.

I'm running 11.0-U4 (54848d13b).

 

[SweetAndLow]

If no one provides how their disks are attached or their hardware info this thread is pointless.

 

[Goose]

Well, the USB flash drives are connected to the USB ports presented on the I/O panel on the mobo. The SATA disks are attached to the 3GBps ports on the mobo.

How do I setup the system specs on here so the hardware shows like yours?

 

[SweetAndLow]

Well, the USB flash drives are connected to the USB ports presented on the I/O panel on the mobo. The SATA disks are attached to the 3GBps ports on the mobo.

How do I setup the system specs on here so the hardware shows like yours?

You don't set it up for when you are asking for help because people can't see it when on mobile platforms.

It seems like you are not using any crazy raid card or funky expanders. So I'm not sure what's going on.

 

[nojohnny101]

All my drives are just connected to my motherboard, no cards or anything in-between. The motherboard is an ASRock C226M WS. The drives are all 3TB Seagate NAS drives. Running 16gb of ECC RAM.

@SweetAndLow you aren't seeing these messages corresponding to scrubs?

 

[SweetAndLow]

I'm not using freenas 11. I'm still on 9.10

 

[droeders]

I get these same messages on three different FreeBSD servers with both SAS and SATA connections.

Much ado about nothing... See the FreeBSD discussion linked above.