Emulated clock in bhyve is local time?

[Patrick M. Hausen]

Hi all,

I just noticed that after reboot the clocks in my bhyve VMs (FreeBSD and Ubuntu) run one hour fast. This is quickly corrected by the NTP I have set up in each VM, but just out of curiosity: is it advisable to touch /etc/wall_cmos_clock for installations inside bhyve? If yes, does anyone know the equivalent for Unbuntu? ;)

Thanks,
Patrick

 

[dlavigne]

Did you find the answer to your question?

 

[Patrick M. Hausen]

Nope :)

 

[melar]

I had the same problem. I solved it by using bhyve with option -u.

 

[Patrick M. Hausen]

How do I change bhyve options for VMs I created with the UI?

Thanks
Patrick

 

[melar]

dont know now. I am still using iohyve

 

[Patrick M. Hausen]

https://redmine.ixsystems.com/issues/26935

 

[dmshimself]

Has anyone found a workaround for this at all? I don't seem to be able to use the date command in rancher to fix it, the documentation on using ros to set this sort of thing seems to be elusive and I have a number of date/time sensitive apps that I'd love to run as dockers in freenas and some of these don't run any sort time sync (well of course they don't!). Those that do are fine but the clock in rancher is a few minutes out.

 

[Patrick M. Hausen]

Well, as far as I'm concerned the clock is only off by an hour or two at boot time. Sooner or later NTP in the guest will fix it ...

Kind regards
Patrick

 

[dmshimself]

For those dockers I have with some sort of with NTP client, that indeed works. But I'm using some dockers which do not have that built in and rely on the host to have the correct time. My previous experiences of other similar techs makes me think that a synchronised time is really important, especially on the host OS and without that, things eventually trip up, usually when you least expect it. If rancheros supported /etc/localtime, that would probably be OK too as it could be passed on, but it doesn't seem to do that either. I have noticed that if you reboot the rancher VM, the time is again OK. For some people in a non-prod environment, this might be enough, but isn't ideal.

 

[melar]

I've thought about the following, but didn't test it. (I'm still using iohyve)

bhyve is located under /usr/sbin/
rename bhyve to bhyve-renamed
add a new script named bhyve

Code:

#!/bin/sh
/usr/sbin/bhyve-renamed -u $*

chown root:wheel bhyve
chmod 555 bhyve

This adds the option -u to every use of bhyve. I think this shouldn't be a problem.
Maybe someone could test this.

 

[dmshimself]

That is very clever and works just fine here for me. Many thanks

 

[dmshimself]

I spoke a little too soon. The clock on rancher does drift off. It starts up fine after restarting the vm for the docker, but over time the clock runs slowly and there then becomes a gap. I'm making enquiries over in the rancheros forum too.

 

[Patrick M. Hausen]

@dmshimself are you running NTP inside the rancheros VM? Because you should ;)

Kind regards
Patrick

 

[melar]

- Google for "bhyve clock sleep"
- Do a cat /sys/devices/system/clocksource/clocksource0/current_clocksource on your rancheros
Is it tsc ?
- Maybe a kern.timecounter.hardware="TSC-low" in /etc/sysctl.conf solves your clock problem.

 

[dmshimself]

@dmshimself are you running NTP inside the rancheros VM? Because you should ;)

Kind regards
Patrick

I'd love to know how to do that but I've searched high and low without any sucecss so far. It might be my own fault because I'm not running the full rancher, just using portainer. The server I have isn't that high a spec and rancher seems to load up a lot of containers. The last time I tried, the server was straining a bit. I've set up NTP loadsoftimes previously, but rancheros seems to be rightly locked down. Any advice appreciated.

Edited - I've just noticed that ntpd is running on the rancher box as default, so I'll have a good look at that and see why the clock is drifting.

 

[dmshimself]

- Google for "bhyve clock sleep"
- Do a cat /sys/devices/system/clocksource/clocksource0/current_clocksource on your rancheros
Is it tsc ?
- Maybe a kern.timecounter.hardware="TSC-low" in /etc/sysctl.conf solves your clock problem.

Cheers. I'll have a good rummage around later on today. The current_clocksource says refined-jifffies (I have no idea what that means)

 

[dmshimself]

I'd love to know how to do that but I've searched high and low without any sucecss so far. It might be my own fault because I'm not running the full rancher, just using portainer. The server I have isn't that high a spec and rancher seems to load up a lot of containers. The last time I tried, the server was straining a bit. I've set up NTP loadsoftimes previously, but rancheros seems to be rightly locked down. Any advice appreciated.

Edited - I've just noticed that ntpd is running on the rancher box as default, so I'll have a good look at that and see why the clock is drifting.

I've done a few system-docker logs ntp one after the other and the results may be just fine but look a little odd to me. The timestamp for the log is about 10 days out, but the date of the rancheros is about right as shown at the end. I'll leave it for a bit and check again, but maybe that's just the way it is. I'm used to having times being many hours out, but not many days.

Code:

[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
+ ntpd -gq
13 May 18:14:07 ntpd[11]: ntpd 4.2.8p10@1.3728-o Wed Jul 26 20:13:20 UTC 2017 (1): Starting
13 May 18:14:07 ntpd[11]: Command line: ntpd -gq
13 May 18:14:08 ntpd[11]: proto: precision = 4000.250 usec (-8)
13 May 18:14:08 ntpd[11]: proto: fuzz beneath 0.342 usec
13 May 18:14:08 ntpd[11]: Listen and drop on 0 v6wildcard [::]:123
13 May 18:14:08 ntpd[11]: Listen and drop on 1 v4wildcard 0.0.0.0:123
13 May 18:14:08 ntpd[11]: Listen normally on 2 lo 127.0.0.1:123
13 May 18:14:08 ntpd[11]: Listen normally on 3 eth1 192.168.18.141:123
13 May 18:14:08 ntpd[11]: Listen normally on 4 lo [::1]:123
[rancher@rancher ~]$ date
Wed May 23 23:41:28 UTC 2018
[rancher@rancher ~]$

 

[dmshimself]

I left things running for a few hours and the time on the rancheros is about 5 minutes compared with the time in freenas (and everywhere else). If I look at the ntp log from the system-docker, I see:

Code:

[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
+ ntpd -gq
13 May 18:14:07 ntpd[11]: ntpd 4.2.8p10@1.3728-o Wed Jul 26 20:13:20 UTC 2017 (1): Starting
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
+ ntpd -gq
13 May 18:14:07 ntpd[11]: ntpd 4.2.8p10@1.3728-o Wed Jul 26 20:13:20 UTC 2017 (1): Starting
13 May 18:14:07 ntpd[11]: Command line: ntpd -gq
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
+ ntpd -gq

And this just repeats endlessly, the time never sees to get set even though the parameters should allow a large mismatch to be corrected.

if I then stop the ntp system docker and run ntpd by hand, I get a definite time change and the clock on rancheros is again correct for a while.

Code:

rancher@rancher ~]$ sudo ntpd -d -n
24 May 05:38:16 ntpd[26584]: ntpd 4.2.8p10@1.3728-o Wed Jul 26 20:13:20 UTC 2017 (1): Starting
24 May 05:38:16 ntpd[26584]: Command line: ntpd -d -n
24 May 05:38:16 ntpd[26584]: proto: precision = 4000.250 usec (-8)
24 May 05:38:16 ntpd[26584]: proto: fuzz beneath 0.280 usec
24 May 05:38:16 ntpd[26584]: Listen and drop on 0 v6wildcard [::]:123
24 May 05:38:16 ntpd[26584]: Listen and drop on 1 v4wildcard 0.0.0.0:123
24 May 05:38:16 ntpd[26584]: Listen normally on 2 lo 127.0.0.1:123
24 May 05:38:16 ntpd[26584]: Listen normally on 3 eth0 10.13.0.101:123
24 May 05:38:16 ntpd[26584]: Listen normally on 4 eth1 192.168.18.113:123
24 May 05:38:16 ntpd[26584]: Listen normally on 5 docker0 172.17.0.1:123
24 May 05:38:16 ntpd[26584]: Listen normally on 6 lo [::1]:123
24 May 05:38:16 ntpd[26584]: Listen normally on 7 eth0 [2a00:1508:a0d:fe00:2a0:98ff:fe25:ca6f]:123
24 May 05:38:16 ntpd[26584]: Listen normally on 8 eth0 [fe80::2a0:98ff:fe25:ca6f%2]:123
24 May 05:38:16 ntpd[26584]: Listen normally on 9 eth1 [fe80::2a0:98ff:fe6c:7b93%3]:123
24 May 05:38:16 ntpd[26584]: Listen normally on 10 docker0 [fe80::42:a4ff:fe08:af3e%6]:123
24 May 05:38:16 ntpd[26584]: Listen normally on 11 veth3c89864 [fe80::988e:b0ff:feb0:ecdb%8]:123
24 May 05:38:16 ntpd[26584]: Listen normally on 12 veth767c584 [fe80::8833:f4ff:fea0:d42%11]:123
24 May 05:38:16 ntpd[26584]: Listening on routing socket on fd #29 for interface updates
24 May 05:38:41 ntpd[26584]: ntpd exiting on signal 2 (Interrupt)
24 May 05:38:41 ntpd[26584]: 203.118.151.32 local addr 10.13.0.101 -> <null>
24 May 05:38:41 ntpd[26584]: 122.252.188.99 local addr 10.13.0.101 -> <null>
24 May 05:38:41 ntpd[26584]: 130.217.226.51 local addr 10.13.0.101 -> <null>
24 May 05:38:41 ntpd[26584]: 202.78.240.38 local addr 10.13.0.101 -> <null>
[rancher@rancher ~]$ date
Thu May 24 05:38:44 UTC 2018

I also note that the running this manually shows a log that doesn't have the huge number of days out that the log for the system-docker seems to show. That might be a red herring, but note the jump form the 13th May to the 24th May on the logs.

It looks to my uninformed eye that the system-docker is trying to change the time with a big jump, but never gets that done and so repeats.

 

[Patrick M. Hausen]

I assume by "dockers" you mean docker containers, right? They run on top of a minimal Linux system very roughly similar to FreeBSD jails.

That minimal Linux is your rancheros VM. And you should run NTP in that VM outside of all docker containers. Otherwise your clock will always drift.

Now searching a bit about rancher it seems like they replaced all system services with containers. So essentially you need to run *one* privileged container that runs NTP and all other ones without, since they get their time from the host.

HTH, that's about as much as I know and can explain since I don't run docker ;)
Patrick