ESX vs Windows Time Sync

Not sure if this is a common issue or if I just hit a perfect storm with an ESX host and a Windows guest OS. Here is how it played out……

We had an application that checks the time and date stamps between its host server and the database server that it writes to.  This is done to ensure accurate records in the database and all that fun stuff.  In any case, when the time difference falls outside of the pre-determined tolerance level, the application service shuts itself down, thereby disabling the application.  So far so good. This is what is should do.

When this started happening, I looked at how the clocks were set up. The Windows guest OS was configured to sync time and data information from a domain controller.(Ok, that’s good).  VMWare Tools in the guest OS had the time sync option unchecked. (Ok, that’s good too).  We know that the VMWare Time Sync option and the Windows Time Service do not recognize each other and in some cases can cause overcorrection of time.  Not the case here, but just a fact we are aware of.

Visually, everything looked good, so where did that leave me? I knew something was causing a discrepancy in the time, so I disabled the Windows Time Service and stopped it, then enabled the Time Sync option in VMWare tools.  As soon as I did that, the clock moved forward 13 minutes.  Hmm, that was the amount of time shown in our application logs as a difference in time between our application and the SQL database.  Interesting.

Talking with the VMWare engineer, they updated the ESX host clock to sync from the domain.  I re-enabled Windows Time Service and removed the Time Sync option from VMWare Tools set the guest OS back to its original state. The guest OS was rebooted, since it always gets the clock update at that time.  This time, it came up without our application service shutting down.  A couple other reboots confirmed the setting was good once again.

Lesson learned:

Apparently, in ESX server, even though a guest OS clock (Windows, for sure) is configured to sync with a domain clock source, it is apparently tunneled through ESX.  When this happens, the guest OS application thinks it is 10:00, but the time is being presented as 10:13.  At this point, I am not sure if it is presented this way going to the SQL server or if it is being translated coming back to the guest OS.  I have a hunch it is the latter, since the application is what complains about the time difference.

If anyone else has seen something similar, let me know.  This is really the first time I have run into this and other guests seem to run just fine with the same configuration.