Someone called me this week about a payroll program called Meridian acting slow. Apparently the program freezes from time-to-time (every fortnight or so) and makes it impossible to do anything useful.
The payroll staff have a client that connects to an SQL Server database on an isolated Windows Server 2008 server. Since all 3 staff who use the program regularly were experiencing the same issue and there was no apparent networking issue, I decided to look at the database server to see what’s going on.
I eventually found a process labelled “MeridianSvc.exe” which is running as a service taking up 1.68 GB of memory (the database server only has 2 GB of available memory). I decided to call the staff and tell them to logoff, then restart the service. After doing this the process size went down to 6.8 MB. The staff were able to connect in again and were satisfied with the performance again.
I monitored the memory usage over the next couple of days and noticed it growing by about 160 MB a day. Suspecting a memory leak in the process, I decided to contact Meridian support to hopefully get rid of the memory leak (it would be awful if every customer of Meridian had to deal with this).
Meridian support notified me this service is used to integrate two products together, and that I need to disable debug logging, increase the amount of seconds it polls data to perform its logic (every 10 seconds), and write a script to restart the service every night. Unfortunately this was their only solution. I think it’s great to have a workaround since they provide a mean time solution while root causes get fixed, however workarounds should never be permanent. It didn’t sound like they were interested in pursuing the issue further.
I wrote a simple batch script to restart the service and schedule it to run at 5:30 AM daily:
net stop “Meridian Service”
net start “Meridian Service”
echo %date% %time% – meridian service service restarted >> c:\meridian\service_restart_log.txt
I went to change the polling interval using the program “ServiceConfig.exe” that they provide from 10 seconds to say 60 seconds, however I noticed that there was still a CPU spike every 10 seconds with the service. The reason was because the program was trying to change settings under the key “HKCU\Software\Meridian” or a similar key however because something wasn’t conforming with UAC the settings were being virtualized and written to a HKCU key (I’m not interested in disabling UAC since it’s there for a good purpose: http://msdn.microsoft.com/en-us/library/aa905330.aspx#wvduac_topic9). After changing the settings under the key that the service actually used in the registry manually, the meridian service still leaked memory, just slower.
I sympathise with the famous quote from HatfulOfHollow from http://bash.org/?4281