Since we've upgraded to Umbraco 7.5.2 we've started receiving 'random' restarts on our production environment for the following reason:
2016-09-05 10:50:04,001 [P13632/D4/T76] INFO Umbraco.Core.UmbracoApplicationBase - Application shutdown. Details: BinDirChangeOrDirectoryRename
_shutDownMessage=Change Notification for critical directories.
bin dir change or directory rename
HostingEnvironment initiated shutdown
Change Notification for critical directories.
bin dir change or directory rename
Change Notification for critical directories.
bin dir change or directory rename
HostingEnvironment caused shutdown
Has anyone noticed this behavior before, we aren't making any changes to the filesystem or bin directory so I'm assuming the application is? Any tips on how we can stop / prevent it happening?
I've checked the last modified stamps within the bin dir and none of the files have an updated last-modified timestamp - they all match the timestamp of the last deliberate build.
I've tried a fresh install and haven't noticed the behavior outside of Azure - the health check dashboard all seems fine and no errors or exceptions other than this shown in the logs.
Some days we don't have a problem other days it can happen several times.
Definitely looks along the same lines of what I'm experiencing, but I forgot to mention we're using the Azure App service rather than vm's so I'd hope this has automatically been applied to their Azure app servers.
I'll contact their support and see if they can shed any more light on it - I just wondered if it was Umbraco related but at the moment it looks environmental.
We don't currently have a support contract so unfortunately got nowhere with them as unless there is a reported problem we can't report any problems with the service.
We're still seeing the same behavior although it's very intermittent.
We've also been having 100% DTU spikes on the database which have been causing us problems but those seem to have stopped in the last few days - again without any intervention.
It's all very strange and we're still investigating.
We're having this exact situation after a recent NuGet upgrade to 7.5.3 (and 7.5.4 today) on our own locally-hosted Win2008SP1 server. Specifically (and ironically) we can cause that shutdown to happen just by clicking the "Check all groups" in the Developer -> Health Check dashboard. It pops up two 500 errors then creates the tracelog shown below. All health checks look complete except Security which keeps the progress circles flowing.
Chrome console shows 3 javascript errors related to dependencyhandler.axd at http://{server}/umbraco/BackOffice/Api/HealthCheck/GetStatus?id=ed0d7e40-971e-4be8-ab6d-8cc5d0a6a5b0
Have hard-reloaded pages in Chrome, removed temp/clientdependency stuff just in case, seems to be persisting.
2016-10-20 16:20:50,270 [P6112/D9/T77] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Web.HealthCheck.HealthCheck
2016-10-20 16:20:50,286 [P6112/D9/T77] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Web.HealthCheck.HealthCheck, found 0 (took 8ms)
2016-10-20 16:20:50,566 [P6112/D9/T38] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledTasks] Terminating
2016-10-20 16:20:50,566 [P6112/D9/T38] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledTasks] Waiting for tasks to complete
2016-10-20 16:20:50,566 [P6112/D9/T38] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledTasks] Tasks completed, terminated
2016-10-20 16:20:50,566 [P6112/D9/T38] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [LogScrubber] Terminating
2016-10-20 16:20:50,566 [P6112/D9/T38] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [LogScrubber] Waiting for tasks to complete
2016-10-20 16:20:50,566 [P6112/D9/T38] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [KeepAlive] Terminating
2016-10-20 16:20:50,566 [P6112/D9/T38] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [KeepAlive] Waiting for tasks to complete
2016-10-20 16:20:50,582 [P6112/D9/T31] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [LogScrubber] Tasks completed, terminated
2016-10-20 16:20:50,582 [P6112/D9/T36] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [KeepAlive] Tasks completed, terminated
2016-10-20 16:20:50,691 [P6112/D9/T38] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [Umbraco.Web.Scheduling.IBackgroundTask] Terminating
2016-10-20 16:20:50,691 [P6112/D9/T38] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [Umbraco.Web.Scheduling.IBackgroundTask] Waiting for tasks to complete
2016-10-20 16:20:50,691 [P6112/D9/T38] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledPublishing] Terminating
2016-10-20 16:20:50,691 [P6112/D9/T38] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledPublishing] Waiting for tasks to complete
2016-10-20 16:20:50,691 [P6112/D9/T79] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [Umbraco.Web.Scheduling.IBackgroundTask] Tasks completed, terminated
2016-10-20 16:20:50,707 [P6112/D9/T22] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledPublishing] Tasks completed, terminated
2016-10-20 16:20:51,331 [P6112/D9/T77] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.PropertyEditors.IParameterEditor
2016-10-20 16:20:51,393 [P6112/D9/T77] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.PropertyEditors.IParameterEditor, found 0 (took 51ms)
2016-10-20 16:21:03,468 [P6112/D9/T77] INFO Umbraco.Core.PluginManager - Starting resolution types of umbraco.cms.businesslogic.macro.IMacroEngine
2016-10-20 16:21:03,468 [P6112/D9/T77] INFO Umbraco.Core.PluginManager - Completed resolution of types of umbraco.cms.businesslogic.macro.IMacroEngine, found 0 (took 8ms)
2016-10-20 16:21:05,433 [P6112/D9/T16] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.PropertyEditors.IParameterEditor
2016-10-20 16:21:05,433 [P6112/D9/T16] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.PropertyEditors.IParameterEditor, found 0 (took 0ms)
2016-10-20 16:21:09,505 [P6112/D10/T1] INFO Umbraco.Core.CoreBootManager - Umbraco 7.5.4 application starting on W3DEV
2016-10-20 16:21:09,536 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Determining hash of code files on disk
2016-10-20 16:21:09,552 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Hash determined (took 12ms)
2016-10-20 16:21:09,677 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of umbraco.interfaces.IApplicationStartupHandler
2016-10-20 16:21:10,051 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of umbraco.interfaces.IApplicationStartupHandler, found 0 (took 379ms)
2016-10-20 16:21:10,082 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.PropertyEditors.IPropertyEditorValueConverter
2016-10-20 16:21:10,082 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.PropertyEditors.IPropertyEditorValueConverter, found 0 (took 6ms)
2016-10-20 16:21:10,082 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.PropertyEditors.IPropertyValueConverter
2016-10-20 16:21:10,098 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.PropertyEditors.IPropertyValueConverter, found 0 (took 11ms)
2016-10-20 16:21:10,113 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Web.Mvc.SurfaceController
2016-10-20 16:21:10,129 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Web.Mvc.SurfaceController, found 0 (took 6ms)
2016-10-20 16:21:10,129 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Web.WebApi.UmbracoApiController
2016-10-20 16:21:10,145 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Web.WebApi.UmbracoApiController, found 0 (took 21ms)
2016-10-20 16:21:10,191 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.Media.IThumbnailProvider
2016-10-20 16:21:10,207 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.Media.IThumbnailProvider, found 0 (took 6ms)
2016-10-20 16:21:10,207 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.Media.IImageUrlProvider
2016-10-20 16:21:10,223 [P6112/D10/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.Media.IImageUrlProvider, found 0 (took 15ms)
2016-10-20 16:21:12,032 [P6112/D10/T1] INFO Umbraco.Core.DatabaseContext - CanConnect = True
2016-10-20 16:21:12,329 [P6112/D10/T1] INFO Umbraco.Core.DatabaseContext - CanConnect = True
2016-10-20 16:21:12,344 [P6112/D10/T1] INFO Umbraco.Core.DatabaseContext - CanConnect = True
2016-10-20 16:21:12,375 [P6112/D10/T1] INFO Umbraco.Web.Cache.CacheRefresherEventHandler - Initializing Umbraco internal event handlers for cache refreshing
2016-10-20 16:21:12,485 [P6112/D10/T1] INFO Umbraco.Web.Search.ExamineEvents - Initializing Examine and binding to business logic events
2016-10-20 16:21:12,500 [P6112/D10/T1] INFO Umbraco.Web.Search.ExamineEvents - Adding examine event handlers for index providers: 3
2016-10-20 16:21:12,500 [P6112/D10/T1] INFO Umbraco.Core.CoreBootManager - Umbraco application startup complete (took 3170ms)
2016-10-20 16:21:12,719 [P6112/D10/T29] INFO Umbraco.Core.Sync.ApplicationUrlHelper - ApplicationUrl: http://w3devwww.nwcouncil.org:80/umbraco (UmbracoModule request)
2016-10-20 16:21:19,302 [P6112/D10/T31] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.PropertyEditors.IParameterEditor
2016-10-20 16:21:19,349 [P6112/D10/T31] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.PropertyEditors.IParameterEditor, found 0 (took 32ms)
2016-10-20 16:21:20,441 [P6112/D10/T36] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Web.Trees.TreeController
2016-10-20 16:21:20,456 [P6112/D10/T36] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Web.Trees.TreeController, found 0 (took 12ms)
2016-10-20 16:21:20,659 [P6112/D10/T36] INFO Umbraco.Core.PluginManager - Starting resolution types of umbraco.interfaces.ITree
2016-10-20 16:21:20,675 [P6112/D10/T36] INFO Umbraco.Core.PluginManager - Completed resolution of types of umbraco.interfaces.ITree, found 0 (took 12ms)
2016-10-20 16:21:20,706 [P6112/D10/T36] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Web.Trees.TreeController
2016-10-20 16:21:20,706 [P6112/D10/T36] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Web.Trees.TreeController, found 0 (took 0ms)
2016-10-20 16:21:20,706 [P6112/D10/T36] INFO Umbraco.Core.PluginManager - Starting resolution types of umbraco.interfaces.ITree
2016-10-20 16:21:20,721 [P6112/D10/T36] INFO Umbraco.Core.PluginManager - Completed resolution of types of umbraco.interfaces.ITree, found 0 (took 12ms)
2016-10-20 16:21:20,737 [P6112/D10/T36] INFO Umbraco.Core.PluginManager - Starting resolution types of umbraco.interfaces.IApplication
2016-10-20 16:21:20,753 [P6112/D10/T36] INFO Umbraco.Core.PluginManager - Completed resolution of types of umbraco.interfaces.IApplication, found 0 (took 16ms)
2016-10-20 16:21:22,453 [P6112/D10/T10] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Web.HealthCheck.HealthCheck
2016-10-20 16:21:22,453 [P6112/D10/T10] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Web.HealthCheck.HealthCheck, found 0 (took 6ms)
2016-10-20 16:21:32,718 [P6112/D10/T10] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [KeepAlive] Terminating
2016-10-20 16:21:32,718 [P6112/D10/T10] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [KeepAlive] Waiting for tasks to complete
2016-10-20 16:21:32,718 [P6112/D10/T32] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [KeepAlive] Tasks completed, terminated
2016-10-20 16:21:32,718 [P6112/D10/T10] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledTasks] Terminating
2016-10-20 16:21:32,718 [P6112/D10/T10] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledTasks] Waiting for tasks to complete
2016-10-20 16:21:32,718 [P6112/D10/T10] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [Umbraco.Web.Scheduling.IBackgroundTask] Terminating
2016-10-20 16:21:32,718 [P6112/D10/T10] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [Umbraco.Web.Scheduling.IBackgroundTask] Waiting for tasks to complete
2016-10-20 16:21:32,718 [P6112/D10/T10] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [LogScrubber] Terminating
2016-10-20 16:21:32,718 [P6112/D10/T10] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [LogScrubber] Waiting for tasks to complete
2016-10-20 16:21:32,718 [P6112/D10/T10] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledPublishing] Terminating
2016-10-20 16:21:32,718 [P6112/D10/T10] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledPublishing] Waiting for tasks to complete
2016-10-20 16:21:32,749 [P6112/D10/T79] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledTasks] Tasks completed, terminated
2016-10-20 16:21:32,765 [P6112/D10/T30] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [Umbraco.Web.Scheduling.IBackgroundTask] Tasks completed, terminated
2016-10-20 16:21:32,765 [P6112/D10/T77] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [LogScrubber] Tasks completed, terminated
2016-10-20 16:21:32,765 [P6112/D10/T16] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledPublishing] Tasks completed, terminated
2016-10-20 16:22:02,795 [P6112/D10/T79] INFO Umbraco.Core.UmbracoApplicationBase - Application shutdown. Details: CodeDirChangeOrDirectoryRename
_shutDownMessage=Change Notification for critical directories.
App_Code dir change or directory rename
HostingEnvironment initiated shutdown
Change Notification for critical directories.
App_Code dir change or directory rename
Change Notification for critical directories.
bin dir change or directory rename
Change Notification for critical directories.
bin dir change or directory rename
Change Notification for critical directories.
bin dir change or directory rename
HostingEnvironment caused shutdown
_shutDownStack= at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal()
at System.Web.Hosting.HostingEnvironment.InitiateShutdownWithoutDemand()
at System.Web.HttpRuntime.ShutdownAppDomain(String stackTrace)
at System.Web.HttpRuntime.OnCriticalDirectoryChange(Object sender, FileChangeEvent e)
at System.Web.FileChangesMonitor.OnCriticaldirChange(Object sender, FileChangeEvent e)
at System.Web.DirectoryMonitor.FireNotifications()
at System.Web.Util.WorkItem.CallCallbackWithAssert(WorkItemCallback callback)
at System.Web.Util.WorkItem.OnQueueUserWorkItemCompletion(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
Random restarts 7.5.2 on Azure
Hi all,
Since we've upgraded to Umbraco 7.5.2 we've started receiving 'random' restarts on our production environment for the following reason:
Has anyone noticed this behavior before, we aren't making any changes to the filesystem or bin directory so I'm assuming the application is? Any tips on how we can stop / prevent it happening?
I've checked the last modified stamps within the bin dir and none of the files have an updated last-modified timestamp - they all match the timestamp of the last deliberate build.
I've tried a fresh install and haven't noticed the behavior outside of Azure - the health check dashboard all seems fine and no errors or exceptions other than this shown in the logs.
Some days we don't have a problem other days it can happen several times.
Thanks, Dan
Hi Dan,
I've had this issue on my prod env. Hope thi help: https://support.microsoft.com/en-us/kb/3052480
Cheers, Mike
Thanks Mike!
Definitely looks along the same lines of what I'm experiencing, but I forgot to mention we're using the Azure App service rather than vm's so I'd hope this has automatically been applied to their Azure app servers.
I'll contact their support and see if they can shed any more light on it - I just wondered if it was Umbraco related but at the moment it looks environmental.
Hi, Did you get anywhere with azure support?
I seem to have the same issue. On azure website as well.
Hi Lee,
We don't currently have a support contract so unfortunately got nowhere with them as unless there is a reported problem we can't report any problems with the service.
We're still seeing the same behavior although it's very intermittent.
We've also been having 100% DTU spikes on the database which have been causing us problems but those seem to have stopped in the last few days - again without any intervention.
It's all very strange and we're still investigating.
We're having this exact situation after a recent NuGet upgrade to 7.5.3 (and 7.5.4 today) on our own locally-hosted Win2008SP1 server. Specifically (and ironically) we can cause that shutdown to happen just by clicking the "Check all groups" in the Developer -> Health Check dashboard. It pops up two 500 errors then creates the tracelog shown below. All health checks look complete except Security which keeps the progress circles flowing.
Chrome console shows 3 javascript errors related to dependencyhandler.axd at http://{server}/umbraco/BackOffice/Api/HealthCheck/GetStatus?id=ed0d7e40-971e-4be8-ab6d-8cc5d0a6a5b0
Have hard-reloaded pages in Chrome, removed temp/clientdependency stuff just in case, seems to be persisting.
is working on a reply...