I have load balanced setup of Umbraco 7.2.1 with a central file system and have followed everything on this page.
I've experienced an issue yesterday and today, around the same time, where our site on node 1 (we have 2 server nodes) will become incredibly slow on the back and front end. When I opened the log, I can see it going crazy with multiple entries every millisecond. I also noticed a trend. The log entries below are getting repeated every 3ms for the better part of an hour. And just like yesterday, they eventually stop and everything works fine. I have recently installed and uninstalled the sassy plugin, so that may have something to do with it.
015-02-13 10:35:17,044 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Determining hash of code files on disk
2015-02-13 10:35:17,107 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of umbraco.interfaces.IApplicationStartupHandler
2015-02-13 10:35:17,169 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of umbraco.interfaces.IApplicationStartupHandler, found 46 (took 49ms)
2015-02-13 10:35:17,341 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.PropertyEditors.IPropertyEditorValueConverter
2015-02-13 10:35:17,357 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.PropertyEditors.IPropertyEditorValueConverter, found 0 (took 2ms)
2015-02-13 10:35:17,357 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.PropertyEditors.IPropertyValueConverter
2015-02-13 10:35:17,357 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.PropertyEditors.IPropertyValueConverter, found 20 (took 3ms)
2015-02-13 10:35:17,372 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Web.Mvc.SurfaceController
2015-02-13 10:35:17,372 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Web.Mvc.SurfaceController, found 5 (took 1ms)
2015-02-13 10:35:17,372 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Web.WebApi.UmbracoApiController
2015-02-13 10:35:17,388 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Web.WebApi.UmbracoApiController, found 66 (took 14ms)
2015-02-13 10:35:17,435 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.Media.IThumbnailProvider
2015-02-13 10:35:17,435 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.Media.IThumbnailProvider, found 3 (took 0ms)
2015-02-13 10:35:17,435 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.Media.IImageUrlProvider
2015-02-13 10:35:17,451 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.Media.IImageUrlProvider, found 1 (took 0ms)
2015-02-13 10:35:18,669 [54] INFO Umbraco.Web.Search.ExamineEvents - [Thread 29] Initializing Examine and binding to business logic events
2015-02-13 10:35:18,919 [54] INFO Umbraco.Web.Search.ExamineEvents - [Thread 29] Adding examine event handlers for index providers: 3
2015-02-13 10:35:19,060 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of umbraco.interfaces.IApplication
2015-02-13 10:35:19,060 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of umbraco.interfaces.IApplication, found 7 (took 0ms)
The behavior would be consistent with having shared Examine Indexes or shared umbraco cache. Make sure the /app_data/temp/umbraco.config is unique per server node - same for /app_data/temp/examineindexes.
Other than that, you can see that the app starts, then shuts down due to the hosting environment less than 0.5 seconds later:
2015-02-13 10:35:19,998 [54] INFO umbraco.content - [Thread 57] Loading content from disk cache... 2015-02-13 10:35:20,373 [54] INFO Umbraco.Core.UmbracoApplicationBase - [Thread 59] Application shutdown. Reason: HostingEnvironment
So check to make sure you don't have something that could cause a shutdown - i.e. change to config or /bin/ folder. It also may be helpful for you to dive into the Windows Event log as something is causing your app to shut down, so anti-virus, etc...should log events there.
I'm using the TempStorage package to handle the local cache for Examine's temp files. I'll double check this to make sure I'm implementing it right. I noticed a line in Umbraco's logs that is inconsistent in subsequent log entries.
Shouldn't this consistently be only one of the domains?
Just for reference, our DNS A record is research.fsu.edu, with each node having an alias of research1.magnet.fsu.edu and research2.magnet.fsu.edu. I have a URL Rewrite that directs /umbraco/ and /umbraco/default/ traffic to research1.magnet.fsu.edu. My distributed calls in umbracoSettings.config look like:
I looked at the Application Event Log and found a warning from ASP.NET. It seems to be thrown as a result of Umbraco attempting to load content from the disk cache. The request path comes from /umbraco/webservices/nodesorter.asmx/js
Exception information:
Exception type: ArgumentOutOfRangeException
Exception message: startIndex cannot be larger than length of string.
Parameter name: startIndex
at System.String.Substring(Int32 startIndex, Int32 length)
at System.Web.Compilation.DiskBuildResultCache.MarkAssemblyAndRelatedFilesForDeletion(String
assemblyName)
at System.Web.Compilation.BuildManager.CacheBuildResultInternal(String
cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart)
at System.Web.Compilation.BuildManager.CompileWebFile(VirtualPath
virtualPath)
at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath
virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean
ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext
context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetVirtualPathObjectFactory(VirtualPath
virtualPath, HttpContext context, Boolean allowCrossApp, Boolean
throwIfNotFound)
at System.Web.Compilation.BuildManager.GetCompiledType(VirtualPath
virtualPath)
at System.Web.Script.Services.WebServiceData.GetWebServiceData(HttpContext
context, String virtualPath, Boolean failIfNoData, Boolean pageMethods, Boolean inlineScript)
at System.Web.Script.Services.WebServiceClientProxyGenerator.GetClientProxyScript(HttpContext
context)
at System.Web.Script.Services.RestClientProxyHandler.ProcessRequest(HttpContext
context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
Request information:
Request URL:
http://research1.magnet.fsu.edu/umbraco/webservices/nodesorter.asmx/js
Request path: /umbraco/webservices/nodesorter.asmx/js
User host address: 146.201.224.125
User: Kevin John
Is authenticated: True
Authentication Type: Forms
Thread account name: AD\DomainDavidsonIIS
Thread information:
Thread ID: 95
Thread account name: AD\DomainDavidsonIIS
Is impersonating: True
Stack trace: at System.String.Substring(Int32 startIndex, Int32 length)
at System.Web.Compilation.DiskBuildResultCache.MarkAssemblyAndRelatedFilesForDeletion(String
assemblyName)
at System.Web.Compilation.BuildManager.CacheBuildResultInternal(String
cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart)
at System.Web.Compilation.BuildManager.CompileWebFile(VirtualPath
virtualPath)
at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath
virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean
ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext
context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetVirtualPathObjectFactory(VirtualPath
virtualPath, HttpContext context, Boolean allowCrossApp, Boolean
throwIfNotFound)
at System.Web.Compilation.BuildManager.GetCompiledType(VirtualPath
virtualPath)
at System.Web.Script.Services.WebServiceData.GetWebServiceData(HttpContext
context, String virtualPath, Boolean failIfNoData, Boolean pageMethods, Boolean inlineScript)
at System.Web.Script.Services.WebServiceClientProxyGenerator.GetClientProxyScript(HttpContext
context)
at System.Web.Script.Services.RestClientProxyHandler.ProcessRequest(HttpContext
context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
I think this issue may have been caused by leaving distributed calls in my dev environment. For now, I have commented out the servers in my umbracoSettings.config until I am ready to publish.
This brings up a bit of a dev nuisance. Since I am using a Website Project in Visual Studio, I am unable to create transformations for any config file other than web.config (and even this has some issues). It would be nice if I could add distributed calls to my web.config as opposed to umbracoSettings. That way I only have to keep track of differences between dev, staging, and release in a single file.
Obviously I would have to add "umbracoSettings-debug.config" to my config folder and replicate to this any changes I make to the regular config, but that would still be an improvement over my current workflow.
Umbraco crashing on one server every 3ms
I have load balanced setup of Umbraco 7.2.1 with a central file system and have followed everything on this page.
I've experienced an issue yesterday and today, around the same time, where our site on node 1 (we have 2 server nodes) will become incredibly slow on the back and front end. When I opened the log, I can see it going crazy with multiple entries every millisecond. I also noticed a trend. The log entries below are getting repeated every 3ms for the better part of an hour. And just like yesterday, they eventually stop and everything works fine. I have recently installed and uninstalled the sassy plugin, so that may have something to do with it.
015-02-13 10:35:17,044 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Determining hash of code files on disk
2015-02-13 10:35:17,107 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Hash determined (took 52ms)
2015-02-13 10:35:17,107 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of umbraco.interfaces.IApplicationStartupHandler
2015-02-13 10:35:17,169 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of umbraco.interfaces.IApplicationStartupHandler, found 46 (took 49ms)
2015-02-13 10:35:17,341 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.PropertyEditors.IPropertyEditorValueConverter
2015-02-13 10:35:17,357 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.PropertyEditors.IPropertyEditorValueConverter, found 0 (took 2ms)
2015-02-13 10:35:17,357 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.PropertyEditors.IPropertyValueConverter
2015-02-13 10:35:17,357 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.PropertyEditors.IPropertyValueConverter, found 20 (took 3ms)
2015-02-13 10:35:17,372 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Web.Mvc.SurfaceController
2015-02-13 10:35:17,372 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Web.Mvc.SurfaceController, found 5 (took 1ms)
2015-02-13 10:35:17,372 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Web.WebApi.UmbracoApiController
2015-02-13 10:35:17,388 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Web.WebApi.UmbracoApiController, found 66 (took 14ms)
2015-02-13 10:35:17,435 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.Media.IThumbnailProvider
2015-02-13 10:35:17,435 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.Media.IThumbnailProvider, found 3 (took 0ms)
2015-02-13 10:35:17,435 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.Media.IImageUrlProvider
2015-02-13 10:35:17,451 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.Media.IImageUrlProvider, found 1 (took 0ms)
2015-02-13 10:35:18,669 [54] INFO Umbraco.Web.Search.ExamineEvents - [Thread 29] Initializing Examine and binding to business logic events
2015-02-13 10:35:18,919 [54] INFO Umbraco.Web.Search.ExamineEvents - [Thread 29] Adding examine event handlers for index providers: 3
2015-02-13 10:35:19,060 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of umbraco.interfaces.IApplication
2015-02-13 10:35:19,060 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of umbraco.interfaces.IApplication, found 7 (took 0ms)
2015-02-13 10:35:19,716 [54] INFO Umbraco.Web.UmbracoModule - [Thread 57] Setting OriginalRequestUrl: www.research.fsu.edu:80/umbraco
2015-02-13 10:35:19,998 [54] INFO umbraco.content - [Thread 57] Loading content from disk cache...
2015-02-13 10:35:20,373 [54] INFO Umbraco.Core.UmbracoApplicationBase - [Thread 59] Application shutdown. Reason: HostingEnvironment
2015-02-13 10:35:20,670 [54] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner`1[[Umbraco.Web.Scheduling.LogScrubber, umbraco, Version=1.0.5462.37518, Culture=neutral, PublicKeyToken=null]] - [Thread 2] Application is shutting down immediately
2015-02-13 10:35:20,670 [54] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner`1[[Umbraco.Web.Scheduling.ScheduledTasks, umbraco, Version=1.0.5462.37518, Culture=neutral, PublicKeyToken=null]] - [Thread 2] Application is shutting down immediately
2015-02-13 10:35:20,670 [54] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner`1[[Umbraco.Web.Scheduling.ScheduledPublishing, umbraco, Version=1.0.5462.37518, Culture=neutral, PublicKeyToken=null]] - [Thread 2] Application is shutting down immediately
@Matthew
The behavior would be consistent with having shared Examine Indexes or shared umbraco cache. Make sure the /app_data/temp/umbraco.config is unique per server node - same for /app_data/temp/examineindexes.
Other than that, you can see that the app starts, then shuts down due to the hosting environment less than 0.5 seconds later:
2015-02-13 10:35:19,998 [54] INFO umbraco.content - [Thread 57] Loading content from disk cache...
2015-02-13 10:35:20,373 [54] INFO Umbraco.Core.UmbracoApplicationBase - [Thread 59] Application shutdown. Reason: HostingEnvironment
So check to make sure you don't have something that could cause a shutdown - i.e. change to config or /bin/ folder. It also may be helpful for you to dive into the Windows Event log as something is causing your app to shut down, so anti-virus, etc...should log events there.
Thanks for the response, Paul.
I'm using the TempStorage package to handle the local cache for Examine's temp files. I'll double check this to make sure I'm implementing it right. I noticed a line in Umbraco's logs that is inconsistent in subsequent log entries.
2015-02-13 10:35:19,716 [54] INFO Umbraco.Web.UmbracoModule - [Thread 57] Setting OriginalRequestUrl: www.research.fsu.edu:80/umbraco
vs
2015-02-13 11:18:14,447 [97] INFO Umbraco.Web.UmbracoModule - [Thread 91] Setting OriginalRequestUrl: research1.magnet.fsu.edu:80/umbraco
Shouldn't this consistently be only one of the domains?
Just for reference, our DNS A record is research.fsu.edu, with each node having an alias of research1.magnet.fsu.edu and research2.magnet.fsu.edu. I have a URL Rewrite that directs /umbraco/ and /umbraco/default/ traffic to research1.magnet.fsu.edu. My distributed calls in umbracoSettings.config look like:
I looked at the Application Event Log and found a warning from ASP.NET. It seems to be thrown as a result of Umbraco attempting to load content from the disk cache. The request path comes from /umbraco/webservices/nodesorter.asmx/js
I think this issue may have been caused by leaving distributed calls in my dev environment. For now, I have commented out the servers in my umbracoSettings.config until I am ready to publish.
This brings up a bit of a dev nuisance. Since I am using a Website Project in Visual Studio, I am unable to create transformations for any config file other than web.config (and even this has some issues). It would be nice if I could add distributed calls to my web.config as opposed to umbracoSettings. That way I only have to keep track of differences between dev, staging, and release in a single file.
Hi Matthew,
You can move umbracoSettings.config to web.config. It would go where the configSource line is:
So instead settings would contain everything in umbracoSettings.
Thanks, Brad. I'll play around with this next time I deploy to staging.
Do you know if config transforms would work with this setup? For example, I want to add in my web.Debug.Config:
Obviously I would have to add "umbracoSettings-debug.config" to my config folder and replicate to this any changes I make to the regular config, but that would still be an improvement over my current workflow.
is working on a reply...