Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Matthew Kirschner 323 posts 611 karma points
    Feb 13, 2015 @ 17:34
    Matthew Kirschner
    0

    Umbraco crashing on one server every 3ms

    I have load balanced setup of Umbraco 7.2.1 with a central file system and have followed everything on this page.

    I've experienced an issue yesterday and today, around the same time, where our site on node 1 (we have 2 server nodes) will become incredibly slow on the back and front end. When I opened the log, I can see it going crazy with multiple entries every millisecond. I also noticed a trend. The log entries below are getting repeated every 3ms for the better part of an hour. And just like yesterday, they eventually stop and everything works fine. I have recently installed and uninstalled the sassy plugin, so that may have something to do with it.

    015-02-13 10:35:17,044 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Determining hash of code files on disk

    2015-02-13 10:35:17,107 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Hash determined (took 52ms)

    2015-02-13 10:35:17,107 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of umbraco.interfaces.IApplicationStartupHandler

    2015-02-13 10:35:17,169 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of umbraco.interfaces.IApplicationStartupHandler, found 46 (took 49ms)

    2015-02-13 10:35:17,341 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.PropertyEditors.IPropertyEditorValueConverter

    2015-02-13 10:35:17,357 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.PropertyEditors.IPropertyEditorValueConverter, found 0 (took 2ms)

    2015-02-13 10:35:17,357 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.PropertyEditors.IPropertyValueConverter

    2015-02-13 10:35:17,357 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.PropertyEditors.IPropertyValueConverter, found 20 (took 3ms)

    2015-02-13 10:35:17,372 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Web.Mvc.SurfaceController

    2015-02-13 10:35:17,372 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Web.Mvc.SurfaceController, found 5 (took 1ms)

    2015-02-13 10:35:17,372 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Web.WebApi.UmbracoApiController

    2015-02-13 10:35:17,388 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Web.WebApi.UmbracoApiController, found 66 (took 14ms)

    2015-02-13 10:35:17,435 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.Media.IThumbnailProvider

    2015-02-13 10:35:17,435 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.Media.IThumbnailProvider, found 3 (took 0ms)

    2015-02-13 10:35:17,435 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of Umbraco.Core.Media.IImageUrlProvider

    2015-02-13 10:35:17,451 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of Umbraco.Core.Media.IImageUrlProvider, found 1 (took 0ms)

    2015-02-13 10:35:18,669 [54] INFO Umbraco.Web.Search.ExamineEvents - [Thread 29] Initializing Examine and binding to business logic events

    2015-02-13 10:35:18,919 [54] INFO Umbraco.Web.Search.ExamineEvents - [Thread 29] Adding examine event handlers for index providers: 3

    2015-02-13 10:35:19,060 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Starting resolution types of umbraco.interfaces.IApplication

    2015-02-13 10:35:19,060 [54] INFO Umbraco.Core.PluginManager - [Thread 29] Completed resolution of types of umbraco.interfaces.IApplication, found 7 (took 0ms)

    2015-02-13 10:35:19,716 [54] INFO Umbraco.Web.UmbracoModule - [Thread 57] Setting OriginalRequestUrl: www.research.fsu.edu:80/umbraco

    2015-02-13 10:35:19,998 [54] INFO umbraco.content - [Thread 57] Loading content from disk cache...

    2015-02-13 10:35:20,373 [54] INFO Umbraco.Core.UmbracoApplicationBase - [Thread 59] Application shutdown. Reason: HostingEnvironment

    2015-02-13 10:35:20,670 [54] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner`1[[Umbraco.Web.Scheduling.LogScrubber, umbraco, Version=1.0.5462.37518, Culture=neutral, PublicKeyToken=null]] - [Thread 2] Application is shutting down immediately

    2015-02-13 10:35:20,670 [54] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner`1[[Umbraco.Web.Scheduling.ScheduledTasks, umbraco, Version=1.0.5462.37518, Culture=neutral, PublicKeyToken=null]] - [Thread 2] Application is shutting down immediately

    2015-02-13 10:35:20,670 [54] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner`1[[Umbraco.Web.Scheduling.ScheduledPublishing, umbraco, Version=1.0.5462.37518, Culture=neutral, PublicKeyToken=null]] - [Thread 2] Application is shutting down immediately

  • Paul Sterling 719 posts 1556 karma points MVP 9x admin c-trib
    Feb 13, 2015 @ 19:02
    Paul Sterling
    0

    @Matthew 

    The behavior would be consistent with having shared Examine Indexes or shared umbraco cache.  Make sure the /app_data/temp/umbraco.config is unique per server node - same for /app_data/temp/examineindexes.

    Other than that, you can see that the app starts, then shuts down due to the hosting environment less than 0.5 seconds later:

    2015-02-13 10:35:19,998 [54] INFO umbraco.content - [Thread 57] Loading content from disk cache...
    2015-02-13 10:35:20,373 [54] INFO Umbraco.Core.UmbracoApplicationBase - [Thread 59] Application shutdown. Reason: HostingEnvironment

    So check to make sure you don't have something that could cause a shutdown - i.e. change to config or /bin/ folder.  It also may be helpful for you to dive into the Windows Event log as something is causing your app to shut down, so anti-virus, etc...should log events there.

  • Matthew Kirschner 323 posts 611 karma points
    Feb 13, 2015 @ 19:58
    Matthew Kirschner
    0

    Thanks for the response, Paul.

    I'm using the TempStorage package to handle the local cache for Examine's temp files. I'll double check this to make sure I'm implementing it right. I noticed a line in Umbraco's logs that is inconsistent in subsequent log entries.

    2015-02-13 10:35:19,716 [54] INFO Umbraco.Web.UmbracoModule - [Thread 57] Setting OriginalRequestUrl: www.research.fsu.edu:80/umbraco

    vs

    2015-02-13 11:18:14,447 [97] INFO Umbraco.Web.UmbracoModule - [Thread 91] Setting OriginalRequestUrl: research1.magnet.fsu.edu:80/umbraco

    Shouldn't this consistently be only one of the domains?

    Just for reference, our DNS A record is research.fsu.edu, with each node having an alias of research1.magnet.fsu.edu and research2.magnet.fsu.edu. I have a URL Rewrite that directs /umbraco/ and /umbraco/default/ traffic to research1.magnet.fsu.edu. My distributed calls in umbracoSettings.config look like:

    <server serverName="Research1">research1.magnet.fsu.edu</server>
    <server serverName="Research2">research2.magnet.fsu.edu</server>
    
  • Matthew Kirschner 323 posts 611 karma points
    Feb 13, 2015 @ 20:02
    Matthew Kirschner
    0

    I looked at the Application Event Log and found a warning from ASP.NET. It seems to be thrown as a result of Umbraco attempting to load content from the disk cache. The request path comes from /umbraco/webservices/nodesorter.asmx/js

    Exception information:
        Exception type: ArgumentOutOfRangeException
        Exception message: startIndex cannot be larger than length of string.
    Parameter name: startIndex
       at System.String.Substring(Int32 startIndex, Int32 length)
       at System.Web.Compilation.DiskBuildResultCache.MarkAssemblyAndRelatedFilesForDeletion(String
    assemblyName)
       at System.Web.Compilation.BuildManager.CacheBuildResultInternal(String
    cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart)
       at System.Web.Compilation.BuildManager.CompileWebFile(VirtualPath
    virtualPath)
       at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath
    virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean
    ensureIsUpToDate)
       at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext
    context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
       at System.Web.Compilation.BuildManager.GetVirtualPathObjectFactory(VirtualPath
    virtualPath, HttpContext context, Boolean allowCrossApp, Boolean
    throwIfNotFound)
       at System.Web.Compilation.BuildManager.GetCompiledType(VirtualPath
    virtualPath)
       at System.Web.Script.Services.WebServiceData.GetWebServiceData(HttpContext
    context, String virtualPath, Boolean failIfNoData, Boolean pageMethods, Boolean inlineScript)
       at System.Web.Script.Services.WebServiceClientProxyGenerator.GetClientProxyScript(HttpContext
    context)
       at System.Web.Script.Services.RestClientProxyHandler.ProcessRequest(HttpContext
    context)
       at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
       at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
    
    
    
    Request information:
        Request URL:
    http://research1.magnet.fsu.edu/umbraco/webservices/nodesorter.asmx/js
        Request path: /umbraco/webservices/nodesorter.asmx/js
        User host address: 146.201.224.125
        User: Kevin John
        Is authenticated: True
        Authentication Type: Forms
        Thread account name: AD\DomainDavidsonIIS
    
    Thread information:
        Thread ID: 95
        Thread account name: AD\DomainDavidsonIIS
        Is impersonating: True
        Stack trace:    at System.String.Substring(Int32 startIndex, Int32 length)
       at System.Web.Compilation.DiskBuildResultCache.MarkAssemblyAndRelatedFilesForDeletion(String
    assemblyName)
       at System.Web.Compilation.BuildManager.CacheBuildResultInternal(String
    cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart)
       at System.Web.Compilation.BuildManager.CompileWebFile(VirtualPath
    virtualPath)
       at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath
    virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean
    ensureIsUpToDate)
       at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext
    context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
       at System.Web.Compilation.BuildManager.GetVirtualPathObjectFactory(VirtualPath
    virtualPath, HttpContext context, Boolean allowCrossApp, Boolean
    throwIfNotFound)
       at System.Web.Compilation.BuildManager.GetCompiledType(VirtualPath
    virtualPath)
       at System.Web.Script.Services.WebServiceData.GetWebServiceData(HttpContext
    context, String virtualPath, Boolean failIfNoData, Boolean pageMethods, Boolean inlineScript)
       at System.Web.Script.Services.WebServiceClientProxyGenerator.GetClientProxyScript(HttpContext
    context)
       at System.Web.Script.Services.RestClientProxyHandler.ProcessRequest(HttpContext
    context)
       at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
       at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
    
  • Matthew Kirschner 323 posts 611 karma points
    Feb 16, 2015 @ 16:13
    Matthew Kirschner
    0

    I think this issue may have been caused by leaving distributed calls in my dev environment. For now, I have commented out the servers in my umbracoSettings.config until I am ready to publish.

    This brings up a bit of a dev nuisance. Since I am using a Website Project in Visual Studio, I am unable to create transformations for any config file other than web.config (and even this has some issues). It would be nice if I could add distributed calls to my web.config as opposed to umbracoSettings. That way I only have to keep track of differences between dev, staging, and release in a single file.

  • Brad 69 posts 97 karma points
    Feb 16, 2015 @ 22:25
    Brad
    0

    Hi Matthew,

    You can move umbracoSettings.config to web.config. It would go where the configSource line is:

    <umbracoConfiguration>
         <settings configSource="config\umbracoSettings.config" />
    </umbracoConfiguration>
    

    So instead settings would contain everything in umbracoSettings.

  • Matthew Kirschner 323 posts 611 karma points
    Feb 16, 2015 @ 23:02
    Matthew Kirschner
    0

    Thanks, Brad. I'll play around with this next time I deploy to staging.

    Do you know if config transforms would work with this setup? For example, I want to add in my web.Debug.Config:

    <umbracoConfiguration>
      <settings configSource="config\umbracoSettings-debug.config" xdt:Transform="Replace" />
    </umbracoConfiguration>
    

    Obviously I would have to add "umbracoSettings-debug.config" to my config folder and replicate to this any changes I make to the regular config, but that would still be an improvement over my current workflow.

  • This forum is in read-only mode while we transition to the new forum.

    You can continue this topic on the new forum by tapping the "Continue discussion" link below.

Please Sign in or register to post replies