Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Wes 2 posts 82 karma points
    May 22, 2022 @ 07:07
    Wes
    0

    Umbraco site keeps crashing due to NuCache.Content.db

    We are having massive issues lately with one of our v8.17.0 Umbraco sites that run on Azure.

    It crashes randomly then the logs start filling up with

    {"@t":"2022-05-22T06:33:34.8055159Z","@mt":"An unhandled exception occurred","@l":"Error","@x":"System.IO.IOException: The process cannot access the file 'D:\\local\\Temp\\UmbracoData\\176c0ea91061639237a28ed82536be7d\\NuCache\\NuCache.Content.db' because it is being used by another process.\r\n   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\r\n   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)\r\n   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)\r\n   at CSharpTest.Net.IO.TransactedCompoundFile..ctor(Options options)\r\n   at CSharpTest.Net.Storage.BTreeFileStoreV2..ctor(Options options)\r\n   at CSharpTest.Net.Collections.BPlusTree`2.OptionsV2.CreateStorage()\r\n   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheBase..ctor(BPlusTreeOptions`2 options)\r\n   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheNone..ctor(BPlusTreeOptions`2 options)\r\n   at CSharpTest.Net.Collections.BPlusTree`2..ctor(BPlusTreeOptions`2 ioptions)\r\n   at Umbraco.Web.PublishedCache.NuCache.DataSource.BTree.GetTree(String filepath, Boolean exists, ContentDataSerializer contentDataSerializer) in D:\\a\\1\\s\\src\\Umbraco.Web\\PublishedCache\\NuCache\\DataSource\\BTree.cs:line 32\r\n   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.MainDomRegister() in D:\\a\\1\\s\\src\\Umbraco.Web\\PublishedCache\\NuCache\\PublishedSnapshotService.cs:line 169\r\n   at Umbraco.Core.Runtime.MainDom.Register(Action install, Action release, Int32 weight) in D:\\a\\1\\s\\src\\Umbraco.Core\\Runtime\\MainDom.cs:line 90\r\n   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.<EnsureCaches>b__36_0() in D:\\a\\1\\s\\src\\Umbraco.Web\\PublishedCache\\NuCache\\PublishedSnapshotService.cs:line 230\r\n   at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)\r\n   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.EnsureCaches() in D:\\a\\1\\s\\src\\Umbraco.Web\\PublishedCache\\NuCache\\PublishedSnapshotService.cs:line 210\r\n   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.CreatePublishedSnapshot(String previewToken) in D:\\a\\1\\s\\src\\Umbraco.Web\\PublishedCache\\NuCache\\PublishedSnapshotService.cs:line 1109\r\n   at System.Lazy`1.CreateValue()\r\n   at System.Lazy`1.LazyInitValue()\r\n   at Umbraco.Web.UmbracoInjectedModule.EnsureHasContent(UmbracoContext context, HttpContextBase httpContext) in D:\\a\\1\\s\\src\\Umbraco.Web\\UmbracoInjectedModule.cs:line 234\r\n   at Umbraco.Web.UmbracoInjectedModule.EnsureUmbracoRoutablePage(UmbracoContext context, HttpContextBase httpContext) in D:\\a\\1\\s\\src\\Umbraco.Web\\UmbracoInjectedModule.cs:line 172\r\n   at Umbraco.Web.UmbracoInjectedModule.ProcessRequest(HttpContextBase httpContext) in D:\\a\\1\\s\\src\\Umbraco.Web\\UmbracoInjectedModule.cs:line 111\r\n   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()\r\n   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)\r\n   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)","SourceContext":"Umbraco.Web.UmbracoApplicationBase","ProcessId":3308,"ProcessName":"w3wp","ThreadId":36,"AppDomainId":2,"AppDomainAppId":"LMW3SVC348075699ROOT","MachineName":"RD0003FF85B9F6","Log4NetLevel":"ERROR","HttpRequestNumber":13037,"HttpRequestId":"81e1ccbd-4518-4e5c-9748-df2bc049eab2"}
    

    So as far as I can tell I've added all the appropriate config for Azure. i.e:

    <add key="Umbraco.Core.MainDom.Lock" value="SqlMainDomLock" />
    <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
    <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />
    
    WEBSITE_DISABLE_OVERLAPPED_RECYCLING =1
    

    This is a fairly large site (100K+ nodes). Has anyone experienced this before?

    Thanks!

  • Peter van den Dungen 63 posts 350 karma points
    May 22, 2022 @ 09:19
    Peter van den Dungen
    0

    https://our.umbraco.com/forum/using-umbraco-and-getting-started/99779-umbraco-crashes-from-time-to-time-nucachecontentdb-is-being-used-by-another-process (the related github issue has been closed)

    But based on your settings, it seems like you tried the recommend solution already without success.

    Here is a more recent topic (load balancing configuration): https://our.umbraco.com/forum/using-umbraco-and-getting-started/108150-nucachecontentdb-still-being-locked-by-another-process

  • Marc Goodson 1718 posts 11411 karma points MVP 7x c-trib
    May 22, 2022 @ 10:50
    Marc Goodson
    100

    Hi Wes when this phenomenon occurs what is the activity on your Sql instance? Is it at 100%...DTUs maxed out??

    What I've seen, even with correct config in place, is the attempt to get the MainDom lock via Sql fails because Sql instance is maxed out and Umbraco keeps retrying every second to get the lock.. But unfortunately this ensures the Sql instance stays maxed out and it gets into a bit of a loop...

    If this is your scenario ... then scaling up the DB instance is a possible short term experiment...

    There is also possibility to move the SqlMain Dom Lock to a different database... https://github.com/umbraco/Umbraco-CMS/pull/11075

    Regards

    Marc

  • Wes 2 posts 82 karma points
    May 22, 2022 @ 11:13
    Wes
    0

    Wow - you're absolutely spot on. I didn't noticed the SQL is maxed out exactly when the issues occur.

    Thank you!

  • Marc Goodson 1718 posts 11411 karma points MVP 7x c-trib
    May 23, 2022 @ 14:49
    Marc Goodson
    0

    Hi Wes

    Following up on this, there is a setting for increasing the Timeout value for the MainDom lock

    <add key="Umbraco.Core.SqlWriteLockTimeOut" value="6000" />
    

    so if you are finding this is timing out, and then the retry is hitting it again, as above, you could also increase this timeout, to give it a better chance of succeeding...

    https://our.umbraco.com/documentation/Reference/Config/webconfig/#umbracocoresqlwritelocktimeout

    regards

    Marc

Please Sign in or register to post replies

Write your reply to:

Draft