models builder error is causing app pool refresh to fail - Using Umbraco and getting started

Chris Dunsing 18 posts 83 karma points

May 03, 2017 @ 17:36

Models builder error is causing APP pool refresh to fail.

Hello,

We had an issue where the app pool could not restart. We disabled Models Builder and the site was able to recycle / recover.

Umbraco Version: 7.5.3

Here are the logs:

_shutDownMessage=Recompilation limit of 15 reached
HostingEnvironment initiated shutdown
HostingEnvironment caused shutdown

_shutDownStack=   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal()
   at System.Web.HttpRuntime.ShutdownAppDomain(String stackTrace)
   at System.Web.Compilation.DiskBuildResultCache.ShutdownCallBack(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
 2017-05-02 10:14:35,994 [P10512/D3/T32] INFO  Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Web.Editors.IEditorValidator
 2017-05-02 10:14:35,995 [P10512/D3/T32] INFO  Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Web.Editors.IEditorValidator, found 0 (took 1ms)
 2017-05-02 10:14:36,572 [P10512/D3/T32] INFO  Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.Persistence.Mappers.BaseMapper
 2017-05-02 10:14:36,573 [P10512/D3/T32] INFO  Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.Persistence.Mappers.BaseMapper, found 0 (took 1ms)
 2017-05-02 10:14:47,575 [P10512/D3/T27] INFO  umbraco.BusinessLogic.Log - Log scrubbed.  Removed all items older than 2017-05-01 10:14:47
 2017-05-02 10:15:14,270 [P10512/D3/T43] WARN  Umbraco.Web.UmbracoModule - Status code is 404 yet TrySkipIisCustomErrors is false - IIS will take over.
 2017-05-02 10:15:19,323 [P10512/D3/T73] WARN  Umbraco.Web.UmbracoModule - Status code is 404 yet TrySkipIisCustomErrors is false - IIS will take over.
 2017-05-02 10:15:57,404 [P10512/D3/T55] ERROR Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Failed to build models.
System.Threading.ThreadAbortException: Thread was being aborted.
   at System.Threading.Monitor.Enter(Object obj)
   at System.Web.Compilation.BuildManager.GetBuildResultFromCacheInternal(String cacheKey, Boolean keyFromVPP, VirtualPath virtualPath, Int64 hashCode, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultFromCacheInternal(VirtualPath virtualPath, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResult(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResult(VirtualPath virtualPath)
   at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.GetModelsAssembly(Boolean forceRebuild) in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Umbraco\PureLiveModelFactory.cs:line 271
   at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.EnsureModels() in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Umbraco\PureLiveModelFactory.cs:line 151
 2017-05-02 10:15:57,408 [P10512/D3/T55] WARN  Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Running without models.
 2017-05-02 10:16:27,597 [P10512/D3/T7] ERROR Umbraco.Web.Scheduling.KeepAlive - Failed (at "http://XXXXXXXXXX.com:80/umbraco").
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Umbraco.Web.Scheduling.KeepAlive.<PerformRunAsync>d__4.MoveNext()
 2017-05-02 10:17:57,504 [P10512/D3/T88] ERROR Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Failed to build models.
System.Threading.ThreadAbortException: Thread was being aborted.
   at System.Threading.Monitor.Enter(Object obj)
   at System.Web.Compilation.BuildManager.GetBuildResultFromCacheInternal(String cacheKey, Boolean keyFromVPP, VirtualPath virtualPath, Int64 hashCode, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultFromCacheInternal(VirtualPath virtualPath, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResult(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResult(VirtualPath virtualPath)
   at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.GetModelsAssembly(Boolean forceRebuild) in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Umbraco\PureLiveModelFactory.cs:line 271
   at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.EnsureModels() in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Umbraco\PureLiveModelFactory.cs:line 151
 2017-05-02 10:17:57,507 [P10512/D3/T88] WARN  Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Running without models.
 2017-05-02 10:19:04,735 [P10512/D3/T48] WARN  Umbraco.Web.UmbracoModule - Status code is 404 yet TrySkipIisCustomErrors is false - IIS will take over.
 2017-05-02 10:19:09,802 [P10512/D3/T93] WARN  Umbraco.Web.UmbracoModule - Status code is 404 yet TrySkipIisCustomErrors is false - IIS will take over.
 2017-05-02 10:19:57,496 [P10512/D3/T85] ERROR Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Failed to build models.
System.Threading.ThreadAbortException: Thread was being aborted.
   at System.Threading.Monitor.Enter(Object obj)
   at System.Web.Compilation.BuildManager.GetBuildResultFromCacheInternal(String cacheKey, Boolean keyFromVPP, VirtualPath virtualPath, Int64 hashCode, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultFromCacheInternal(VirtualPath virtualPath, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResult(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResult(VirtualPath virtualPath)
   at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.GetModelsAssembly(Boolean forceRebuild) in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Umbraco\PureLiveModelFactory.cs:line 271
   at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.EnsureModels() in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Umbraco\PureLiveModelFactory.cs:line 151
 2017-05-02 10:19:57,498 [P10512/D3/T85] WARN  Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Running without models.
 2017-05-02 10:19:57,529 [P10512/D3/T47] ERROR Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Failed to build models.
System.Threading.ThreadAbortException: Thread was being aborted.
   at System.String.GetHashCode()
   at System.Collections.Generic.GenericEqualityComparer`1.GetHashCode(T obj)
   at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
   at Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamespaceSymbol.GetMembers(String name)
   at Microsoft.CodeAnalysis.CSharp.Symbols.MergedNamespaceSymbol.SlowGetChildrenOfName(String name)
   at Microsoft.CodeAnalysis.Collections.CachingDictionary`2.CreateFullyPopulatedMap(IDictionary`2 existingMap)
   at Microsoft.CodeAnalysis.Collections.CachingDictionary`2.EnsureFullyPopulated()
   at Microsoft.CodeAnalysis.Collections.CachingDictionary`2.AddValues(ArrayBuilder`1 array)
   at Microsoft.CodeAnalysis.CSharp.Symbols.MergedNamespaceSymbol.GetMembers()
   at Microsoft.CodeAnalysis.CSharp.Binder.AddMemberLookupSymbolsInfoInNamespace(LookupSymbolsInfo result, NamespaceSymbol ns, LookupOptions options, Binder originalBinder)
   at Microsoft.CodeAnalysis.CSharp.InContainerBinder.AddLookupSymbolsInfoInSingleBinder(LookupSymbolsInfo result, LookupOptions options, Binder originalBinder)
   at Microsoft.CodeAnalysis.CSharp.Binder.AddLookupSymbolsInfo(LookupSymbolsInfo result, LookupOptions options)
   at Microsoft.CodeAnalysis.CSharp.CSharpSemanticModel.LookupSymbolsInternal(Int32 position, NamespaceOrTypeSymbol container, String name, LookupOptions options, Boolean useBaseReferenceAccessibility)
   at Microsoft.CodeAnalysis.CSharp.CSharpSemanticModel.LookupNamespacesAndTypes(Int32 position, INamespaceOrTypeSymbol container, String name)
   at Umbraco.ModelsBuilder.Building.Builder.IsAmbiguousSymbol(String symbol, String match) in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Building\Builder.cs:line 261
   at Umbraco.ModelsBuilder.Building.TextBuilder.WriteNonGenericClrType(StringBuilder sb, String s) in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Building\TextBuilder.cs:line 404
   at Umbraco.ModelsBuilder.Building.TextBuilder.WriteProperty(StringBuilder sb, TypeModel type, PropertyModel property, String mixinClrName) in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Building\TextBuilder.cs:line 290
   at Umbraco.ModelsBuilder.Building.TextBuilder.WriteContentTypeProperties(StringBuilder sb, TypeModel type) in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Building\TextBuilder.cs:line 212
   at Umbraco.ModelsBuilder.Building.TextBuilder.WriteContentType(StringBuilder sb, TypeModel type) in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Building\TextBuilder.cs:line 201
   at Umbraco.ModelsBuilder.Building.TextBuilder.Generate(StringBuilder sb, IEnumerable`1 typeModels) in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Building\TextBuilder.cs:line 83
   at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.GenerateModelsCode(IDictionary`2 ourFiles, IList`1 typeModels) in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Umbraco\PureLiveModelFactory.cs:line 319
   at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.GetModelsAssembly(Boolean forceRebuild) in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Umbraco\PureLiveModelFactory.cs:line 256
   at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.EnsureModels() in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Umbraco\PureLiveModelFactory.cs:line 151
 2017-05-02 10:19:57,541 [P10512/D3/T47] WARN  Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Running without models.
 2017-05-02 10:20:42,548 [P10512/D3/T94] ERROR Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Failed to build models.
System.Threading.ThreadAbortException: Thread was being aborted.
   at System.Threading.Monitor.Enter(Object obj)
   at System.Web.Compilation.BuildManager.GetBuildResultFromCacheInternal(String cacheKey, Boolean keyFromVPP, VirtualPath virtualPath, Int64 hashCode, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultFromCacheInternal(VirtualPath virtualPath, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResult(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResult(VirtualPath virtualPath)
   at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.GetModelsAssembly(Boolean forceRebuild) in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Umbraco\PureLiveModelFactory.cs:line 271
   at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.EnsureModels() in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Umbraco\PureLiveModelFactory.cs:line 151
 2017-05-02 10:20:42,549 [P10512/D3/T94] WARN  Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Running without models.
 2017-05-02 10:22:12,540 [P10512/D3/T69] ERROR Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Failed to build models.
System.Threading.ThreadAbortException: Thread was being aborted.
   at System.Threading.Monitor.Enter(Object obj)
   at System.Web.Compilation.BuildManager.GetBuildResultFromCacheInternal(String cacheKey, Boolean keyFromVPP, VirtualPath virtualPath, Int64 hashCode, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultFromCacheInternal(VirtualPath virtualPath, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResult(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResult(VirtualPath virtualPath)
   at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.GetModelsAssembly(Boolean forceRebuild) in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Umbraco\PureLiveModelFactory.cs:line 271
   at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.EnsureModels() in X:\Projects\Umbraco\ModelsBuilder\Zbu.ModelsBuilder\Umbraco.ModelsBuilder\Umbraco\PureLiveModelFactory.cs:line 151
 2017-05-02 10:22:12,542 [P10512/D3/T69] WARN  Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Running without models.

Here is the config Change we made to stop what we were seeing the logs (ModelsMode is purelive).

<add key="Umbraco.ModelsBuilder.Enable" value="false" />

I assume that this is against best practice. However, in this case we did not include Generated Models in our Views so we were safe to disable and better understand the issue.

Should I be removing hashes and generated files from app_data?
Should I be attempting a full site republish. (Seems unrelated to Models generating)

Any comments welcome to help us better understand the logs.

Another note: We moved the site back to a staging server. This did not generate the same error. I suspect that this could be deployment related? Permissions have been checked to be correct.

Copy Link

Paul Yates 32 posts 86 karma points

May 16, 2017 @ 09:20

We've been having pretty much the same problem for a while now. I believe this identifies the cause:

https://www.zpqrtbnk.net/posts/of-threadabortexception-deadlocks-and-windbg

Copy Link

Chris Dunsing 18 posts 83 karma points

May 16, 2017 @ 17:57

Awesome, We'll upgrade to 7.5.13 and observe. Thanks for the handy link.

Copy Link

Gary 80 posts 377 karma points

Apr 10, 2018 @ 15:51

Did the upgrade fix this issue? We had this hit our intranet and just wanted to ensure this has resolved the issue?

Thanks!

Copy Link

Paul Yates 32 posts 86 karma points

Apr 10, 2018 @ 16:34

Yes it fixed it for us!

Copy Link

Arlan Galvez 44 posts 175 karma points

Mar 02, 2019 @ 16:25

Hello I'm facing the same issue with Umbraco 7.6.3 and Azure with load balancing. Any advance on this?

Regards, Arlan

Copy Link

trfletch 598 posts 604 karma points

Mar 21, 2019 @ 14:39

I think I am facing the same issue on an Umbraco 7.5.4 website which I just posted about (link below).

I was going to try updating to the latest version of Umbraco 7 (7.14 at time of writing) to resolve this based people above saying that Umbraco 7.5.13 fixed it but I am now concerned that it has not fixed it based on what @Arlan Galvez has said?

Any suggestions as to which version of Umbraco I should upgrade to, it is a large complex website therefore upgrading is going to be a big task so I want to make sure I do it to a version that is going to resolve the problem, and ideally to the latest possible version.

https://our.umbraco.com/forum/extending-umbraco-and-using-the-api//96428-umbraco-754-large-site-intermittently-hangs-app-pool-using-high-cpu-app-pool-task-need-to-be-killed-to-recover

Copy Link

is working on a reply...

Flag this post as spam?

Models builder error is causing APP pool refresh to fail.