Umbraco 7.6.3 - Site shuts down after saving doctypes, datatypes etc.
Dear guru's,
I have something weird going on with Umbraco 7.6.3 where the site becomes unavailable after saving doctypes or datatypes.
I see the following errors (it repeats every second or so) in the trace log:
2017-08-30 10:49:04,159 [P6200/D2/T74] ERROR Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache - Could not load data from Examine index for media, the app domain is most likely in a shutdown state
Lucene.Net.Store.AlreadyClosedException: this IndexReader is closed
at Lucene.Net.Index.IndexReader.EnsureOpen()
at Lucene.Net.Index.DirectoryReader.GetFieldNames(FieldOption fieldNames)
at Examine.LuceneEngine.Providers.LuceneSearcher.GetSearchFields() in X:\Projects\Examine\Examine\src\Examine\LuceneEngine\Providers\LuceneSearcher.cs:line 248
at UmbracoExamine.UmbracoExamineSearcher.GetSearchFields()
at Examine.LuceneEngine.Providers.BaseLuceneSearcher.CreateSearchCriteria(String type, BooleanOperation defaultOperation) in X:\Projects\Examine\Examine\src\Examine\LuceneEngine\Providers\BaseLuceneSearcher.cs:line 117
at UmbracoExamine.UmbracoExamineSearcher.CreateSearchCriteria(String type, BooleanOperation defaultOperation)
at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetUmbracoMediaCacheValues(Int32 id)
After some waiting everything becomes available again.
And the trace log stops.
Some more info:
Umbraco 7.6.3
Umbraco Packages:
Articulate 2.0.5
Cultiv Search Engine Sitemap 1.1.0
Cultiv Dynamic Robots 1.0.0
Robots.txt Editor 3.0
MVCDonutCache 1.3.0
Nuget Packages:
Archetye 1.14.1
Vorto 1.5.3
Skybrud.Umbraco.Redirects
Umbraco.AzureCDNToolkit 0.1.2
UmbracoFileSystemProdivers 0.5.8
Donut caching is on the controllers, views are using partial cache, but caching stops working when I edit a doc type or data type.
Application Insights is not showing anything related to this problem.
This is the only thing showing up in the logs. It feels like it's recycling the app pool.
The site is running on Azure, I know i'm sort of load balancing with my local machine (they are using the same DB) but I have another setup which runs multiple sites on an older version of Umbraco (7.4.1) and this has never been a problem.
Maybe I've missed a configuration issue ;'( but I don't know what I've screwed up.
Can someone point me in the right direction of how to proceed? I'm feeling a bit lost here.
Application Insights is actually showing something,
my local machine is showing up in there.
But I guess that's because I made the doctype change locally.
Time 10:09:14 AM
Exception message Could not find file 'C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\1f187d4e\dad57d67\App_Data\TEMP\ExamineIndexes\Internal\segments_1fy'.
Instance name mymachine
Message Unhandled exception in AppDomain (terminating)
System.IO.FileNotFoundException: Could not find file 'C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\1f187d4e\dad57d67\App_Data\TEMP\ExamineIndexes\Internal\segments_1fy'.
File name: 'C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\1f187d4e\dad57d67\App_Data\TEMP\ExamineIndexes\Internal\segments_1fy'
at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit)
at Lucene.Net.Index.SegmentInfos.Read(Directory directory)
at Lucene.Net.Index.SegmentInfos.ReadCurrentVersion(Directory directory)
at Lucene.Net.Index.DirectoryReader.IsCurrent()
at Examine.LuceneEngine.LuceneExtensions.GetReaderStatus(IndexReader reader) in X:\Projects\Examine\Examine\src\Examine\LuceneEngine\LuceneExtensions.cs:line 68
at Examine.LuceneEngine.Providers.LuceneSearcher.ReaderReopener.MaybeReopen() in X:\Projects\Examine\Examine\src\Examine\LuceneEngine\Providers\LuceneSearcher.cs:line 565
at Examine.LuceneEngine.Providers.LuceneSearcher.ReaderReopener.TimerRelease() in X:\Projects\Examine\Examine\src\Examine\LuceneEngine\Providers\LuceneSearcher.cs:line 521
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.TimerQueueTimer.CallCallback()
at System.Threading.TimerQueueTimer.Fire()
at System.Threading.TimerQueue.FireNextTimers()
After that I see a lot of media access in the log, 100 second timeouts etc.
Time 10:21:03 AM
Outgoing Command /media/1010/86936.jpg
Duration 100.00 sec
Instance name mymachine
And then this one:
Error resolving media url from the CDN
System.Net.WebException: The request was aborted: The operation has timed out.
at System.Net.HttpWebRequest.GetResponse()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.<>c__DisplayClass6_1.<UrlToCdnUrl>b__0()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.TryFiveTimes(Action delegateAction)
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.UrlToCdnUrl(String cropUrl, Boolean htmlEncode, String currentDomain)
Followed by:
Could not load data from Examine index for media, the app domain is most likely in a shutdown state
Lucene.Net.Store.AlreadyClosedException: this IndexReader is closed
at Lucene.Net.Index.IndexReader.EnsureOpen()
at Lucene.Net.Index.DirectoryReader.GetFieldNames(FieldOption fieldNames)
at Examine.LuceneEngine.Providers.LuceneSearcher.GetSearchFields() in X:\Projects\Examine\Examine\src\Examine\LuceneEngine\Providers\LuceneSearcher.cs:line 248
at UmbracoExamine.UmbracoExamineSearcher.GetSearchFields()
at Examine.LuceneEngine.Providers.BaseLuceneSearcher.CreateSearchCriteria(String type, BooleanOperation defaultOperation) in X:\Projects\Examine\Examine\src\Examine\LuceneEngine\Providers\BaseLuceneSearcher.cs:line 117
at UmbracoExamine.UmbracoExamineSearcher.CreateSearchCriteria(String type, BooleanOperation defaultOperation)
at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetUmbracoMediaCacheValues(Int32 id)
Could it be the CDN Toolkit configuration that's not happy on my local machine? Everything is working on the front-end though...
So... I messed up the caching for the homepage controller.
That was the main issue that caused CPU to spike.
The Examine errors appeared to be fixed after tweaking the paths in the config. I removed the {machinekey} part. After placing it back those errors were gone too.
Umbraco 7.6.3 - Site shuts down after saving doctypes, datatypes etc.
Dear guru's,
I have something weird going on with Umbraco 7.6.3 where the site becomes unavailable after saving doctypes or datatypes.
I see the following errors (it repeats every second or so) in the trace log:
After some waiting everything becomes available again. And the trace log stops.
Some more info:
Umbraco Packages:
Nuget Packages:
Donut caching is on the controllers, views are using partial cache, but caching stops working when I edit a doc type or data type.
Application Insights is not showing anything related to this problem. This is the only thing showing up in the logs. It feels like it's recycling the app pool.
The site is running on Azure, I know i'm sort of load balancing with my local machine (they are using the same DB) but I have another setup which runs multiple sites on an older version of Umbraco (7.4.1) and this has never been a problem.
Maybe I've missed a configuration issue ;'( but I don't know what I've screwed up.
Can someone point me in the right direction of how to proceed? I'm feeling a bit lost here.
Ok little update...
Application Insights is actually showing something, my local machine is showing up in there.
But I guess that's because I made the doctype change locally.
After that I see a lot of media access in the log, 100 second timeouts etc.
And then this one:
Followed by:
Could it be the CDN Toolkit configuration that's not happy on my local machine? Everything is working on the front-end though...
So... I messed up the caching for the homepage controller. That was the main issue that caused CPU to spike.
The Examine errors appeared to be fixed after tweaking the paths in the config. I removed the {machinekey} part. After placing it back those errors were gone too.
Silly me.
is working on a reply...