Hi,

Umbraco Version 8.18.13

Env: Azure App service, Blob storage and SQL database

It's been couple of years we are using this site hosted on the same infrastructure (Azure App service). This issue cropped up recently i would say after upgrading the V8.18.9.

Production web.config contains the recommended configuration as per the Doc:

  <add key="Umbraco.Core.MainDom.Lock" value="SqlMainDomLock" xdt:Transform="Insert" />
  <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" xdt:Transform="Insert" />
  <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" xdt:Transform="Insert" />

Examine search internal and external stop working after every few days and we cannot rebuild the internal and external indexes from the back office, which cause the global search and site search to throw an error, it also effect other features like save and preview option.

Only way to bring back the site to the normal state is to remove all the files from App_data/TEMP/ExamineIndexes and restart the app service. Which work for another few days.

Here is full error:

       Received an error from the server 
An error occurred  The 'ObjectContent`1' type failed to serialize the response body for content type 'application/json; charset=utf-8'.
    Exception Details
    System.InvalidOperationException: The 'ObjectContent`1' type failed to serialize the response body for content type 'application/json; charset=utf-8'.
    Stacktrace
    Inner Exception
    System.ApplicationException: Could not create an index searcher with the supplied lucene directory
    at Examine.LuceneEngine.Providers.LuceneSearcher.ValidateSearcher() in D:\a\1\s\src\Examine\LuceneEngine\Providers\LuceneSearcher.cs:line 215
       at Examine.LuceneEngine.Providers.LuceneSearcher.GetLuceneSearcher() in D:\a\1\s\src\Examine\LuceneEngine\Providers\LuceneSearcher.cs:line 80
       at Umbraco.Examine.ExamineExtensions.GetIndexDocumentCount(LuceneIndex indexer) in D:\a\1\s\src\Umbraco.Examine\ExamineExtensions.cs:line 181
       at Umbraco.Examine.LuceneIndexDiagnostics.get_DocumentCount() in D:\a\1\s\src\Umbraco.Examine\LuceneIndexDiagnostics.cs:line 28
       at Umbraco.Examine.UmbracoExamineIndex.get_DocumentCount() in D:\a\1\s\src\Umbraco.Examine\UmbracoExamineIndex.cs:line 197
       at Umbraco.Web.Editors.ExamineManagementController.CreateModel(IIndex index) in D:\a\1\s\src\Umbraco.Web\Editors\ExamineManagementController.cs:line 179
       at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
       at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
       at System.Linq.OrderedEnumerable`1.<GetEnumerator>d__1.MoveNext()
       at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty) in /_/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalWriter.cs:line 677
       at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty) in /_/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalWriter.cs:line 179
       at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType) in /_/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalWriter.cs:line 95
       at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType) in /_/Src/Newtonsoft.Json/JsonSerializer.cs:line 1149
       at System.Net.Http.Formatting.BaseJsonMediaTypeFormatter.WriteToStream(Type type, Object value, Stream writeStream, Encoding effectiveEncoding)
       at System.Net.Http.Formatting.JsonMediaTypeFormatter.WriteToStream(Type type, Object value, Stream writeStream, Encoding effectiveEncoding)
       at System.Net.Http.Formatting.BaseJsonMediaTypeFormatter.WriteToStream(Type type, Object value, Stream writeStream, HttpContent content)
       at System.Net.Http.Formatting.BaseJsonMediaTypeFormatter.WriteToStreamAsync(Type type, Object value, Stream writeStream, HttpContent content, TransportContext transportContext, CancellationToken cancellationToken)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Umbraco.Web.WebApi.AngularJsonMediaTypeFormatter.<WriteToStreamAsync>d__1.MoveNext() in D:\a\1\s\src\Umbraco.Web\WebApi\AngularJsonMediaTypeFormatter.cs:line 52
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at System.Web.Http.WebHost.HttpControllerHandler.<WriteBufferedResponseContentAsync>d__22.MoveNext()
    Inner Exception
    System.IO.FileNotFoundException: _4q.tvf
    at Examine.LuceneEngine.Directories.SyncDirectory.OpenInput(String name) in D:\a\1\s\src\Examine\LuceneEngine\Directories\SyncDirectory.cs:line 252
       at Lucene.Net.Store.Directory.OpenInput(String name, Int32 bufferSize) in d:\Lucene.Net\FullRepo\trunk\src\core\Store\Directory.cs:line 98
       at Lucene.Net.Index.TermVectorsReader..ctor(Directory d, String segment, FieldInfos fieldInfos, Int32 readBufferSize, Int32 docStoreOffset, Int32 size) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\TermVectorsReader.cs:line 83
       at Lucene.Net.Index.SegmentReader.CoreReaders.OpenDocStores(SegmentInfo si) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\SegmentReader.cs:line 367
       at Lucene.Net.Index.SegmentReader.Get(Boolean readOnly, Directory dir, SegmentInfo si, Int32 readBufferSize, Boolean doOpenStores, Int32 termInfosIndexDivisor) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\SegmentReader.cs:line 782
       at Lucene.Net.Index.IndexWriter.ReaderPool.Get(SegmentInfo info, Boolean doOpenStores, Int32 readBufferSize, Int32 termsIndexDivisor) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 689
       at Lucene.Net.Index.IndexWriter.ReaderPool.GetReadOnlyClone(SegmentInfo info, Boolean doOpenStores, Int32 termInfosIndexDivisor) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 630
       at Lucene.Net.Index.DirectoryReader..ctor(IndexWriter writer, SegmentInfos infos, Int32 termInfosIndexDivisor) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\DirectoryReader.cs:line 204
       at Lucene.Net.Index.IndexWriter.GetReader(Int32 termInfosIndexDivisor) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 392
       at Lucene.Net.Index.IndexWriter.GetReader() in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 353
       at Examine.LuceneEngine.Providers.LuceneSearcher.ValidateSearcher() in D:\a\1\s\src\Examine\LuceneEngine\Providers\LuceneSearcher.cs:line 204
    Inner Exception
    System.IO.FileNotFoundException: Could not find file 'C:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\_4q.tvf'.
    at Examine.LuceneEngine.Directories.SyncIndexInput.SyncLocally(String fileName) in D:\a\1\s\src\Examine\LuceneEngine\Directories\SyncIndexInput.cs:line 182
       at Examine.LuceneEngine.Directories.SyncIndexInput..ctor(SyncDirectory directory, String name, ILoggingService loggingService) in D:\a\1\s\src\Examine\LuceneEngine\Directories\SyncIndexInput.cs:line 98
       at Examine.LuceneEngine.Directories.SyncDirectory.OpenInput(String name) in D:\a\1\s\src\Examine\LuceneEngine\Directories\SyncDirectory.cs:line 248

Seems like Examine no longer has the permission to regenerate the cache files. Only restarting the app service restore that permission and then from back office we can rebuild the indexes until it stop working again.

System.IO.FileNotFoundException: _c.fnm ---> System.IO.FileNotFoundException: Could not find file 'C:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\_c.fnm'.
File name: 'C:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\_c.fnm'
   at Examine.LuceneEngine.Directories.SyncIndexInput.SyncLocally(String fileName) in D:\a\1\s\src\Examine\LuceneEngine\Directories\SyncIndexInput.cs:line 182
   at Examine.LuceneEngine.Directories.SyncIndexInput..ctor(SyncDirectory directory, String name, ILoggingService loggingService) in D:\a\1\s\src\Examine\LuceneEngine\Directories\SyncIndexInput.cs:line 98
   at Examine.LuceneEngine.Directories.SyncDirectory.OpenInput(String name) in D:\a\1\s\src\Examine\LuceneEngine\Directories\SyncDirectory.cs:line 248
   at Examine.LuceneEngine.Directories.SyncDirectory.OpenInput(String name) in D:\a\1\s\src\Examine\LuceneEngine\Directories\SyncDirectory.cs:line 252
   at Lucene.Net.Index.FieldInfos..ctor(Directory d, String name) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\FieldInfos.cs:line 73
   at Lucene.Net.Index.SegmentReader.CoreReaders..ctor(SegmentReader origInstance, Directory dir, SegmentInfo si, Int32 readBufferSize, Int32 termsIndexDivisor) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\SegmentReader.cs:line 125
   at Lucene.Net.Index.SegmentReader.Get(Boolean readOnly, Directory dir, SegmentInfo si, Int32 readBufferSize, Boolean doOpenStores, Int32 termInfosIndexDivisor) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\SegmentReader.cs:line 779
   at Lucene.Net.Index.IndexWriter.ReaderPool.Get(SegmentInfo info, Boolean doOpenStores, Int32 readBufferSize, Int32 termsIndexDivisor) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 689
   at Lucene.Net.Index.IndexWriter.ReaderPool.Get(SegmentInfo info, Boolean doOpenStores) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 656
   at Lucene.Net.Index.DocumentsWriter.ApplyDeletes(SegmentInfos infos) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\DocumentsWriter.cs:line 1265
   at Lucene.Net.Index.IndexWriter.ApplyDeletes() in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 5367
   at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 4374
   at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 4197
   at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary`2 commitUserData) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 4034
   at Lucene.Net.Index.IndexWriter.Commit(IDictionary`2 commitUserData) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 4116
   at Examine.LuceneEngine.Providers.LuceneIndex.IndexCommiter.TimerRelease() in D:\a\1\s\src\Examine\LuceneEngine\Providers\LuceneIndex.cs:line 1098 

Really appreciate if you can point me in the right direction. Something has changed recently which effecting the Azure app services only.