We've been having a problem with Lucene indexing for a while already.

Basically quite often when we try to publish a content the indexing just throws an error like this:

2020-01-13 22:22:38,068 [P36840/D2/TLucene Merge Thread #0] ERROR Umbraco.Core.UmbracoApplicationBase - Unhandled exception in AppDomain (terminating)
Lucene.Net.Index.MergePolicy+MergeException: Exception of type 'Lucene.Net.Index.MergePolicy+MergeException' was thrown. ---> System.IO.IOException: read past EOF
   at Lucene.Net.Index.IndexWriter.HandleMergeException(Exception t, OneMerge merge)
   at Lucene.Net.Index.IndexWriter.Merge(OneMerge merge)
   at Lucene.Net.Index.ConcurrentMergeScheduler.DoMerge(OneMerge merge)
   at Lucene.Net.Index.ConcurrentMergeScheduler.MergeThread.Run()
   --- End of inner exception stack trace ---
   at Lucene.Net.Index.ConcurrentMergeScheduler.HandleMergeException(Exception exc)
   at Lucene.Net.Index.ConcurrentMergeScheduler.MergeThread.Run()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

We get Secure Connection Failed on the public site as the result, the content is being saved but not included in cache so it does not display on the website.

Then we are unable to access Examine Management dashboard to restart indexes.

The error sometime slooks like this too:

 2019-12-30 14:49:14,895 [P18524/D2/T28] ERROR Umbraco.Core.UmbracoApplicationBase - Unhandled exception in AppDomain (terminating)
System.IO.IOException: Cannot overwrite: D:\Inetpub\vhosts\mysite.com\httpdocs\App_Data\TEMP\ExamineIndexes\Internal\Index\_pe.cfs
   at Lucene.Net.Store.FSDirectory.InitOutput(String name)
   at Lucene.Net.Store.SimpleFSDirectory.CreateOutput(String name)
   at Lucene.Net.Index.CompoundFileWriter.Close()
   at Lucene.Net.Index.DocumentsWriter.CreateCompoundFile(String segment)
   at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes)
   at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes)
   at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary`2 commitUserData, Boolean internal_Renamed)
   at Lucene.Net.Index.IndexWriter.Commit(IDictionary`2 commitUserData)
   at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.TimerRelease() in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1711
   at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.<ScheduleCommit>b__7_0(Object _) in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1663
   at System.Threading.TimerQueueTimer.CallCallbackInContext(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

In such case publishing a content becomes practically ineffective.

Any idea what's the cause? And how to fix it? I can see the index becomes corrupted but why?

I have checked several similar issues like these:

https://stackoverflow.com/questions/30468541/umbraco-indexing-error-filling-up-logs

https://issues.apache.org/jira/browse/LUCENENET-527 but none of them is our case - I can't see any 0kB segment files.