I have a load balanced site (2 servers) and occasionally, for reasons currently unknown, the search index on the slave node might lose content and stop returning the results it should.

When I look at the Index directory in the Temp folder after this is brought to my attention it looks very much like it was starting to rebuild the index and stopped. Nothing of any interest in the logs and it's not a regular occurrence but is cause for concern at the moment due to a critically important time for the site and in particular the data that disappears.

The data is in an Index of its own and the configuration looks like this:

ExamineIndex.config

<IndexSet SetName="AwardIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/{machinename}/Award/">
    <IndexAttributeFields>
        <add Name="id" />
        <add Name="nodeName" />
        <add Name="updateDate" />
        <add Name="writerName" />
        <add Name="path" />
        <add Name="nodeTypeAlias" />
        <add Name="parentID" />
    </IndexAttributeFields>
    <IncludeNodeTypes>
        <add Name="Award" />
    </IncludeNodeTypes>
</IndexSet>

ExamineSettings.config

ExamineIndexProvider

<add name="AwardIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
   supportUnpublished="false"
   supportProtected="false"
   interval="10"
   analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"/>

ExamineSearchProvider

<add name="AwardSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
     analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" enableLeadingWildcards="true"/>

What I am looking to do is to somehow get notified when the index gets into this state and looking for some ideas.

My thoughts at the moment are to perhaps implement something in a similar fashion the lastsync file and write a index document count to file on disk and then perform some sort of comparison between the file on each server.

Obviously, this is a short-term solution as I would like to identify and fix the problem at source but if anyone has any ideas in the meantime I'd appreciate any input.

Thanks, Simon