Make sure you follow the load balancing guide to the letter:

https://our.umbraco.org/documentation/Getting-Started/Setup/Server-Setup/load-balancing/

LocalOnly means that the indexes will be stored on the webworker machines that the site is running on and not in the central folder, which is definitely what you want in a LB setup.

Note: I'm no LB expert so for your next 12 questions the problem will probably be: dunno. :) I know that when the documentation is followed completely it should work.

I'm starting to think now that it's nothing to do with scaling - I don't think my site is indexing at all anymore. Is there any common scenario that might cause that to happen? I'll go back and check my config.

Is the site under heavy load? If not, I would turn switch to debug level. To be sure you are not missing something. My guess is that we are missing some events where it is stated the Examine index can not be created. Also the "directoryNotFound" exception means that the startup or server sync has had a problem earlier.

PS: Is your appsetting umbracoContentXMLUseLocalTemp set to true?

Hi,

When you bring a new site online (scale out) it will not be synced - it will not have the umbraco.config xml cache file and will not have any Examine indexes. When this happens, we call this a 'cold boot'. The process is

• Check the local \App_Data\TEMP\DistCache[SERVERNAME][AppDomainAppId]-lastsynced.txt file to see what data the site has in it's caches. If this file doesn't exist, it performs a cold boot = rebuilds the xml cache file and Examine indexes
• If this file does exist but the number of new instructions in the instructions table exceeds the number MaxProcessingInstructionCount then a cold boot will occur (default is 1000). Otherwise it will read in the un-processed instructions and sync all caches to the last id in the instructions table

So, when you bring a new site online you should get the message indicating that the lastsynced.txt doesn't exist and that a cold boot will occur. You should get logs indicating that the indexes will be rebuilding. You can also force a cold boot by deleting the lastsynced.txt file ... though this should only be done for testing purposes since you definitely don't want to cold boot every time your site restarts.

Hi, we're having the same issue, the result is the production website unexpectedly and suddenly empty.

We have only one instance enabled on Azure, and this problem happens sometimes, when Azure decides to change machine, terminating the current web application on the current server, and bringing a new instance on a new machine online.

From the log file, I see that the cold boot is started, but at the end no Examine indexes are created. And in the meanwhile SaveLastSynced(lastId) is executed, without actually having synced. This put the application in an invalid state.

I cannot understand why Examine indexes aren't created.. I attach the log when the application starts on a new machine, and my ExamineSettings.config.

Log:

2016-03-11 22:07:23,624 [P3412/D2/T1] INFO Umbraco.Core.CoreBootManager - Umbraco 7.3.7 application starting on RD000D3A22DBDF

2016-03-11 22:07:23,702 [P3412/D2/T1] INFO Umbraco.Core.PluginManager - Determining hash of code files on disk

2016-03-11 22:07:23,780 [P3412/D2/T1] INFO Umbraco.Core.PluginManager - Hash determined (took 83ms)

{...}

2016-03-11 22:07:28,888 [P3412/D2/T1] INFO Umbraco.Core.DatabaseContext - CanConnect = True

2016-03-11 22:07:28,916 [P3412/D2/T1] WARN Umbraco.Core.Sync.DatabaseServerMessenger - No last synced Id found, this generally means this is a new server/install. The server will build its caches and indexes, and then adjust its last synced Id to the latest found in the database and maintain cache updates based on that Id.

2016-03-11 22:07:28,963 [P3412/D2/T1] INFO umbraco.content - Load Xml from file...

2016-03-11 22:07:28,963 [P3412/D2/T1] ERROR umbraco.content - Failed to load Xml from file. System.IO.DirectoryNotFoundException: Could not find a part of the path 'D:\local\Temporary ASP.NET Files\root\25b6bfe3\4b69c977\UmbracoData\umbraco.config'. at System.IO._Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITYATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share) at umbraco.content.LoadXmlFromFile()

2016-03-11 22:07:28,979 [P3412/D2/T1] INFO umbraco.content - Loading content from database...

2016-03-11 22:07:30,588 [P3412/D2/T1] INFO umbraco.content - Loading content from database...

2016-03-11 22:07:31,808 [P3412/D2/T1] INFO Umbraco.Web.Cache.CacheRefresherEventHandler - Initializing Umbraco internal event handlers for cache refreshing

2016-03-11 22:07:31,855 [P3412/D2/T1] INFO Umbraco.Web.Search.ExamineEvents - Initializing Examine and binding to business logic events

2016-03-11 22:07:31,855 [P3412/D2/T1] INFO Umbraco.Web.Search.ExamineEvents - Adding examine event handlers for index providers: 4

2016-03-11 22:07:31,871 [P3412/D2/T1] INFO MyProject.EventHandlers.MyProjectApplicationEventHandler - OnApplicationStarted: add OnGatheringNodeData to ExternalIndexer

2016-03-11 22:07:33,982 [P3412/D2/T1] INFO Umbraco.Core.CoreBootManager - Umbraco application startup complete (took 11087ms)

2016-03-11 22:07:34,636 [P3412/D2/T13] INFO umbraco.content - Save Xml to file...

2016-03-11 22:07:34,636 [P3412/D2/T6] INFO umbraco.content - Saved Xml to file.

2016-03-11 22:34:41,331 [P3412/D2/T35] INFO Umbraco.Core.Sync.ApplicationUrlHelper - ApplicationUrl: https://MyProject.azurewebsites.net/umbraco (using scheduledTasks/@baseUrl)

ExamineSettings.config

<Examine> <ExamineIndexProviders> <providers>
  <add name="InternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
       supportUnpublished="true"
       supportProtected="true"
       analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net"
       useTempStorage="Sync"
       tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>

  <add name="InternalMemberIndexer" type="UmbracoExamine.UmbracoMemberIndexer, UmbracoExamine"
       supportUnpublished="true"
       supportProtected="true"
       analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"
       useTempStorage="Sync"
       tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>

  <!-- default external indexer, which excludes protected and unpublished pages-->
  <add name="ExternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
       useTempStorage="Sync"
       tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>


  <add name="ExternalMemberModelIndexer" type="UmbracoExamine.UmbracoMemberIndexer, UmbracoExamine"
       supportUnpublished="false"
       supportProtected="true"
       analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"
       useTempStorage="Sync"
       tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>
</providers> </ExamineIndexProviders> <ExamineSearchProviders defaultProvider="ExternalSearcher">
<providers>
  <add name="InternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
       analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net"
       useTempStorage="Sync"
       tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>

  <add name="ExternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
       analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" enableLeadingWildcard="true"
       useTempStorage="Sync"
       tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>

  <add name="InternalMemberSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
       analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" enableLeadingWildcard="true"
       useTempStorage="Sync"
       tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>

  <add name="ExternalMemberModelSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
       analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" enableLeadingWildcard="true"
       useTempStorage="Sync"
       tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>

</providers></ExamineSearchProviders></Examine>

Thanks Giordano, your config is slightly different to mine. For reference here's my ExamineSettings.config:

<Examine>
  <ExamineIndexProviders>
    <providers>
      <add name="InternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
           supportUnpublished="true"
           supportProtected="true" useTempStorage="LocalOnly"
           analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net"/>

      <add name="InternalMemberIndexer" type="UmbracoExamine.UmbracoMemberIndexer, UmbracoExamine"
           supportUnpublished="true"
           supportProtected="true" useTempStorage="LocalOnly"
           analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"/>

        <!-- default external indexer, which excludes protected and unpublished pages-->
        <add name="ExternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"  useTempStorage="LocalOnly" />

    </providers>
  </ExamineIndexProviders>

  <ExamineSearchProviders defaultProvider="ExternalSearcher">
    <providers>
      <add name="InternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
           analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net" useTempStorage="LocalOnly"/>

      <add name="ExternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine" useTempStorage="LocalOnly"/>

      <add name="InternalMemberSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
           analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" enableLeadingWildcard="true" useTempStorage="LocalOnly"/>

    </providers>
  </ExamineSearchProviders>

</Examine>

Hi James, thanks for posting your configuration.

If I well understand the Shannon's post, if I stop my Azure web app, then I remove the \App_Data\TEMP\DistCache[SERVERNAME][AppDomainAppId]-lastsynced.txt file, and I start again the web app, xml cache file and Examine indexes should be rebuilt. But this doesn't happen to my web application.

I see 2 differences between my and your ExamineSettings configuration

1. You set useTempStorage="LocalOnly". I tried using it, but the result doesn't change
2. I set tempStorageDirectory See: http://issues.umbraco.org/issue/U4-7614

I hope someone could help us.

Thanks, Giordano

Hi Giordano,

I didn't see any relevant behaviour difference between LocalOnly and Sync either. I'm not sure about the tempStorageDirectory, I haven't seen that before and is not mentioned in the Azure / Load Balancing guidance.

When I get a chance I will try out Shannon's suggestions.

James.

Hi, as a workaround, I have extended the WebBootManager class with a custom one, and I have overridden Complete(Action

    public override IBootManager Complete(Action<ApplicationContext> afterComplete)
    {
        base.Complete(afterComplete);
        LogHelper.Info(typeof(CustomWebBootManager), "Custom rebuilding index");

        ExamineManager.Instance.IndexProviderCollection.Where(i => !i.IndexExists())
            .ForEach(i =>
            {
                i.RebuildIndex();
            });
        return this;
    }

Hi,

Do you know yet if that has had the desired effect?

James.

Hi James,

from the logs I can see the custom rebuilding is called, but I don't know what happens next time Azure will switch machine.

thanks,

Giordano

Has anyone managed to resolve this problem? I'm experiencing exact same issue. The custom WebBootManager above unfortunately didn't help at all :(

Hi Bartosz,

I still have this issue, but I have not tried anything else to fix it. We have monitoring set up to alert us when the search pages stop working and we then log in and manually run the indexes. Not ideal but I can't invest any more time on this at the moment. It only seems to happen once every 2 weeks or so.

We'll probably upgrade to 7.5 (or more likely 7.5.2 ;0)) once that comes out and we're hoping this will just start working again...

James.

Any update on this?

We are seeing the same issue as well. Every time Azure migrates the web application to a new server - our Examine indexes are empty and do not rebuild.. We are using a 'staging' deployment slot in our Azure WebApp - not sure if it makes any difference.

I haven't tried the BootManager extension. Has it worked for anyone else?

EDIT: I just noticed an umbraco issue has been created for this thread http://issues.umbraco.org/issue/U4-8249

Does anyone have anything further to report? I note the issue on the tracker, but it seems to be mothballed and I really don't believe this a live issue - otherwise Umbraco Cloud would have a serious problem. Has anyone managed to fix this with a config change, an Umbraco upgrade or a workaround?

I'm planning to try @Giordiano's index rebuilder but as the problem only surfaces when Azure do some overnight maintenance, I'm not sure I'll be able to test it.

Would appreciate any further info if anyone has something to add.

I've sort of resolved this issue, I could have sworn I posted the solution somewhere on the Umbraco forum, but can't find it.

Anyway, in 7.3.x cold boot indexing is botched and will not work, so every time Umbraco needs to do so it will leave you with (almost) empty index folders and an index with 0-10 items inside it. The only way I could solve it without updating to 7.4 was removing the {machineName} from the index location path. Then you need to rebuild the index manually in CMS on the content management node and also on the content delivery node (hopefully you didn't block the /umbraco path permanently). It's also good to set the RebuildOnAppStart = false in your ExamineSettings.config file, so that Umbraco will not try to do a cold boot index rebuild at any time.

This allows the indexes to persist even if Azure decides to switch the underlying machine of your AppService, while it will still allow Umbraco to update the index when changes are made in CMS. However, it has two big drawbacks:

• You can't have multiple Content Delivery nodes in Azure as they can't both achieve a lock in one folder,
• every time you make many changes at once (e.g. Courier many new content items) you will need to manually rebuild the Examine index both on CM and CD nodes.

Fortunately, updating to 7.4.x fixes the cold boot indexing issue. If anyone is curious, I use a following setup:

• staging slot for CD node,
• RunAsync = false on each index,
• RebuildOnAppStart = true or just removed, it's true by default,
• {machineName} in the index location path,
• 1 CM node and 2 CD nodes.

The Content Delivery nodes have separate folders based on the current machine name, so there's no problem with locking their indexes. Each time I deploy a new version of CMS project, it's deployed to the staging slot, then Umbraco starts rebuilding the index (usually there are many changes). In the meantime a slot-swap command is executed, it awaits till the machine responds with 200 and then swaps the staging slot for production. However, the RunAsync = false setting prevents the staging slot from returning 200 before the indexes are ready. The slot-swap is being retried each time it fails, so when the indexes are ready, the actual swap takes place. And voila a deploy with no downtime is born.

Thanks for taking the time to reply Bartosz. I'm already on v7.4.3 so I don't think your scenario is the same as mine. I will try the runAsync flag though, that definitely sounds promising. Will post back here any findings. #H5YR

We are seeing the same error on 7.4.3, but don't think it's a problem with how the indexes are built, but with the query used to retrieve media items. We're seeing it even on a single local development instance.

If I use the Examine Manager to run a query, it does find the media item in the external index, e.g.

__NodeId:2020

However running a debug instance of Umbraco, and stepping through PublishedMediaCache.GetUmbracoMediaCacheValues, I can see the filter being built looks like this:

+(+null -__Path:-1,-21,*) +__IndexType:media

even though the inline code comment says it should build one like this:

+(+__NodeId:2020 -__Path:-1,-21,*) +__IndexType:media

The null is being returned from Examine, but I'm not yet sure whether this is an Examine bug, a Lucene.NET bug, or whether Umbraco is using Examine incorrectly.

I'm trying my own builds of Examine and Lucene.NET to try and find out.

is working on a reply...