Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • James Strugnell 84 posts 192 karma points
    Feb 22, 2016 @ 14:05
    James Strugnell
    0

    Azure Website not initializing Examine Index after scaling

    Hi,

    We upgraded one of our websites from 7.3.1 to 7.3.7 last week. This particular website is required to scale out to run at least 2 instances at the weekend. When the site scaled out to 2 instances, for the first time since we upgraded, I noticed that the Examine indexes were not being initialized on the newly provisioned instance. I noticed this due to the thousands of lines in the log files resembling the following:

    Could not retrieve media * from Examine index, reverting to looking up media via legacy library.GetMedia method

    Search also did not work on the scaled out instance.

    The site did not appear to have any obvious performance issues as a result of this but I imagine there must be, not least due to all the logging it was having to do. I resolved this temporarily by logging into the site and ensuring I was on the scaled instance (by repeatedly deleting the ARRAfinnity cookie) and then running the indexing manually on that server, through the Examine Management page in the browser.

    I thought I would investigate this further today so I scaled the site up from 1 to 2 instances again. I also reviewed the Umbraco code in DatabaseServerMessenger.cs as this is the class that seems to be responsible for the initialization. This helped me get a basic understanding of the Initialisation code (although I can't quite tell which bit of code kicks off the Examine Indexing). After scaling I again saw the same issue. The new, secondary server was not initializing it's Examine indexes.

    Looking in the DistCache and ExamineIndexes folders in the temp directory, I could see that the DistCache structure was correctly setting the Last Sync ID for the new machine. However, the Examine Indexes were remaining empty for the new machine. At this point I should state that I have also changed my Examine config to use the UseTempStorage="LocalOnly" setting rather than "Sync", but I did this in an attempt to resolve this issue, having read about this recommended configuration in another post. So it may be that the LocalOnly setting is the reason for not being able to see any indexes within the Temp directory (as it is only being stored locally on the machine, not in the website share Temp directory).

    However, when I manually ran the Examine Indexes again, this did then populate the associated folder in the Temp/ExamineIndexes/{MACHINENAME} folder. So I'm not sure I'm interpreting the "LocalOnly" setting correctly.

    I can't tell at the moment whether this change in behaviour is due to losing a configuration value during the upgrade process or if there is a bug in 7.3.7, it would be great if someone could help me verify that?

    If I need to provide any more information on my configuration please let me know. Thanks for your help.

    James.

  • Sebastiaan Janssen 4988 posts 15101 karma points MVP admin hq
    Feb 22, 2016 @ 15:10
    Sebastiaan Janssen
    0

    Make sure you follow the load balancing guide to the letter:

    https://our.umbraco.org/documentation/Getting-Started/Setup/Server-Setup/load-balancing/

    LocalOnly means that the indexes will be stored on the webworker machines that the site is running on and not in the central folder, which is definitely what you want in a LB setup.

    Note: I'm no LB expert so for your next 12 questions the problem will probably be: dunno. :) I know that when the documentation is followed completely it should work.

  • James Strugnell 84 posts 192 karma points
    Feb 22, 2016 @ 16:26
    James Strugnell
    0

    Thanks Sebastiaan, I will review the guidance again. I should point out that this was working prior to the recent upgrade, so it seems that either my upgrade process has changed something in the configuration (my mistake), or the new version has introduced/revealed the issue. I'll continue investigating.

  • Sebastiaan Janssen 4988 posts 15101 karma points MVP admin hq
    Feb 22, 2016 @ 16:31
    Sebastiaan Janssen
    0

    Strange, if there are no Examine indexes, Umbraco will start building them when it starts so they should be back as soon as the first page appears from the fresh webserver.

  • James Strugnell 84 posts 192 karma points
    Feb 22, 2016 @ 17:04
    James Strugnell
    0

    I'm starting to think now that it's nothing to do with scaling - I don't think my site is indexing at all anymore. Is there any common scenario that might cause that to happen? I'll go back and check my config.

  • Damiaan 440 posts 1295 karma points MVP 4x c-trib
    Feb 22, 2016 @ 17:09
    Damiaan
    0

    I have had issues with indexing in the past with custom events swallowing exceptions.
    Or I forgot to setup write permissions.

  • James Strugnell 84 posts 192 karma points
    Feb 24, 2016 @ 09:27
    James Strugnell
    0

    Hi Damiaan, I've investigated this further, without a great deal of success though. Another thing I should have pointed out is that if I manually run the indexes through the browser then they run just fine. It also seems to pick up newly published content ok. It's just when the site is on a new server (happens quite a lot in Azure Web Apps) then the index doesn't initialize itself correctly. It happened last night again, so I've had to go in and manually kick off the indexer.

    These are the log entries when a new instance is added: enter image description here

    Could permissions be a problem if I'm able to run it just fine manually through the browser? We are using the LocalOnly setting which means it's putting the index in the ASP.Net temporary file location, so presumably that would also rule out permissions as being the cause.

    Thanks, James.

  • Damiaan 440 posts 1295 karma points MVP 4x c-trib
    Feb 24, 2016 @ 11:15
    Damiaan
    0

    Is the site under heavy load? If not, I would turn switch to debug level. To be sure you are not missing something. My guess is that we are missing some events where it is stated the Examine index can not be created. Also the "directoryNotFound" exception means that the startup or server sync has had a problem earlier.

    PS: Is your appsetting umbracoContentXMLUseLocalTemp set to true?

  • James Strugnell 84 posts 192 karma points
    Feb 24, 2016 @ 11:22
    James Strugnell
    0

    Hi Damiaan,

    Yes we are using that setting. I'm not concerned about that "error" in the logs - that is just because it is a new machine/instance and thus the umbraco.config doesn't exist yet. Not sure that should be an error in the logs, but I believe we can ignore that.

    I will look to increase the logging level and scale up later today, to see if we get any additional information.

    Thanks for your input.

    James.

  • James Strugnell 84 posts 192 karma points
    Feb 24, 2016 @ 12:50
    James Strugnell
    0

    OK, I've increased the logging (DEBUG) and apart from flooding the logs I don't think it provides anything useful. Below are all of the logs that mention "Examine":

    *2016-02-24 12:27:39,956 [11] DEBUG Umbraco.Core.CoreBootManager - Executing Umbraco.Web.Search.ExamineEvents in ApplicationInitialized

    2016-02-24 12:27:40,591 [11] DEBUG Umbraco.Core.CoreBootManager - Executing Umbraco.Web.Search.ExamineEvents in ApplicationStarting

    2016-02-24 12:27:43,688 [11] DEBUG UmbracoExamine.DataServices.UmbracoLogService - InternalIndexer indexer initializing, Provider=InternalIndexer, NodeId=-1

    2016-02-24 12:27:44,047 [11] DEBUG UmbracoExamine.DataServices.UmbracoLogService - InternalMemberIndexer indexer initializing, Provider=InternalMemberIndexer, NodeId=-1

    2016-02-24 12:27:44,125 [11] DEBUG UmbracoExamine.DataServices.UmbracoLogService - ExternalIndexer indexer initializing, Provider=ExternalIndexer, NodeId=-1

    2016-02-24 12:27:55,513 [11] DEBUG Umbraco.Core.CoreBootManager - Executing Umbraco.Web.Search.ExamineEvents in ApplicationStarted

    2016-02-24 12:27:55,525 [11] INFO Umbraco.Web.Search.ExamineEvents - Initializing Examine and binding to business logic events

    2016-02-24 12:27:55,525 [11] INFO Umbraco.Web.Search.ExamineEvents - Adding examine event handlers for index providers: 3

    2016-02-24 12:27:55,525 [11] DEBUG Umbraco.Core.CoreBootManager - Complete (took 12ms)

    2016-02-24 12:28:12,043 [11] WARN Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache - Could not retrieve media 16684 from Examine index, reverting to looking up media via legacy library.GetMedia method*

    There's plenty more lines in the log files but it's hard to know what's relevant.

  • James Strugnell 84 posts 192 karma points
    Feb 24, 2016 @ 13:03
    James Strugnell
    0

    I've just manually run the index just to capture the process in a log file and these are the key first few lines I see -

    2016-02-24 12:52:47,851 [11] DEBUG Umbraco.Web.UmbracoModule - Begin request: http://[DOMAIN]/umbraco/BackOffice/Api/ExamineManagementApi/PostRebuildIndex?indexerName=ExternalIndexer. 2016-02-24 12:52:47,913 [11] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Rebuilding index, Provider=ExternalIndexer, NodeId=-1 2016-02-24 12:52:47,945 [11] DEBUG UmbracoExamine.DataServices.UmbracoLogService - (ExternalIndexer) PerformIndexAll with XPATH: //[(number(@id) > 0 and (@isDoc or @nodeTypeAlias))], Provider=ExternalIndexer, NodeId=-1 2016-02-24 12:52:49,324 [11] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Index deleted for term: __IndexType with value content, Provider=ExternalIndexer, NodeId=-1 2016-02-24 12:52:49,804 [11] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Index created for node 1090, Provider=ExternalIndexer, NodeId=1090 2016-02-24 12:52:49,804 [11] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Index created for node 3106, Provider=ExternalIndexer, NodeId=3106*

    Repeat, repeat, repeat (for that last line as it indexes each node)

    So none of this shows up when the site first loads, only with a force index.

  • Shannon Deminick 1521 posts 5241 karma points
    Mar 07, 2016 @ 12:11
    Shannon Deminick
    0

    Hi,

    When you bring a new site online (scale out) it will not be synced - it will not have the umbraco.config xml cache file and will not have any Examine indexes. When this happens, we call this a 'cold boot'. The process is

    • Check the local \App_Data\TEMP\DistCache[SERVERNAME][AppDomainAppId]-lastsynced.txt file to see what data the site has in it's caches. If this file doesn't exist, it performs a cold boot = rebuilds the xml cache file and Examine indexes
    • If this file does exist but the number of new instructions in the instructions table exceeds the number MaxProcessingInstructionCount then a cold boot will occur (default is 1000). Otherwise it will read in the un-processed instructions and sync all caches to the last id in the instructions table

    So, when you bring a new site online you should get the message indicating that the lastsynced.txt doesn't exist and that a cold boot will occur. You should get logs indicating that the indexes will be rebuilding. You can also force a cold boot by deleting the lastsynced.txt file ... though this should only be done for testing purposes since you definitely don't want to cold boot every time your site restarts.

  • Giordano 8 posts 78 karma points
    Mar 15, 2016 @ 11:35
    Giordano
    0

    Hi, we're having the same issue, the result is the production website unexpectedly and suddenly empty.

    We have only one instance enabled on Azure, and this problem happens sometimes, when Azure decides to change machine, terminating the current web application on the current server, and bringing a new instance on a new machine online.

    From the log file, I see that the cold boot is started, but at the end no Examine indexes are created. And in the meanwhile SaveLastSynced(lastId) is executed, without actually having synced. This put the application in an invalid state.

    I cannot understand why Examine indexes aren't created.. I attach the log when the application starts on a new machine, and my ExamineSettings.config.

    Log:

    2016-03-11 22:07:23,624 [P3412/D2/T1] INFO Umbraco.Core.CoreBootManager - Umbraco 7.3.7 application starting on RD000D3A22DBDF

    2016-03-11 22:07:23,702 [P3412/D2/T1] INFO Umbraco.Core.PluginManager - Determining hash of code files on disk

    2016-03-11 22:07:23,780 [P3412/D2/T1] INFO Umbraco.Core.PluginManager - Hash determined (took 83ms)

    {...}

    2016-03-11 22:07:28,888 [P3412/D2/T1] INFO Umbraco.Core.DatabaseContext - CanConnect = True

    2016-03-11 22:07:28,916 [P3412/D2/T1] WARN Umbraco.Core.Sync.DatabaseServerMessenger - No last synced Id found, this generally means this is a new server/install. The server will build its caches and indexes, and then adjust its last synced Id to the latest found in the database and maintain cache updates based on that Id.

    2016-03-11 22:07:28,963 [P3412/D2/T1] INFO umbraco.content - Load Xml from file...

    2016-03-11 22:07:28,963 [P3412/D2/T1] ERROR umbraco.content - Failed to load Xml from file. System.IO.DirectoryNotFoundException: Could not find a part of the path 'D:\local\Temporary ASP.NET Files\root\25b6bfe3\4b69c977\UmbracoData\umbraco.config'. at System.IO._Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITYATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share) at umbraco.content.LoadXmlFromFile()

    2016-03-11 22:07:28,979 [P3412/D2/T1] INFO umbraco.content - Loading content from database...

    2016-03-11 22:07:30,588 [P3412/D2/T1] INFO umbraco.content - Loading content from database...

    2016-03-11 22:07:31,808 [P3412/D2/T1] INFO Umbraco.Web.Cache.CacheRefresherEventHandler - Initializing Umbraco internal event handlers for cache refreshing

    2016-03-11 22:07:31,855 [P3412/D2/T1] INFO Umbraco.Web.Search.ExamineEvents - Initializing Examine and binding to business logic events

    2016-03-11 22:07:31,855 [P3412/D2/T1] INFO Umbraco.Web.Search.ExamineEvents - Adding examine event handlers for index providers: 4

    2016-03-11 22:07:31,871 [P3412/D2/T1] INFO MyProject.EventHandlers.MyProjectApplicationEventHandler - OnApplicationStarted: add OnGatheringNodeData to ExternalIndexer

    2016-03-11 22:07:33,982 [P3412/D2/T1] INFO Umbraco.Core.CoreBootManager - Umbraco application startup complete (took 11087ms)

    2016-03-11 22:07:34,636 [P3412/D2/T13] INFO umbraco.content - Save Xml to file...

    2016-03-11 22:07:34,636 [P3412/D2/T6] INFO umbraco.content - Saved Xml to file.

    2016-03-11 22:34:41,331 [P3412/D2/T35] INFO Umbraco.Core.Sync.ApplicationUrlHelper - ApplicationUrl: https://MyProject.azurewebsites.net/umbraco (using scheduledTasks/@baseUrl)

    ExamineSettings.config

    <Examine> <ExamineIndexProviders> <providers>
      <add name="InternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
           supportUnpublished="true"
           supportProtected="true"
           analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net"
           useTempStorage="Sync"
           tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>
    
      <add name="InternalMemberIndexer" type="UmbracoExamine.UmbracoMemberIndexer, UmbracoExamine"
           supportUnpublished="true"
           supportProtected="true"
           analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"
           useTempStorage="Sync"
           tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>
    
      <!-- default external indexer, which excludes protected and unpublished pages-->
      <add name="ExternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
           useTempStorage="Sync"
           tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>
    
    
      <add name="ExternalMemberModelIndexer" type="UmbracoExamine.UmbracoMemberIndexer, UmbracoExamine"
           supportUnpublished="false"
           supportProtected="true"
           analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"
           useTempStorage="Sync"
           tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>
    </providers> </ExamineIndexProviders> <ExamineSearchProviders defaultProvider="ExternalSearcher">
    <providers>
      <add name="InternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
           analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net"
           useTempStorage="Sync"
           tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>
    
      <add name="ExternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
           analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" enableLeadingWildcard="true"
           useTempStorage="Sync"
           tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>
    
      <add name="InternalMemberSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
           analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" enableLeadingWildcard="true"
           useTempStorage="Sync"
           tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>
    
      <add name="ExternalMemberModelSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
           analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" enableLeadingWildcard="true"
           useTempStorage="Sync"
           tempStorageDirectory="UmbracoExamine.LocalStorage.AzureLocalStorageDirectory, UmbracoExamine"/>
    
    </providers></ExamineSearchProviders></Examine>
    
  • James Strugnell 84 posts 192 karma points
    Mar 16, 2016 @ 12:07
    James Strugnell
    0

    Thanks Giordano, your config is slightly different to mine. For reference here's my ExamineSettings.config:

    <Examine>
      <ExamineIndexProviders>
        <providers>
          <add name="InternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
               supportUnpublished="true"
               supportProtected="true" useTempStorage="LocalOnly"
               analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net"/>
    
          <add name="InternalMemberIndexer" type="UmbracoExamine.UmbracoMemberIndexer, UmbracoExamine"
               supportUnpublished="true"
               supportProtected="true" useTempStorage="LocalOnly"
               analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"/>
    
            <!-- default external indexer, which excludes protected and unpublished pages-->
            <add name="ExternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"  useTempStorage="LocalOnly" />
    
        </providers>
      </ExamineIndexProviders>
    
      <ExamineSearchProviders defaultProvider="ExternalSearcher">
        <providers>
          <add name="InternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
               analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net" useTempStorage="LocalOnly"/>
    
          <add name="ExternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine" useTempStorage="LocalOnly"/>
    
          <add name="InternalMemberSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
               analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" enableLeadingWildcard="true" useTempStorage="LocalOnly"/>
    
        </providers>
      </ExamineSearchProviders>
    
    </Examine>
    
  • Giordano 8 posts 78 karma points
    Mar 16, 2016 @ 13:15
    Giordano
    0

    Hi James, thanks for posting your configuration.

    If I well understand the Shannon's post, if I stop my Azure web app, then I remove the \App_Data\TEMP\DistCache[SERVERNAME][AppDomainAppId]-lastsynced.txt file, and I start again the web app, xml cache file and Examine indexes should be rebuilt. But this doesn't happen to my web application.

    I see 2 differences between my and your ExamineSettings configuration

    1. You set useTempStorage="LocalOnly". I tried using it, but the result doesn't change
    2. I set tempStorageDirectory See: http://issues.umbraco.org/issue/U4-7614

    I hope someone could help us.

    Thanks, Giordano

  • James Strugnell 84 posts 192 karma points
    Mar 16, 2016 @ 14:08
    James Strugnell
    0

    Hi Giordano,

    I didn't see any relevant behaviour difference between LocalOnly and Sync either. I'm not sure about the tempStorageDirectory, I haven't seen that before and is not mentioned in the Azure / Load Balancing guidance.

    When I get a chance I will try out Shannon's suggestions.

    James.

  • Giordano 8 posts 78 karma points
    Mar 17, 2016 @ 13:47
    Giordano
    0

    Hi, as a workaround, I have extended the WebBootManager class with a custom one, and I have overridden Complete(Action

        public override IBootManager Complete(Action<ApplicationContext> afterComplete)
        {
            base.Complete(afterComplete);
            LogHelper.Info(typeof(CustomWebBootManager), "Custom rebuilding index");
    
            ExamineManager.Instance.IndexProviderCollection.Where(i => !i.IndexExists())
                .ForEach(i =>
                {
                    i.RebuildIndex();
                });
            return this;
        }
    
  • James Strugnell 84 posts 192 karma points
    Mar 17, 2016 @ 15:27
    James Strugnell
    0

    Hi,

    Do you know yet if that has had the desired effect?

    James.

  • Giordano 8 posts 78 karma points
    Mar 17, 2016 @ 15:59
    Giordano
    0

    Hi James,

    from the logs I can see the custom rebuilding is called, but I don't know what happens next time Azure will switch machine.

    thanks,

    Giordano

  • Bartosz Ladziński 20 posts 154 karma points
    Aug 03, 2016 @ 16:05
    Bartosz Ladziński
    0

    Has anyone managed to resolve this problem? I'm experiencing exact same issue. The custom WebBootManager above unfortunately didn't help at all :(

  • James Strugnell 84 posts 192 karma points
    Aug 03, 2016 @ 16:11
    James Strugnell
    0

    Hi Bartosz,

    I still have this issue, but I have not tried anything else to fix it. We have monitoring set up to alert us when the search pages stop working and we then log in and manually run the indexes. Not ideal but I can't invest any more time on this at the moment. It only seems to happen once every 2 weeks or so.

    We'll probably upgrade to 7.5 (or more likely 7.5.2 ;0)) once that comes out and we're hoping this will just start working again...

    James.

  • Cameron 1 post 71 karma points
    Sep 07, 2016 @ 02:58
    Cameron
    0

    Any update on this?

    We are seeing the same issue as well. Every time Azure migrates the web application to a new server - our Examine indexes are empty and do not rebuild.. We are using a 'staging' deployment slot in our Azure WebApp - not sure if it makes any difference.

    I haven't tried the BootManager extension. Has it worked for anyone else?

    EDIT: I just noticed an umbraco issue has been created for this thread http://issues.umbraco.org/issue/U4-8249

  • Barry Fogarty 493 posts 1129 karma points
    Nov 03, 2016 @ 12:32
    Barry Fogarty
    0

    Does anyone have anything further to report? I note the issue on the tracker, but it seems to be mothballed and I really don't believe this a live issue - otherwise Umbraco Cloud would have a serious problem. Has anyone managed to fix this with a config change, an Umbraco upgrade or a workaround?

    I'm planning to try @Giordiano's index rebuilder but as the problem only surfaces when Azure do some overnight maintenance, I'm not sure I'll be able to test it.

    Would appreciate any further info if anyone has something to add.

  • Bartosz Ladziński 20 posts 154 karma points
    Nov 03, 2016 @ 13:06
    Bartosz Ladziński
    0

    I've sort of resolved this issue, I could have sworn I posted the solution somewhere on the Umbraco forum, but can't find it.

    Anyway, in 7.3.x cold boot indexing is botched and will not work, so every time Umbraco needs to do so it will leave you with (almost) empty index folders and an index with 0-10 items inside it. The only way I could solve it without updating to 7.4 was removing the {machineName} from the index location path. Then you need to rebuild the index manually in CMS on the content management node and also on the content delivery node (hopefully you didn't block the /umbraco path permanently). It's also good to set the RebuildOnAppStart = false in your ExamineSettings.config file, so that Umbraco will not try to do a cold boot index rebuild at any time.

    This allows the indexes to persist even if Azure decides to switch the underlying machine of your AppService, while it will still allow Umbraco to update the index when changes are made in CMS. However, it has two big drawbacks:

    • You can't have multiple Content Delivery nodes in Azure as they can't both achieve a lock in one folder,
    • every time you make many changes at once (e.g. Courier many new content items) you will need to manually rebuild the Examine index both on CM and CD nodes.

    Fortunately, updating to 7.4.x fixes the cold boot indexing issue. If anyone is curious, I use a following setup:

    • staging slot for CD node,
    • RunAsync = false on each index,
    • RebuildOnAppStart = true or just removed, it's true by default,
    • {machineName} in the index location path,
    • 1 CM node and 2 CD nodes.

    The Content Delivery nodes have separate folders based on the current machine name, so there's no problem with locking their indexes. Each time I deploy a new version of CMS project, it's deployed to the staging slot, then Umbraco starts rebuilding the index (usually there are many changes). In the meantime a slot-swap command is executed, it awaits till the machine responds with 200 and then swaps the staging slot for production. However, the RunAsync = false setting prevents the staging slot from returning 200 before the indexes are ready. The slot-swap is being retried each time it fails, so when the indexes are ready, the actual swap takes place. And voila a deploy with no downtime is born.

  • Barry Fogarty 493 posts 1129 karma points
    Nov 03, 2016 @ 14:55
    Barry Fogarty
    0

    Thanks for taking the time to reply Bartosz. I'm already on v7.4.3 so I don't think your scenario is the same as mine. I will try the runAsync flag though, that definitely sounds promising. Will post back here any findings. #H5YR

  • Julian C. 16 posts 73 karma points
    Dec 02, 2016 @ 10:57
    Julian C.
    0

    We are seeing the same error on 7.4.3, but don't think it's a problem with how the indexes are built, but with the query used to retrieve media items. We're seeing it even on a single local development instance.

    If I use the Examine Manager to run a query, it does find the media item in the external index, e.g.

    __NodeId:2020

    However running a debug instance of Umbraco, and stepping through PublishedMediaCache.GetUmbracoMediaCacheValues, I can see the filter being built looks like this:

    +(+null -__Path:-1,-21,*) +__IndexType:media

    even though the inline code comment says it should build one like this:

    +(+__NodeId:2020 -__Path:-1,-21,*) +__IndexType:media

    The null is being returned from Examine, but I'm not yet sure whether this is an Examine bug, a Lucene.NET bug, or whether Umbraco is using Examine incorrectly.

    I'm trying my own builds of Examine and Lucene.NET to try and find out.

Please Sign in or register to post replies

Write your reply to:

Draft