Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Thomsen 112 posts 335 karma points
    Aug 21, 2020 @ 12:43
    Thomsen
    0

    Unhandled exception in AppDomain (terminating) – an Examine index and Lucene errors showcase

    Since launch of v8 we (among others) have experienced a lot of random Examine/Lucene index errors and application crashes when running a Umbraco setup with two (or more) root nodes with different domains or subdomains added to the root nodes. After running a short while, the log explodes with errors saying:

    • Unhandled exception in AppDomain (terminating). (The worst one!)
    • Cannot index queue items, the index is currently locked
    • Error indexing queue items
    • Concurrent merge failed for index: ExternalIndex if this error is persistent then index rebuilding is necessary
    • An error occurred during the index commit operation, if this error is persistent then index rebuilding is necessary

    When opening the Examine Management tab, it fails and says:

    • Failed to retrieve indexer details
    • System.ApplicationException: Could not create an index searcher with the supplied lucene directory

    There are a lot of threads out there already pointing to some or all of these errors, and there are a lot of answers and people trying to help and add some quickfix suggestions, that eliminates some of the errors – but not all of them – and sometimes only periodically. Most of you help contributors ask for a specific setup and configuration to help reproduce the problem, so we spent some time here building a showcase that addresses the problem live:

    This is a live demo site that contains one Umbraco installation with two root nodes – each with “their own respectively” domain attached in culture and hostnames:

    Please, check it out:

    https://www.play-box.dk (root node 1)

    https://www.examinetest.dk (root node 2)

    In our DNS both domains have A-records pointing to the same server IP address

    The site saves a property via ContentService everytime someone visits the site on either of the two domains. To demonstrate further and to provoke the different Examine errors, a cronjob is set up on the server to access both domains once an hour, triggering the content saving.

    Please, feel free to login to the Umbraco demo to see the culture and hostnames set up, the Examine Management tab – and especially the log, that is incrementing on red errors.

    Login:

    https://www.play-box.dk/umbraco

    Email: [email protected]
    Password: examinetester

    And please feel free to download the whole Umbraco site .vs project here including a backup (.bak ) of the SQL server to check out the setup in details.

    https://www.dropbox.com/s/c1e3vkw2u95jax4/Examinetest.zip?dl=1

    About this live setup:

    .NET Framework 4.7.2 Umbraco v.8.6.4 running on a MSSQL Server DB Examine v.1.0.4

    Hosting environment:

    Simply.com (previously UnoEuro) on a standard .NET hosting solution (see specs here: https://www.simply.com/dk/products/#specs)

    HTTPS/Redirects

    There is a Let’s Encrypt certificate installed on both domains – and some rewrite/redirect rules added in the web.config to accommodate this:

    <rewrite>
      <rules>
        <clear />
        <rule name="AcmeChallenge" stopProcessing="true">
          <match url=".well-known/acme-challenge/.*" />
          <conditions />
          <serverVariables />
          <action type="Rewrite" url="{R:0}.txt" appendQueryString="false" />
        </rule>
        <rule name="Add WWW" stopProcessing="true">
          <match url="^(.*)$" />
          <conditions>
            <add input="{HTTP_HOST}" pattern="^(?!www\.)(.*)$" />
          </conditions>
          <action type="Redirect" url="http://www.{C:0}{PATH_INFO}" redirectType="Permanent" />
        </rule>
        <rule name="Redirect to https" stopProcessing="true">
          <match url=".*" />
          <conditions logicalGrouping="MatchAll">
            <add input="{HTTPS}" pattern="off" ignoreCase="true" />
            <add input="{REQUEST_URI}" negate="true" pattern="^/.well-known" ignoreCase="true" />
          </conditions>
          <action type="Redirect" url="https://{HTTP_HOST}{REQUEST_URI}" redirectType="Permanent" appendQueryString="false" />
        </rule>
      </rules>
    </rewrite>
    

    NOTES:

    In the demo/project we have included some of the quickfixes from other threads, which are:

    This composer here, that is supposed to unlock the Examine directories if they are locked:

    using Examine;
    using Examine.LuceneEngine.Directories;
    using Examine.LuceneEngine.Providers;
    using Lucene.Net.Index;
    using Umbraco.Core.Composing;
    using Umbraco.Examine;
    using Umbraco.Web.Search;
    
    namespace ExamineErrors.Components
    {
    public class CustomizedExamineComposer : ComponentComposer<CustomizedExamineComponent> { }
    
    public class CustomizedExamineComponent : IComponent
    {
        private readonly IExamineManager _examineManager;
        private readonly IUmbracoIndexesCreator _indexCreator;
    
        public CustomizedExamineComponent(IExamineManager examineManager, IUmbracoIndexesCreator indexCreator)
        {
            _examineManager = examineManager;
            _indexCreator = indexCreator;
        }
    
        public void Initialize()
        {
            DirectoryFactory.DefaultLockFactory = d =>
            {
                var simpleFsLockFactory = new NoPrefixSimpleFsLockFactory(d);
                return simpleFsLockFactory;
            };
    
            foreach (var index in _indexCreator.Create())
            {
                if (index is LuceneIndex)
                {
                    var luceneIndex = index as LuceneIndex;
                    var dir = luceneIndex.GetLuceneDirectory();
                    if (IndexWriter.IsLocked(dir))
                    {
                        IndexWriter.Unlock(dir);
                    }
                }
            }
        }
        public void Terminate()
        {
        }
    }
    }
    

    And the IgnoreLocalDB-composer which seems to mute some errors in the Umbraco backoffice:

    using Umbraco.Core.Composing;
    using Umbraco.Web.PublishedCache.NuCache;
    
        namespace ExamineErrors.Composers
    {
        public class IgnoreLocalDbComposer : IUserComposer
        {
            public void Compose(Composition composition)
            {
                composition.Register(factory => new 
    PublishedSnapshotServiceOptions
                {
                    IgnoreLocalDb = true
                });
            }
        }
    }
    

    It is really important for us to find a solution to these problems, since it is gradually critical for our clients and more of them are asking disliking questions about Umbraco because they are frustrated.

    The setup described above has worked for us in Umbraco v7 with the same setup and configuration since v7 was born. We would be really thankful if we could continue this in v8 without having to split up each site or variation in to their own Umbraco installation. Therefore, we have brushed this out in details, to help narrowing down the problems – and we are ready to answer any questions that would be helpful to find a solution.

    Thank you!

  • Andy Finn 51 posts 183 karma points
    Aug 21, 2020 @ 19:23
    Andy Finn
    0

    @thornsen , i had a quick look ,

                foreach (var index in _indexCreator.Create()) { 
                if (index is LuceneIndex)
                {
                    var luceneIndex = index as LuceneIndex;
                    var dir = luceneIndex.GetLuceneDirectory();
                    if (IndexWriter.IsLocked(dir))
                    {
                        IndexWriter.Unlock(dir);
                    }
                }
            _examineManager.AddIndex(index);
            }
    

    I noticed that the unlocked index is not been added back, this line is missing .. _examineManager.AddIndex(index); Now i didnt test it .. it was just a quick observation Maybe also add some logging to see if it is unlocking. The ironic thing the examine index is locked at the moment . ;) hence the error when you try to look at indexes in backoffice . If get more time , i will see if i can load your solution

    edit: Exception of type 'Lucene.Net.Index.MergePolicy+MergeException' was thrown. , not sure about this , only suggestion here is to delete the search index in app-data/ , maybe its become corrupt because of lack of the rebuild ..so not sure if that is fall out from the other issues . Also we rebuild the external index in composer . I noticed the error relates to that too external index . so that may stop it getting corrupt . but i need to test that on your solution.

    Andy

  • Andy Finn 51 posts 183 karma points
    Aug 22, 2020 @ 14:46
    Andy Finn
    0

    @Thomsen : after a bit of editing i believe it now works :

    Forcing index "InternalIndex" to be unlocked since it was left in a locked state
    Forcing index "ExternalIndex" to be unlocked since it was left in a locked state
    Forcing index "MembersIndex" to be unlocked since it was left in a locked state
    

    I couldnt connect to db in the zip do you have its connection string ? to test further .. i connect to my testdb and but it looks like it works .

    Andy

  • Thomsen 112 posts 335 karma points
    Aug 24, 2020 @ 07:35
    Thomsen
    0

    Hi Andy

    Thank you for digging into it (once again). We would be happy to see what you have done so far!

    You should be able to import the DB from .bak file into your local SQL server and then use the connection string here, to accecs it:

    <connectionStrings>
      <remove name="umbracoDbDSN" />
      <add name="umbracoDbDSN" connectionString="Data 
      Source=localhost;Initial Catalog=ExamineTestErrorsDB;Integrated 
      Security=True" providerName="System.Data.SqlClient" />
     </connectionStrings>
    

    Your first post: I am not re-adding the index because it resulted in an error. I think we in a previous thread concluded that you only nee to add the indexer again, if you are using a custom indexer:

    _examineManager.AddIndex(index); 
    
  • Andy Finn 51 posts 183 karma points
    Aug 24, 2020 @ 10:13
    Andy Finn
    0

    Hi Thomsen , Ok , i use azure db so that is what thrown me dont use sql compact or express , but got connected to local db , and your backup . so all connected and working quite well ( that i can see in logs ) . I see it is unlocking the indexes and i dont see any serious errors in logs . So had to make some changes : Main ones are in the components folder: enter image description here

    First the component - renamed the class : And removed any code not required .

    using Examine;
    using Examine.LuceneEngine.Directories;
    using Examine.LuceneEngine.Providers;
    using Lucene.Net.Index;
    using Umbraco.Core.Composing;
    using Umbraco.Examine;
    using Umbraco.Web.Search;
    
    namespace ExamineErrors.Components
    {
        public class ExamineComponent : IComponent
        {
             private readonly IUmbracoIndexesCreator _indexCreator;
    
             public ExamineComponent(IUmbracoIndexesCreator indexCreator)
             {
                  _indexCreator = indexCreator;
             }
    
            public void Initialize()
            {
                DirectoryFactory.DefaultLockFactory = d =>
                {
                    var simpleFsLockFactory = new NoPrefixSimpleFsLockFactory(d);
                    return simpleFsLockFactory;
                };
                foreach (var index in _indexCreator.Create())
                {
                     if (index is LuceneIndex)
                     {
                         var luceneIndex = index as LuceneIndex;
                         var dir = luceneIndex.GetLuceneDirectory();
                         if (IndexWriter.IsLocked(dir))
                         {
                             IndexWriter.Unlock(dir);
                         }
                     }
                }
            }
        public void Terminate()
        {
        }
    }
    

    }

    In the Examine Composer So i can add some logging too :

    using Umbraco.Core;
    using Umbraco.Core.Composing;
    using Umbraco.Web.PublishedCache.NuCache;
    using Umbraco.Web.Search;
    
    namespace Examineerrors.Components
    {
    [RuntimeLevel(MinLevel = RuntimeLevel.Run)]
    public class ExamineComposer : IUserComposer
    {
        public void Compose(Composition composition)
        {
            composition.Register(factory => new 
            PublishedSnapshotServiceOptions
            {
                IgnoreLocalDb = true
            });
            composition.Components().Append<ExamineComponent>();
        }
    }
    

    }

    In webconfig : in app settings :

    <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
    <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.TempEnvDirectoryFactory, Examine" />
    <add key="Umbraco.Core.MainDom.Lock" value="SqlMainDomLock" />
    

    Not sure if you need the sqlMaindomlock ( as i think it more azure db) . also in Prod you might need to change this : Examine.LuceneEngine.Directories.TempEnvDirectoryFactory change to Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory

    Hope that works for you . Andy

    ps : if the unlock was required you will see this in the logs :

    Forcing index "ExternalIndex" to be unlocked since it was left in a locked state
    
  • Thomsen 112 posts 335 karma points
    Aug 24, 2020 @ 11:26
    Thomsen
    0

    Hi Andy

    Thank you very much. I have implemented all of your suggestions at the live demo site and on another one of mine causing the same problems. I have uploaded the changes at about 1.00 PM and the Examine Management tab is "still alive" ...

    Let's give it a few hours and see how it works out.

    Best regards.

  • Thomsen 112 posts 335 karma points
    Aug 25, 2020 @ 06:36
    Thomsen
    0

    Hi again

    Just checked the log this morning after the demo site has been running with the corrections over night, and it shows there is a lot of these errors since yesterday:

    enter image description here

    Also the Examine Management tab has been corrupted with errors:

    enter image description here

  • Andy Finn 51 posts 183 karma points
    Aug 25, 2020 @ 07:41
    Andy Finn
    0

    Thomsen , have you tried to delete the indexes in appdata/ temp/examineindexes/after the other code change, just to see if they were already corrupt to allow it to create new indexes .
    Also once the setting / indexers is back after a restart , try the rebuild in the settings enter image description here

    Some more on this here : https://our.umbraco.com/forum/ourumb-dev-forum/bugs/16066-Missing-LuceneNet-assembly I assume your not low on space on DB ( just a thought)

  • Wouter van de Weerd 6 posts 117 karma points notactivated
    Aug 28, 2020 @ 13:04
    Wouter van de Weerd
    101

    I am experiencing the exact same problems. Thus I have studied your solution and it looked promising.

    However, the code in Initialize method in the custom ExamineComponent is only run whenever Umbraco starts...

    So if any index becomes locked again after the Umbraco startup, this component will not unlock it until the entire website (or app pool) is restarted.

    Thus I think another solution is still necessay to really solve this issue. If I find something, I'll post it here.

  • Thomsen 112 posts 335 karma points
    Aug 28, 2020 @ 16:46
    Thomsen
    0

    @Wouter thanks, I was not aware that the unlocking component was only run once. It makes sense then that the index gets locked again after a while. As you say, we need an overall solution to this instead of various quickfixes, so therefore we have reported this to the Umbraco issue tracker.

  • Thomsen 112 posts 335 karma points
    Aug 28, 2020 @ 13:46
    Thomsen
    0

    Hi again and thanks for your help, Andy. We have tried deleting all the temp and cache folders and files, stopping and recycling the IIS and rebuilding the indexes over again. It works for a while, and then the error rain and AppDomain terminating starts over again.

    For now our only solution is to split our v8 sites up into separate Umbraco installations until this get solved, since it is too critical.

    We have raised an issue here, referencing this thread: https://github.com/umbraco/Umbraco-CMS/issues/8740

    Thanks

Please Sign in or register to post replies

Write your reply to:

Draft