Unhandled exception in AppDomain (terminating) – an Examine index and Lucene errors showcase
Since launch of v8 we (among others) have experienced a lot of random Examine/Lucene index errors and application crashes when running a Umbraco setup with two (or more) root nodes with different domains or subdomains added to the root nodes. After running a short while, the log explodes with errors saying:
Unhandled exception in AppDomain (terminating). (The worst one!)
Cannot index queue items, the index is currently locked
Error indexing queue items
Concurrent merge failed for index: ExternalIndex if this error is persistent then index rebuilding is necessary
An error occurred during the index commit operation, if this error is persistent then index rebuilding is necessary
When opening the Examine Management tab, it fails and says:
Failed to retrieve indexer details
System.ApplicationException: Could not create an index searcher with the supplied lucene directory
There are a lot of threads out there already pointing to some or all of these errors, and there are a lot of answers and people trying to help and add some quickfix suggestions, that eliminates some of the errors – but not all of them – and sometimes only periodically. Most of you help contributors ask for a specific setup and configuration to help reproduce the problem,
so we spent some time here building a showcase that addresses the problem live:
This is a live demo site that contains one Umbraco installation with two root nodes – each with “their own respectively” domain attached in culture and hostnames:
In our DNS both domains have A-records pointing to the same server IP address
The site saves a property via ContentService everytime someone visits the site on either of the two domains. To demonstrate further and to provoke the different Examine errors, a cronjob is set up on the server to access both domains once an hour, triggering the content saving.
Please, feel free to login to the Umbraco demo to see the culture and hostnames set up, the Examine Management tab – and especially the log, that is incrementing on red errors.
And please feel free to download the whole Umbraco site .vs project here including a backup (.bak ) of the
SQL server to check out the setup in details.
In the demo/project we have included some of the quickfixes from other threads, which are:
This composer here, that is supposed to unlock the Examine directories if they are locked:
using Examine;
using Examine.LuceneEngine.Directories;
using Examine.LuceneEngine.Providers;
using Lucene.Net.Index;
using Umbraco.Core.Composing;
using Umbraco.Examine;
using Umbraco.Web.Search;
namespace ExamineErrors.Components
{
public class CustomizedExamineComposer : ComponentComposer<CustomizedExamineComponent> { }
public class CustomizedExamineComponent : IComponent
{
private readonly IExamineManager _examineManager;
private readonly IUmbracoIndexesCreator _indexCreator;
public CustomizedExamineComponent(IExamineManager examineManager, IUmbracoIndexesCreator indexCreator)
{
_examineManager = examineManager;
_indexCreator = indexCreator;
}
public void Initialize()
{
DirectoryFactory.DefaultLockFactory = d =>
{
var simpleFsLockFactory = new NoPrefixSimpleFsLockFactory(d);
return simpleFsLockFactory;
};
foreach (var index in _indexCreator.Create())
{
if (index is LuceneIndex)
{
var luceneIndex = index as LuceneIndex;
var dir = luceneIndex.GetLuceneDirectory();
if (IndexWriter.IsLocked(dir))
{
IndexWriter.Unlock(dir);
}
}
}
}
public void Terminate()
{
}
}
}
And the IgnoreLocalDB-composer which seems to mute some errors in the Umbraco backoffice:
using Umbraco.Core.Composing;
using Umbraco.Web.PublishedCache.NuCache;
namespace ExamineErrors.Composers
{
public class IgnoreLocalDbComposer : IUserComposer
{
public void Compose(Composition composition)
{
composition.Register(factory => new
PublishedSnapshotServiceOptions
{
IgnoreLocalDb = true
});
}
}
}
It is really important for us to find a solution to these problems, since it is gradually critical for our clients
and more of them are asking disliking questions about Umbraco because they are frustrated.
The setup described above has worked for us in Umbraco v7 with the same setup and configuration since v7 was born. We would be really thankful if we could continue this in v8 without having to split up each site or variation in to their own Umbraco installation. Therefore, we have brushed this out in details, to help narrowing down the problems – and we are ready to answer any questions that would be helpful to find a solution.
foreach (var index in _indexCreator.Create()) {
if (index is LuceneIndex)
{
var luceneIndex = index as LuceneIndex;
var dir = luceneIndex.GetLuceneDirectory();
if (IndexWriter.IsLocked(dir))
{
IndexWriter.Unlock(dir);
}
}
_examineManager.AddIndex(index);
}
I noticed that the unlocked index is not been added back,
this line is missing ..
_examineManager.AddIndex(index);
Now i didnt test it .. it was just a quick observation
Maybe also add some logging to see if it is unlocking. The ironic thing the examine index is locked at the moment . ;) hence the error when you try to look at indexes in backoffice .
If get more time , i will see if i can load your solution
edit: Exception of type 'Lucene.Net.Index.MergePolicy+MergeException' was thrown. , not sure about this , only suggestion here is to delete the search index in app-data/ , maybe its become corrupt because of lack of the rebuild ..so not sure if that is fall out from the other issues .
Also we rebuild the external index in composer . I noticed the error relates to that too external index . so that may stop it getting corrupt . but i need to test that on your solution.
@Thomsen : after a bit of editing i believe it now works :
Forcing index "InternalIndex" to be unlocked since it was left in a locked state
Forcing index "ExternalIndex" to be unlocked since it was left in a locked state
Forcing index "MembersIndex" to be unlocked since it was left in a locked state
I couldnt connect to db in the zip do you have its connection string ? to test further .. i connect to my testdb and but it looks like it works .
Your first post:
I am not re-adding the index because it resulted in an error. I think we in a previous thread concluded that you only nee to add the indexer again, if you are using a custom indexer:
Hi Thomsen ,
Ok , i use azure db so that is what thrown me dont use sql compact or express , but got connected to local db , and your backup . so all connected and working quite well ( that i can see in logs ) .
I see it is unlocking the indexes and i dont see any serious errors in logs .
So had to make some changes : Main ones are in the components folder:
First the component - renamed the class :
And removed any code not required .
using Examine;
using Examine.LuceneEngine.Directories;
using Examine.LuceneEngine.Providers;
using Lucene.Net.Index;
using Umbraco.Core.Composing;
using Umbraco.Examine;
using Umbraco.Web.Search;
namespace ExamineErrors.Components
{
public class ExamineComponent : IComponent
{
private readonly IUmbracoIndexesCreator _indexCreator;
public ExamineComponent(IUmbracoIndexesCreator indexCreator)
{
_indexCreator = indexCreator;
}
public void Initialize()
{
DirectoryFactory.DefaultLockFactory = d =>
{
var simpleFsLockFactory = new NoPrefixSimpleFsLockFactory(d);
return simpleFsLockFactory;
};
foreach (var index in _indexCreator.Create())
{
if (index is LuceneIndex)
{
var luceneIndex = index as LuceneIndex;
var dir = luceneIndex.GetLuceneDirectory();
if (IndexWriter.IsLocked(dir))
{
IndexWriter.Unlock(dir);
}
}
}
}
public void Terminate()
{
}
}
}
In the Examine Composer
So i can add some logging too :
using Umbraco.Core;
using Umbraco.Core.Composing;
using Umbraco.Web.PublishedCache.NuCache;
using Umbraco.Web.Search;
namespace Examineerrors.Components
{
[RuntimeLevel(MinLevel = RuntimeLevel.Run)]
public class ExamineComposer : IUserComposer
{
public void Compose(Composition composition)
{
composition.Register(factory => new
PublishedSnapshotServiceOptions
{
IgnoreLocalDb = true
});
composition.Components().Append<ExamineComponent>();
}
}
Not sure if you need the sqlMaindomlock ( as i think it more azure db) .
also in Prod you might need to change this :
Examine.LuceneEngine.Directories.TempEnvDirectoryFactory
change to
Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory
Hope that works for you .
Andy
ps : if the unlock was required you will see this in the logs :
Forcing index "ExternalIndex" to be unlocked since it was left in a locked state
Thank you very much. I have implemented all of your suggestions at the live demo site and on another one of mine causing the same problems. I have uploaded the changes at about 1.00 PM and the Examine Management tab is "still alive" ...
Let's give it a few hours and see how it works out.
Just checked the log this morning after the demo site has been running with the corrections over night, and it shows there is a lot of these errors since yesterday:
Also the Examine Management tab has been corrupted with errors:
Thomsen , have you tried to delete the indexes in appdata/ temp/examineindexes/after the other code change, just to see if they were already corrupt to allow it to create new indexes .
Also once the setting / indexers is back after a restart , try the rebuild in the settings
@Wouter thanks, I was not aware that the unlocking component was only run once. It makes sense then that the index gets locked again after a while. As you say, we need an overall solution to this instead of various quickfixes, so therefore we have reported this to the Umbraco issue tracker.
Hi again and thanks for your help, Andy. We have tried deleting all the temp and cache folders and files, stopping and recycling the IIS and rebuilding the indexes over again. It works for a while, and then the error rain and AppDomain terminating starts over again.
For now our only solution is to split our v8 sites up into separate Umbraco installations until this get solved, since it is too critical.
Unhandled exception in AppDomain (terminating) – an Examine index and Lucene errors showcase
Since launch of v8 we (among others) have experienced a lot of random Examine/Lucene index errors and application crashes when running a Umbraco setup with two (or more) root nodes with different domains or subdomains added to the root nodes. After running a short while, the log explodes with errors saying:
When opening the Examine Management tab, it fails and says:
There are a lot of threads out there already pointing to some or all of these errors, and there are a lot of answers and people trying to help and add some quickfix suggestions, that eliminates some of the errors – but not all of them – and sometimes only periodically. Most of you help contributors ask for a specific setup and configuration to help reproduce the problem, so we spent some time here building a showcase that addresses the problem live:
This is a live demo site that contains one Umbraco installation with two root nodes – each with “their own respectively” domain attached in culture and hostnames:
Please, check it out:
https://www.play-box.dk (root node 1)
https://www.examinetest.dk (root node 2)
In our DNS both domains have A-records pointing to the same server IP address
The site saves a property via ContentService everytime someone visits the site on either of the two domains. To demonstrate further and to provoke the different Examine errors, a cronjob is set up on the server to access both domains once an hour, triggering the content saving.
Please, feel free to login to the Umbraco demo to see the culture and hostnames set up, the Examine Management tab – and especially the log, that is incrementing on red errors.
Login:
https://www.play-box.dk/umbraco
Email: [email protected]
Password: examinetester
And please feel free to download the whole Umbraco site .vs project here including a backup (.bak ) of the SQL server to check out the setup in details.
https://www.dropbox.com/s/c1e3vkw2u95jax4/Examinetest.zip?dl=1
About this live setup:
.NET Framework 4.7.2 Umbraco v.8.6.4 running on a MSSQL Server DB Examine v.1.0.4
Hosting environment:
Simply.com (previously UnoEuro) on a standard .NET hosting solution (see specs here: https://www.simply.com/dk/products/#specs)
HTTPS/Redirects
There is a Let’s Encrypt certificate installed on both domains – and some rewrite/redirect rules added in the web.config to accommodate this:
NOTES:
In the demo/project we have included some of the quickfixes from other threads, which are:
This composer here, that is supposed to unlock the Examine directories if they are locked:
And the IgnoreLocalDB-composer which seems to mute some errors in the Umbraco backoffice:
It is really important for us to find a solution to these problems, since it is gradually critical for our clients and more of them are asking disliking questions about Umbraco because they are frustrated.
The setup described above has worked for us in Umbraco v7 with the same setup and configuration since v7 was born. We would be really thankful if we could continue this in v8 without having to split up each site or variation in to their own Umbraco installation. Therefore, we have brushed this out in details, to help narrowing down the problems – and we are ready to answer any questions that would be helpful to find a solution.
Thank you!
@thornsen , i had a quick look ,
I noticed that the unlocked index is not been added back, this line is missing .. _examineManager.AddIndex(index); Now i didnt test it .. it was just a quick observation Maybe also add some logging to see if it is unlocking. The ironic thing the examine index is locked at the moment . ;) hence the error when you try to look at indexes in backoffice . If get more time , i will see if i can load your solution
edit: Exception of type 'Lucene.Net.Index.MergePolicy+MergeException' was thrown. , not sure about this , only suggestion here is to delete the search index in app-data/ , maybe its become corrupt because of lack of the rebuild ..so not sure if that is fall out from the other issues . Also we rebuild the external index in composer . I noticed the error relates to that too external index . so that may stop it getting corrupt . but i need to test that on your solution.
Andy
@Thomsen : after a bit of editing i believe it now works :
I couldnt connect to db in the zip do you have its connection string ? to test further .. i connect to my testdb and but it looks like it works .
Andy
Hi Andy
Thank you for digging into it (once again). We would be happy to see what you have done so far!
You should be able to import the DB from .bak file into your local SQL server and then use the connection string here, to accecs it:
Your first post: I am not re-adding the index because it resulted in an error. I think we in a previous thread concluded that you only nee to add the indexer again, if you are using a custom indexer:
Hi Thomsen , Ok , i use azure db so that is what thrown me dont use sql compact or express , but got connected to local db , and your backup . so all connected and working quite well ( that i can see in logs ) . I see it is unlocking the indexes and i dont see any serious errors in logs . So had to make some changes : Main ones are in the components folder:
First the component - renamed the class : And removed any code not required .
}
In the Examine Composer So i can add some logging too :
}
In webconfig : in app settings :
Not sure if you need the sqlMaindomlock ( as i think it more azure db) . also in Prod you might need to change this : Examine.LuceneEngine.Directories.TempEnvDirectoryFactory change to Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory
Hope that works for you . Andy
ps : if the unlock was required you will see this in the logs :
Hi Andy
Thank you very much. I have implemented all of your suggestions at the live demo site and on another one of mine causing the same problems. I have uploaded the changes at about 1.00 PM and the Examine Management tab is "still alive" ...
Let's give it a few hours and see how it works out.
Best regards.
Hi again
Just checked the log this morning after the demo site has been running with the corrections over night, and it shows there is a lot of these errors since yesterday:
Also the Examine Management tab has been corrupted with errors:
Thomsen , have you tried to delete the indexes in appdata/ temp/examineindexes/after the other code change, just to see if they were already corrupt to allow it to create new indexes .
Also once the setting / indexers is back after a restart , try the rebuild in the settings
Some more on this here : https://our.umbraco.com/forum/ourumb-dev-forum/bugs/16066-Missing-LuceneNet-assembly I assume your not low on space on DB ( just a thought)
I am experiencing the exact same problems. Thus I have studied your solution and it looked promising.
However, the code in Initialize method in the custom ExamineComponent is only run whenever Umbraco starts...
So if any index becomes locked again after the Umbraco startup, this component will not unlock it until the entire website (or app pool) is restarted.
Thus I think another solution is still necessay to really solve this issue. If I find something, I'll post it here.
@Wouter thanks, I was not aware that the unlocking component was only run once. It makes sense then that the index gets locked again after a while. As you say, we need an overall solution to this instead of various quickfixes, so therefore we have reported this to the Umbraco issue tracker.
Hi again and thanks for your help, Andy. We have tried deleting all the temp and cache folders and files, stopping and recycling the IIS and rebuilding the indexes over again. It works for a while, and then the error rain and AppDomain terminating starts over again.
For now our only solution is to split our v8 sites up into separate Umbraco installations until this get solved, since it is too critical.
We have raised an issue here, referencing this thread: https://github.com/umbraco/Umbraco-CMS/issues/8740
Thanks
is working on a reply...