Boot Error - caused by Examine "write.lock file is denied" in azure
Hi,
A client production site has gone down recently showing the boot failure screen and it was caused by the following error:
Umbraco.Core.Exceptions.BootFailedException: Boot failed: Umbraco cannot run. See Umbraco's log file for more details.-> Umbraco.Core.Exceptions.BootFailedException: Boot failed.-> System.UnauthorizedAccessException: Access to the path 'C:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\write.lock' is denied.
We are currently running Umbraco 8.13.1 and using Examine version 1.1.0
I found an identical error mentioned in https://github.com/Shazwazza/Examine/issues/161 and I believe this issue was fixed in 8.6.4 but unfortunately we have encountered the error on a more recent version. Restarting the frontend app service fixed the issue.
The frontend of the site is Load balanced in Azure using Front Door and the CMS app service is running on a single instance and uses the recommended best practice configuration for load balancing umbraco. See below:
You get these kinds of errors if a site is scaled out in Azure and using the SyncTempEnvFactoryDirectory...
The SyncTempEnvFactoryDirectory is meant to be for a single instance non scaled out instance where it provides a small performance gain over TempEnvFactoryDirectory
But if you have to choose one... Then TempEnvFactoryDirectory will work in both scenarios...
We do have the ability to set differing config values on each web app frontend (Replica) and content (Master) so we won't have to choose between the two.
In that case, based on your suggestion I will update frontend value for LuceneDirectoryFactory to TempEnvDirectoryFactory
and back office web app to SyncTempEnvDirectoryFactory in line with the documentation.
I read that same documentation (a million times, it feel like!) to mean that there are distinct settings that are appropriate for a specific type of instance:
There's a single "Backoffice Administrative Web App" which should be using SyncTempEnvDirectoryFactory
There are also (1-n) instances "Scalable Public Web App" that should use TempEnvDirectoryFactory
I followed this thread as I've just spotted some index corruption on our, as the linked Umbraco docs refers to it, "Backoffice Administrative Web App" using SyncTempEnvDirectoryFactory. The other nodes are configured to be the "Scalable Public Web App" (front-end replicas) and haven't had any issue so far.
Is my understanding off and should i modify all the instances to use TempEnvDirectoryFactory ?
You can run the backoffice instance using TempEnvDirectoryFactory
It depends on your site of course eg size of indexes and how many thousands of editors are concurrently editing it... But you probably won't notice the performance difference...
With Sync you get those errors if the Web app is scaled out (it shouldn't be) for back office or anecdotally there appears to be a condition perhaps when azure changes underlying server for your Web app... And examine thinks its shutting down, locks and doesn't recover without a restart.. Hard to reliably reproduce.. But people report those kind of issues sporadically...
Boot Error - caused by Examine "write.lock file is denied" in azure
Hi,
A client production site has gone down recently showing the boot failure screen and it was caused by the following error:
Umbraco.Core.Exceptions.BootFailedException: Boot failed: Umbraco cannot run. See Umbraco's log file for more details.-> Umbraco.Core.Exceptions.BootFailedException: Boot failed.-> System.UnauthorizedAccessException: Access to the path 'C:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\write.lock' is denied.
We are currently running Umbraco 8.13.1 and using Examine version 1.1.0
I found an identical error mentioned in https://github.com/Shazwazza/Examine/issues/161 and I believe this issue was fixed in 8.6.4 but unfortunately we have encountered the error on a more recent version. Restarting the frontend app service fixed the issue.
The frontend of the site is Load balanced in Azure using Front Door and the CMS app service is running on a single instance and uses the recommended best practice configuration for load balancing umbraco. See below:
Is anybody else still experiencing this issue and is there a way to avoid it in the future?
Thanks, Andy
Hi Andy
The front end scalable instance should I think have the Examine Lucene directory set to TempEnvDirectoryFactory and not SyncTempEnvDirectoryFactory...
https://our.umbraco.com/Documentation/Fundamentals/Setup/Server-Setup/Load-Balancing/azure-web-apps-v8
You get these kinds of errors if a site is scaled out in Azure and using the SyncTempEnvFactoryDirectory...
The SyncTempEnvFactoryDirectory is meant to be for a single instance non scaled out instance where it provides a small performance gain over TempEnvFactoryDirectory
But if you have to choose one... Then TempEnvFactoryDirectory will work in both scenarios...
Regards
Marc
Thanks Marc for the helpful and detailed reply.
We do have the ability to set differing config values on each web app frontend (Replica) and content (Master) so we won't have to choose between the two.
In that case, based on your suggestion I will update frontend value for LuceneDirectoryFactory to
TempEnvDirectoryFactoryand back office web app to
SyncTempEnvDirectoryFactoryin line with the documentation.Curious.
I read that same documentation (a million times, it feel like!) to mean that there are distinct settings that are appropriate for a specific type of instance:
SyncTempEnvDirectoryFactoryTempEnvDirectoryFactoryI followed this thread as I've just spotted some index corruption on our, as the linked Umbraco docs refers to it, "Backoffice Administrative Web App" using
SyncTempEnvDirectoryFactory. The other nodes are configured to be the "Scalable Public Web App" (front-end replicas) and haven't had any issue so far.Is my understanding off and should i modify all the instances to use
TempEnvDirectoryFactory?Umbraco v8.17.0
Hi Damian
You can run the backoffice instance using TempEnvDirectoryFactory
It depends on your site of course eg size of indexes and how many thousands of editors are concurrently editing it... But you probably won't notice the performance difference...
With Sync you get those errors if the Web app is scaled out (it shouldn't be) for back office or anecdotally there appears to be a condition perhaps when azure changes underlying server for your Web app... And examine thinks its shutting down, locks and doesn't recover without a restart.. Hard to reliably reproduce.. But people report those kind of issues sporadically...
Regards
Marc
is working on a reply...