We have an v7.5.14 site running on Azure (Web Apps) and we've had a couple of downtime issues over the last few days where all requests would fail with the following error:
Lock obtain timed out: NativeFSLock@D:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\Index\write.lock: System.IO.IOException: The process cannot access the file 'D:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\Index\write.lock' because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
at Lucene.Net.Store.NativeFSLock.Obtain()
at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
Lock obtain timed out: NativeFSLock@D:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\Index\write.lock: System.IO.IOException: The process cannot access the file 'D:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\Index\write.lock' because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
at Lucene.Net.Store.NativeFSLock.Obtain()
at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout)
at Examine.LuceneEngine.Directories.MultiIndexLock.Obtain(Int64 lockWaitTimeout)
at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean create, Boolean closeDir, IndexDeletionPolicy deletionPolicy, Boolean autoCommit, Int32 maxFieldLength, IndexingChain indexingChain, IndexCommit commit)
at Lucene.Net.Index.IndexWriter..ctor(Directory d, Analyzer a, Boolean create, MaxFieldLength mfl)
at Examine.LuceneEngine.Providers.LuceneIndexer.WriterFactory(Directory d)
at Examine.LuceneEngine.WriterTracker.<>c__DisplayClass7_0.
<GetWriter>b__0(String s)
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at Examine.LuceneEngine.WriterTracker.GetWriter(Directory dir, Func`2 factory)
at Examine.LuceneEngine.Providers.LuceneIndexer.CreateIndexWriter()
at UmbracoExamine.BaseUmbracoIndexer.CreateIndexWriter()
at Examine.LuceneEngine.Providers.LuceneIndexer.EnsureIndex(Boolean forceOverwrite)
at Examine.LuceneEngine.Providers.LuceneIndexer.RebuildIndex()
at UmbracoExamine.UmbracoContentIndexer.RebuildIndex()
at Umbraco.Web.WebBootManager.
<InitializeResolvers>b__f()
at Umbraco.Core.Sync.DatabaseServerMessenger.Initialize()
at Umbraco.Core.Sync.DatabaseServerMessenger.Boot()
at Umbraco.Web.BatchedDatabaseServerMessenger.Startup()
at Umbraco.Web.BatchedDatabaseServerMessengerStartup.ApplicationStarted(UmbracoApplicationBase umbracoApplication, ApplicationContext applicationContext)
at Umbraco.Core.ApplicationEventHandler.OnApplicationStarted(UmbracoApplicationBase umbracoApplication, ApplicationContext applicationContext)
at Umbraco.Core.CoreBootManager.
<Complete>b__a(IApplicationEventHandler x)
at Umbraco.Core.EnumerableExtensions.ForEach[TItem](IEnumerable`1 items, Action`1 action)
at Umbraco.Core.CoreBootManager.Complete(Action`1 afterComplete)
at Umbraco.Web.WebBootManager.Complete(Action`1 afterComplete)
at Umbraco.Core.UmbracoApplicationBase.StartApplication(Object sender, EventArgs e)
at Umbraco.Core.UmbracoApplicationBase.Application_Start(Object sender, EventArgs e)
The process cannot access the file 'D:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\Index\write.lock' because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
at Lucene.Net.Store.NativeFSLock.Obtain()
<?xml version="1.0"?>
<ExamineLuceneIndexSets>
<!-- The internal index set used by Umbraco back-office - DO NOT REMOVE -->
<IndexSet SetName="InternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/Internal/" />
<!-- The internal index set used by Umbraco back-office for indexing members - DO NOT REMOVE -->
<IndexSet SetName="InternalMemberIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/InternalMember/">
<IndexAttributeFields>
<add Name="id" />
<add Name="nodeName"/>
<add Name="updateDate" />
<add Name="writerName" />
<add Name="loginName" />
<add Name="email" />
<add Name="nodeTypeAlias" />
</IndexAttributeFields>
</IndexSet>
<!-- Default Indexset for external searches, this indexes all fields on all types of nodes-->
<IndexSet SetName="ExternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/External/" />
</ExamineLuceneIndexSets>
I just noticed that there is no {machinename} in the IndexPath attributes in ExamineIndex.config.
The documentation mentions that for Examine pre-v0.1.80 this needs to be removed, but does that mean for v0.1.80+ it does need to be there?
(the site is running Examine v0.1.82)
SyncTempEnvDirectoryFactory is designed so that indexes don't have to be rebuilt when your app is auto migrated from one server to another as they will be synced back from the file system to the temp folder and then accessed from there. However it seems that the sync process is encountering a file system lock issue. It might be worth creating an issue on the Examine repo as your settings are correct. https://github.com/Shazwazza/Examine/issues
I have seen horrible lock issues writing to the file system on Azure Apps that seem to come from the low latency and possible migration of the storage location, this is what lead to the creation of the umbracoLocalTempStorage setting.
You could try using the TempEnvDirectoryFactory instead of SyncTempEnvDirectoryFactory, this factory doesn't do any syncing and only uses the local temp file system for index storage so doesn't suffer lock issues. However when you app auto migrated from one server to another the indexes will have to be rebuilt from scratch.
Yes, I've been looking through the event log for the app and I can see that the service instance ID did change around the time the errors started occurring, so that matches your theory it happened during the index file sync.
But I guess the advice in this case would be to get the site upgraded to v7.7.3+ to make use of the new umbracoLocalTempStorage setting that should avoid locking issues?
In this case your issue is with Examine so changing to TempEnvDirectoryFactory would be my recommendation (you will need to update Examine to 0.1.83+). That said I would recommend using umbracoLocalTempStorage:EnvironmentTemp which will require your updating Umbraco to (7.7.3+) too to avoid other locking issues with auto server migration in Azure Apps.
Hi, I have realtive issue. I have Umbraco 7.7.9 on Azure and large indexes. I have tried SyncTempEnvDirectoryFactory setting but I got the locking issue. I switched to TempEnvDirectoryFactory and I have issues only when Azure switches Web Workers. The website is broken by the time the indexes get rebuild. I understand the difference between the two settings and I think that TempEnvDirectoryFactory is better option in my case but I still need a solution when the Web Workers switch.
We've started getting issues like this. Was our first site using Umbraco and it's been a buggy experience with little things like this brining down the whole site. Sadly, this will also be our last Umbraco site.
I have a similar problem but with Umbraco 8. I've registered a custom index like this : https://our.umbraco.com/forum/umbraco-8/95915-umbraco-8-examine-indexes. But now, when i restart the app, I get a lock exception. I've manage to work around it and force clear the lock but I don't think it's the right way. Also, Umbraco 8 doesn't contains any examine config file so i'm not able to use SyncTempEnvDirectoryFactory. If anyone knows how this should work, I'll gladly take an helping hand on this one.
For anyone still having issues with the Examine Lock ,. we changed the ExamineComponent.cs file to check for lock and unlock it .
DirectoryFactory.DefaultLockFactory = d =>
{
var simpleFsLockFactory = new NoPrefixSimpleFsLockFactory(d);
return simpleFsLockFactory;
};
foreach (var index in _indexCreator.Create()) {
if (index is LuceneIndex)
{
var luceneIndex = index as LuceneIndex;
var dir = luceneIndex.GetLuceneDirectory();
if (IndexWriter.IsLocked(dir))
{
IndexWriter.Unlock(dir);
}
}
_examineManager.AddIndex(index);
}
Examine lock error brings down Azure site
We have an v7.5.14 site running on Azure (Web Apps) and we've had a couple of downtime issues over the last few days where all requests would fail with the following error:
We've followed all the recommended Azure settings, so I'm not sure what could cause this lock issue.
Has anyone seen this before?
What settings are you using for ExamineSettings.config as you are almost certainly missing something.
https://our.umbraco.org/documentation/Getting-Started/Setup/Server-Setup/azure-web-apps#examine-v0-1-80
It seems set up right to me, but I might be missing something.
ExamineSettings.config:
ExamineIndex.config:
Are you load balanced or single instance?
Also what version of Examine do you have?
A single instance.
I just noticed that there is no
{machinename}in the IndexPath attributes in ExamineIndex.config.The documentation mentions that for Examine pre-v0.1.80 this needs to be removed, but does that mean for v0.1.80+ it does need to be there? (the site is running Examine v0.1.82)
No, that shouldn't be needed, do you have large indexes?
Nothing in the InternalMemberIndexer.
InternalIndexer: 25062 documents (96 fields)
ExternalIndexer: 24752 documents (96 fields)
SyncTempEnvDirectoryFactoryis designed so that indexes don't have to be rebuilt when your app is auto migrated from one server to another as they will be synced back from the file system to the temp folder and then accessed from there. However it seems that the sync process is encountering a file system lock issue. It might be worth creating an issue on the Examine repo as your settings are correct. https://github.com/Shazwazza/Examine/issuesI have seen horrible lock issues writing to the file system on Azure Apps that seem to come from the low latency and possible migration of the storage location, this is what lead to the creation of the
umbracoLocalTempStoragesetting.You could try using the
TempEnvDirectoryFactoryinstead ofSyncTempEnvDirectoryFactory, this factory doesn't do any syncing and only uses the local temp file system for index storage so doesn't suffer lock issues. However when you app auto migrated from one server to another the indexes will have to be rebuilt from scratch.Yes, I've been looking through the event log for the app and I can see that the service instance ID did change around the time the errors started occurring, so that matches your theory it happened during the index file sync.
But I guess the advice in this case would be to get the site upgraded to v7.7.3+ to make use of the new
umbracoLocalTempStoragesetting that should avoid locking issues?In this case your issue is with Examine so changing to
TempEnvDirectoryFactorywould be my recommendation (you will need to update Examine to 0.1.83+). That said I would recommend usingumbracoLocalTempStorage:EnvironmentTempwhich will require your updating Umbraco to (7.7.3+) too to avoid other locking issues with auto server migration in Azure Apps.There is more info on why we added umbracoLocalTempStorage here
That's great. Thanks for your suggestions, and explaining why the issue occurred!
Hopefully one day we will be able to store Examine indexes in blob storage and then sync to local temp so we can have the best of all worlds. There has been some work on this but it's not production ready. https://github.com/Shazwazza/Examine/tree/master/src/Examine.AzureDirectory
Hi, I have realtive issue. I have Umbraco 7.7.9 on Azure and large indexes. I have tried SyncTempEnvDirectoryFactory setting but I got the locking issue. I switched to TempEnvDirectoryFactory and I have issues only when Azure switches Web Workers. The website is broken by the time the indexes get rebuild. I understand the difference between the two settings and I think that TempEnvDirectoryFactory is better option in my case but I still need a solution when the Web Workers switch.
Mila
We've started getting issues like this. Was our first site using Umbraco and it's been a buggy experience with little things like this brining down the whole site. Sadly, this will also be our last Umbraco site.
That’s sad to hear you haven’t had a good experience.
Do you have the above umbracoLocalTempStorage setting in place?
Hi, Jeavon, The setting you asked is set:
Should we expect fixes for this in the later Umbraco versions?
That should be set and also do you have the Examine directoryFactory set to use SyncTempEnvDirectoryFactory?
Hi, Jeavon, My directoryFactory is set to TempEnvDirectoryFactory.
You need to set
<Examine RebuildOnAppStart="false">explicitly in examine configuration if you are intend to use SyncTempEnvDirectoryFactory .Is there a v8 equivalent to this?
Does disabling RebuildOnAppStart setting is really needed? Anyone tested that?
I have a similar problem but with Umbraco 8. I've registered a custom index like this : https://our.umbraco.com/forum/umbraco-8/95915-umbraco-8-examine-indexes. But now, when i restart the app, I get a lock exception. I've manage to work around it and force clear the lock but I don't think it's the right way. Also, Umbraco 8 doesn't contains any examine config file so i'm not able to use SyncTempEnvDirectoryFactory. If anyone knows how this should work, I'll gladly take an helping hand on this one.
Upgrading Examine to v1.0.1 fixed the issue for me in Azure.
What version of umbraco are you using?
Upgrade examine to 1.0.1 seems to fix it for me on Cloud but not locally, and i was using Umbraco Cloud 8.1.1 and Examine 1.0.0
Be aware that Examine also installs Lucene.Net and SharpZipLib
For anyone still having issues with the Examine Lock ,. we changed the ExamineComponent.cs file to check for lock and unlock it .
Where did you put that code?
We are using Umbraco 8.2 on Azure Web App: Web.Config settings are:
We don't use any custom indexes.
Hi , in V8 , examine is under application/initialization folder. We tried the code in config too , But this seems to work for us
Andy
is working on a reply...