Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Tom van Enckevort 86 posts 358 karma points
    Mar 02, 2018 @ 09:57
    Tom van Enckevort
    0

    Examine lock error brings down Azure site

    We have an v7.5.14 site running on Azure (Web Apps) and we've had a couple of downtime issues over the last few days where all requests would fail with the following error:

    Lock obtain timed out: NativeFSLock@D:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\Index\write.lock: System.IO.IOException: The process cannot access the file 'D:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\Index\write.lock' because it is being used by another process.
       at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
       at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
       at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
       at Lucene.Net.Store.NativeFSLock.Obtain()
       at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
       at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
       at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
       at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
       at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
    
    Lock obtain timed out: NativeFSLock@D:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\Index\write.lock: System.IO.IOException: The process cannot access the file 'D:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\Index\write.lock' because it is being used by another process.
       at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
       at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
       at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
       at Lucene.Net.Store.NativeFSLock.Obtain()
       at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout)
       at Examine.LuceneEngine.Directories.MultiIndexLock.Obtain(Int64 lockWaitTimeout)
       at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean create, Boolean closeDir, IndexDeletionPolicy deletionPolicy, Boolean autoCommit, Int32 maxFieldLength, IndexingChain indexingChain, IndexCommit commit)
       at Lucene.Net.Index.IndexWriter..ctor(Directory d, Analyzer a, Boolean create, MaxFieldLength mfl)
       at Examine.LuceneEngine.Providers.LuceneIndexer.WriterFactory(Directory d)
       at Examine.LuceneEngine.WriterTracker.<>c__DisplayClass7_0.
    
    <GetWriter>b__0(String s)
       at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
       at Examine.LuceneEngine.WriterTracker.GetWriter(Directory dir, Func`2 factory)
       at Examine.LuceneEngine.Providers.LuceneIndexer.CreateIndexWriter()
       at UmbracoExamine.BaseUmbracoIndexer.CreateIndexWriter()
       at Examine.LuceneEngine.Providers.LuceneIndexer.EnsureIndex(Boolean forceOverwrite)
       at Examine.LuceneEngine.Providers.LuceneIndexer.RebuildIndex()
       at UmbracoExamine.UmbracoContentIndexer.RebuildIndex()
       at Umbraco.Web.WebBootManager.
    
        <InitializeResolvers>b__f()
       at Umbraco.Core.Sync.DatabaseServerMessenger.Initialize()
       at Umbraco.Core.Sync.DatabaseServerMessenger.Boot()
       at Umbraco.Web.BatchedDatabaseServerMessenger.Startup()
       at Umbraco.Web.BatchedDatabaseServerMessengerStartup.ApplicationStarted(UmbracoApplicationBase umbracoApplication, ApplicationContext applicationContext)
       at Umbraco.Core.ApplicationEventHandler.OnApplicationStarted(UmbracoApplicationBase umbracoApplication, ApplicationContext applicationContext)
       at Umbraco.Core.CoreBootManager.
    
            <Complete>b__a(IApplicationEventHandler x)
       at Umbraco.Core.EnumerableExtensions.ForEach[TItem](IEnumerable`1 items, Action`1 action)
       at Umbraco.Core.CoreBootManager.Complete(Action`1 afterComplete)
       at Umbraco.Web.WebBootManager.Complete(Action`1 afterComplete)
       at Umbraco.Core.UmbracoApplicationBase.StartApplication(Object sender, EventArgs e)
       at Umbraco.Core.UmbracoApplicationBase.Application_Start(Object sender, EventArgs e)
    
    The process cannot access the file 'D:\home\site\wwwroot\App_Data\TEMP\ExamineIndexes\Internal\Index\write.lock' because it is being used by another process.
       at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
       at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
       at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
       at Lucene.Net.Store.NativeFSLock.Obtain()
    

    We've followed all the recommended Azure settings, so I'm not sure what could cause this lock issue.

    Has anyone seen this before?

  • Jeavon Leopold 2993 posts 13061 karma points MVP 5x admin c-trib
    Mar 02, 2018 @ 09:59
    Jeavon Leopold
    0

    What settings are you using for ExamineSettings.config as you are almost certainly missing something.

    https://our.umbraco.org/documentation/Getting-Started/Setup/Server-Setup/azure-web-apps#examine-v0-1-80

  • Tom van Enckevort 86 posts 358 karma points
    Mar 02, 2018 @ 10:05
    Tom van Enckevort
    0

    It seems set up right to me, but I might be missing something.

    ExamineSettings.config:

    <?xml version="1.0"?>
    <Examine>
        <ExamineIndexProviders>
            <providers>
                <add name="InternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine" supportUnpublished="true" supportProtected="true" analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net" directoryFactory="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory,Examine" />
                <add name="InternalMemberIndexer" type="UmbracoExamine.UmbracoMemberIndexer, UmbracoExamine" supportUnpublished="true" supportProtected="true" analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" directoryFactory="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory,Examine" />
                <!-- default external indexer, which excludes protected and unpublished pages-->
                <add name="ExternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine" directoryFactory="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory,Examine" />
            </providers>
        </ExamineIndexProviders>
        <ExamineSearchProviders defaultProvider="ExternalSearcher">
            <providers>
                <add name="InternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine" analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net" directoryFactory="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory,Examine" />
                <add name="ExternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine" enableDefaultEventHandler="true" />
                <add name="InternalMemberSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine" analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" directoryFactory="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory,Examine" enableLeadingWildcard="true" />
            </providers>
        </ExamineSearchProviders>
    </Examine>
    

    ExamineIndex.config:

    <?xml version="1.0"?>
    <ExamineLuceneIndexSets>
        <!-- The internal index set used by Umbraco back-office - DO NOT REMOVE -->
        <IndexSet SetName="InternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/Internal/" />
        <!-- The internal index set used by Umbraco back-office for indexing members - DO NOT REMOVE -->
        <IndexSet SetName="InternalMemberIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/InternalMember/">
            <IndexAttributeFields>
                <add Name="id" />
                <add Name="nodeName"/>
                <add Name="updateDate" />
                <add Name="writerName" />
                <add Name="loginName" />
                <add Name="email" />
                <add Name="nodeTypeAlias" />
            </IndexAttributeFields>
        </IndexSet>
        <!-- Default Indexset for external searches, this indexes all fields on all types of nodes-->
        <IndexSet SetName="ExternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/External/" />
    </ExamineLuceneIndexSets>
    
  • Jeavon Leopold 2993 posts 13061 karma points MVP 5x admin c-trib
    Mar 02, 2018 @ 10:14
    Jeavon Leopold
    0

    Are you load balanced or single instance?

  • Jeavon Leopold 2993 posts 13061 karma points MVP 5x admin c-trib
    Mar 02, 2018 @ 10:16
    Jeavon Leopold
    0

    Also what version of Examine do you have?

  • Tom van Enckevort 86 posts 358 karma points
    Mar 02, 2018 @ 10:18
    Tom van Enckevort
    0

    A single instance.

    I just noticed that there is no {machinename} in the IndexPath attributes in ExamineIndex.config.

    The documentation mentions that for Examine pre-v0.1.80 this needs to be removed, but does that mean for v0.1.80+ it does need to be there? (the site is running Examine v0.1.82)

  • Jeavon Leopold 2993 posts 13061 karma points MVP 5x admin c-trib
    Mar 02, 2018 @ 10:22
    Jeavon Leopold
    0

    No, that shouldn't be needed, do you have large indexes?

  • Tom van Enckevort 86 posts 358 karma points
    Mar 02, 2018 @ 10:25
    Tom van Enckevort
    0

    Nothing in the InternalMemberIndexer.

    InternalIndexer: 25062 documents (96 fields)

    ExternalIndexer: 24752 documents (96 fields)

  • Jeavon Leopold 2993 posts 13061 karma points MVP 5x admin c-trib
    Mar 02, 2018 @ 10:42
    Jeavon Leopold
    100

    SyncTempEnvDirectoryFactory is designed so that indexes don't have to be rebuilt when your app is auto migrated from one server to another as they will be synced back from the file system to the temp folder and then accessed from there. However it seems that the sync process is encountering a file system lock issue. It might be worth creating an issue on the Examine repo as your settings are correct. https://github.com/Shazwazza/Examine/issues

    I have seen horrible lock issues writing to the file system on Azure Apps that seem to come from the low latency and possible migration of the storage location, this is what lead to the creation of the umbracoLocalTempStorage setting.

    You could try using the TempEnvDirectoryFactory instead of SyncTempEnvDirectoryFactory, this factory doesn't do any syncing and only uses the local temp file system for index storage so doesn't suffer lock issues. However when you app auto migrated from one server to another the indexes will have to be rebuilt from scratch.

  • Tom van Enckevort 86 posts 358 karma points
    Mar 02, 2018 @ 10:51
    Tom van Enckevort
    0

    Yes, I've been looking through the event log for the app and I can see that the service instance ID did change around the time the errors started occurring, so that matches your theory it happened during the index file sync.

    But I guess the advice in this case would be to get the site upgraded to v7.7.3+ to make use of the new umbracoLocalTempStorage setting that should avoid locking issues?

  • Jeavon Leopold 2993 posts 13061 karma points MVP 5x admin c-trib
    Mar 02, 2018 @ 11:08
    Jeavon Leopold
    1

    In this case your issue is with Examine so changing to TempEnvDirectoryFactory would be my recommendation (you will need to update Examine to 0.1.83+). That said I would recommend using umbracoLocalTempStorage:EnvironmentTemp which will require your updating Umbraco to (7.7.3+) too to avoid other locking issues with auto server migration in Azure Apps.

  • Jeavon Leopold 2993 posts 13061 karma points MVP 5x admin c-trib
    Mar 02, 2018 @ 11:11
    Jeavon Leopold
    0

    There is more info on why we added umbracoLocalTempStorage here

  • Tom van Enckevort 86 posts 358 karma points
    Mar 02, 2018 @ 11:15
    Tom van Enckevort
    0

    That's great. Thanks for your suggestions, and explaining why the issue occurred!

  • Jeavon Leopold 2993 posts 13061 karma points MVP 5x admin c-trib
    Mar 02, 2018 @ 11:17
    Jeavon Leopold
    1

    Hopefully one day we will be able to store Examine indexes in blob storage and then sync to local temp so we can have the best of all worlds. There has been some work on this but it's not production ready. https://github.com/Shazwazza/Examine/tree/master/src/Examine.AzureDirectory

  • Mila Pandurska 18 posts 87 karma points
    Nov 18, 2018 @ 16:00
    Mila Pandurska
    0

    Hi, I have realtive issue. I have Umbraco 7.7.9 on Azure and large indexes. I have tried SyncTempEnvDirectoryFactory setting but I got the locking issue. I switched to TempEnvDirectoryFactory and I have issues only when Azure switches Web Workers. The website is broken by the time the indexes get rebuild. I understand the difference between the two settings and I think that TempEnvDirectoryFactory is better option in my case but I still need a solution when the Web Workers switch.

    Mila

  • Tony Kenny 2 posts 72 karma points
    Nov 20, 2018 @ 22:10
    Tony Kenny
    0

    We've started getting issues like this. Was our first site using Umbraco and it's been a buggy experience with little things like this brining down the whole site. Sadly, this will also be our last Umbraco site.

  • Jeavon Leopold 2993 posts 13061 karma points MVP 5x admin c-trib
    Nov 20, 2018 @ 22:21
    Jeavon Leopold
    0

    That’s sad to hear you haven’t had a good experience.

    Do you have the above umbracoLocalTempStorage setting in place?

  • Mila Pandurska 18 posts 87 karma points
    Nov 21, 2018 @ 08:46
    Mila Pandurska
    0

    Hi, Jeavon, The setting you asked is set:

      <add key="umbracoLocalTempStorage" value="EnvironmentTemp" />
    

    Should we expect fixes for this in the later Umbraco versions?

  • Jeavon Leopold 2993 posts 13061 karma points MVP 5x admin c-trib
    Nov 22, 2018 @ 10:10
    Jeavon Leopold
    0

    That should be set and also do you have the Examine directoryFactory set to use SyncTempEnvDirectoryFactory?

  • Mila Pandurska 18 posts 87 karma points
    Nov 29, 2018 @ 10:02
    Mila Pandurska
    0

    Hi, Jeavon, My directoryFactory is set to TempEnvDirectoryFactory.

    directoryFactory="Examine.LuceneEngine.Directories.TempEnvDirectoryFactory,Examine"
    
  • Sraddhananda 1 post 71 karma points
    Mar 05, 2019 @ 12:21
    Sraddhananda
    0

    You need to set <Examine RebuildOnAppStart="false"> explicitly in examine configuration if you are intend to use SyncTempEnvDirectoryFactory .

  • Chris Spanellis 12 posts 122 karma points
    Apr 18, 2019 @ 12:29
    Chris Spanellis
    0

    Is there a v8 equivalent to this?

  • Łukasz Koruba 16 posts 116 karma points
    1 week ago
    Łukasz Koruba
    0

    Does disabling RebuildOnAppStart setting is really needed? Anyone tested that?

  • Stherm 5 posts 75 karma points
    Mar 11, 2019 @ 18:59
    Stherm
    0

    I have a similar problem but with Umbraco 8. I've registered a custom index like this : https://our.umbraco.com/forum/umbraco-8/95915-umbraco-8-examine-indexes. But now, when i restart the app, I get a lock exception. I've manage to work around it and force clear the lock but I don't think it's the right way. Also, Umbraco 8 doesn't contains any examine config file so i'm not able to use SyncTempEnvDirectoryFactory. If anyone knows how this should work, I'll gladly take an helping hand on this one.

  • Chris Spanellis 12 posts 122 karma points
    29 days ago
    Chris Spanellis
    0

    Upgrading Examine to v1.0.1 fixed the issue for me in Azure.

Please Sign in or register to post replies

Write your reply to:

Draft