Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • nickornotto 279 posts 611 karma points
    24 days ago
    nickornotto
    0

    What's the best Umbraco websites, subdomains and indexes configuration to avoid Lucene errors?

    We are experiencing a series of application errors on Lucene indexes that result in the website crash and apppool restart.

    I suspect this might be caused by our websites, hostnames and indexes configuration. We have multiple root nodes in our Umbraco CMS (v 7.15.3 currently). The Umbraco content is structured in the following way:

    • Home -> site under mydomain.com hostname
    • Providers -> site under business.mydomain.com hostname
    • Reviews
    • Bookings
    • Settings etc.

    As per the above two first roots have separate hostnames: mydomain.com and business.mydomain.com. In practice both point to the same Umbraco instance on IIS of course.

    Both domains have separate IIS site entries and pools (with the same pool settings).

    Both domains though use the same InternalIndexSet. This is because the main website uses it for indexing the Umbraco content, and our members can access business.mydomain.com to update their content. They are able to see unpublished content and publish it.

    The error occurs normally when we are publishing a content in the CMS while a member is saving his content in the business area at the same time. We are able to replicate the issue while doing this (updating content on both sites).

    We had before the following index setup:

    <ExamineLuceneIndexSets>
      <IndexSet SetName="InternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/Internal/"/>
        <IndexSet SetName="MemberProductsIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/MemberProducts/" IndexParentId="1179">      </IndexSet>
    <IndexSet SetName="PublishedContentIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/PublishedContent/" IndexParentId="1179">
    </IndexSet>
    </ExamineLuceneIndexSets>
    

    where the IndexParentId is the id of the Home root (although I think it doesn't matter in this case). MemberProducts index was for both published & unpublished content, and PublishedContent is for only published content.

    And the errors were once related to the InternalIndex another time to MemberProducts index depending probably which one was in use first.

    We reduced the two above indexes to this:

    <ExamineLuceneIndexSets>
      <IndexSet SetName="InternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/Internal/"/>
    <IndexSet SetName="PublishedContentIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/PublishedContent/" IndexParentId="1179">
    </IndexSet>
    </ExamineLuceneIndexSets>
    

    So now both use the same InternalIndex but we are still experiencing the problems, just on InternalIndex only now since the other index has gone.

    The error we are getting can be one of this kind:

    Exception: System.IO.IOException
    
    Message: Cannot overwrite: C:\Inetpub\hosts\mydomain.com\httpdocs\App_Data\TEMP\ExamineIndexes\Internal\Index\_1.cfs
    
    StackTrace:    at Lucene.Net.Store.FSDirectory.InitOutput(String name)
       at Lucene.Net.Store.SimpleFSDirectory.CreateOutput(String name)
       at Lucene.Net.Index.CompoundFileWriter.Close()
       at Lucene.Net.Index.DocumentsWriter.CreateCompoundFile(String segment)
       at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes)
       at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes)
       at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary`2 commitUserData, Boolean internal_Renamed)
       at Lucene.Net.Index.IndexWriter.Commit(IDictionary`2 commitUserData)
       at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.TimerRelease() in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1711
       at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.<ScheduleCommit>b__7_0(Object _) in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1663
       at System.Threading.TimerQueueTimer.CallCallbackInContext(Object state)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.TimerQueueTimer.CallCallback()
       at System.Threading.TimerQueueTimer.Fire()
       at System.Threading.TimerQueue.FireNextTimers()
       at System.Threading.TimerQueue.AppDomainTimerCallback(Int32 id)
    

    or

    Exception: System.IO.IOException
    
    Message: Cannot overwrite: C:\Inetpub\hosts\mydomain.com\httpdocs\App_Data\TEMP\ExamineIndexes\Internal\Index\_52.cfs
    
    StackTrace:    at Lucene.Net.Store.FSDirectory.InitOutput(String name)
       at Lucene.Net.Store.SimpleFSDirectory.CreateOutput(String name)
       at Lucene.Net.Index.CompoundFileWriter.Close()
       at Lucene.Net.Index.DocumentsWriter.CreateCompoundFile(String segment)
       at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes)
       at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes)
       at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary`2 commitUserData, Boolean internal_Renamed)
       at Lucene.Net.Index.IndexWriter.Commit(IDictionary`2 commitUserData)
       at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.TimerRelease() in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1711
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.TimerQueueTimer.CallCallback()
       at System.Threading.TimerQueueTimer.Fire()
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()
    

    or

    Exception: System.IndexOutOfRangeException
    
    Message: Index of bound 1
    
    StackTrace:    at Lucene.Net.Util.BitVector.GetAndSet(Int32 bit)
       at Lucene.Net.Index.SegmentReader.DoDelete(Int32 docNum)
       at Lucene.Net.Index.IndexReader.DeleteDocument(Int32 docNum)
       at Lucene.Net.Index.DocumentsWriter.ApplyDeletes(IndexReader reader, Int32 docIDStart)
       at Lucene.Net.Index.DocumentsWriter.ApplyDeletes(SegmentInfos infos)
       at Lucene.Net.Index.IndexWriter.ApplyDeletes()
       at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes)
       at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes)
       at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary`2 commitUserData, Boolean internal_Renamed)
       at Lucene.Net.Index.IndexWriter.Commit(IDictionary`2 commitUserData)
       at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.TimerRelease() in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1711
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.TimerQueueTimer.CallCallback()
       at System.Threading.TimerQueueTimer.Fire()
       at System.Threading.TimerQueue.FireNextTimers()
    

    depending probably on the sequence of events, eg. which webiste tries to access the index first etc.

    As I mentioned the error happens when a content is being updated and saved/published at the same time on both mydomain.com and business.mydomain.com. And it doesn't need to be the same content. The same issue were happening on previous versions of Umbraco (7.6 & 7.15.1).

    I wonder if our settings are correct eg. the fact that we have different hostname using the same files and indexes but different pools is causing the issues?

    If so what would be a correct/ suggested configuration?

    We have the same configuration on localhost and stage site. The only difference is that they don't have a separate business subdomain but rather use stage.mydomain.com/members hostname instead of separate subdomain for members area.

    This issue does not occur in any of them.

    We are using a dedicated server with IIS10 on Windows Server 2016 Standard, no load balancing and no machine key.

  • nickornotto 279 posts 611 karma points
    20 days ago
    nickornotto
    0

    Anyone can help?

  • Shannon Deminick 1498 posts 5073 karma points hq
    20 days ago
    Shannon Deminick
    1

    I'm unsure about that ExamineLuceneIndexSets you've posted above since it shows that you have 2x indexes with the same name PublishedContentIndexSet pointing to different places which cannot be done and I'm surprised it lets you do that. Each index set name needs to be unique and correspond with the index name, see https://github.com/Shazwazza/Examine/wiki/NamingConventions

    Examine doesn't care about your domain configuration, it doesn't actually know anything about that.

    You should never use the internal index for your front-end since this will contain unpublished data (that is why the ExternalIndex is available for you to use), but maybe that what you want, not sure. .

    In the GH post https://github.com/umbraco/Umbraco-CMS/issues/3683 you've said you are not load balancing but here you say:

    depending probably on the sequence of events, eg. which webiste tries to access the index first etc.

    and

    We are able to replicate the issue while doing this (updating content on both sites).

    To me this sounds like load balancing.

    Then you say:

    eg. the fact that we have different hostname using the same files and indexes but different pools is causing the issues?

    Can you please explain this? This also sounds like multiple processes accessing the same lucene files at the same time which is not supported. Does this mean you have multiple IIS processes/sites pointing to the same website files?? If so, then this is not supported in this setup and is actually load balancing. If you have more than one process accessing your single database, you are load balancing and you will need to configure your sites accordingly.

    This is not an issue we've seen and there are a ton of sites running on 7.x and in load balanced scenarios so this will most certainly be down to a configuration/hosting setup issue.

  • nickornotto 279 posts 611 karma points
    18 days ago
    nickornotto
    0

    Hi, sorry for the confusion. Yes, of course we cannot have two indexes with the same name, there was a mistake in my description above - I corrected it - and here's the correct set:

    <ExamineLuceneIndexSets>
      <IndexSet SetName="InternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/Internal/"/>
        <IndexSet SetName="MemberProductsIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/MemberProducts/" IndexParentId="1179">      </IndexSet>
    <IndexSet SetName="PublishedContentIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/PublishedContent/" IndexParentId="1179">
    </IndexSet>
    </ExamineLuceneIndexSets>
    

    In the GH post https://github.com/umbraco/Umbraco-CMS/issues/3683 you've said you are not load balancing but here you say:

    depending probably on the sequence of events, eg. which webiste tries to access the index first etc.

    and

    We are able to replicate the issue while doing this (updating content on both sites).

    To me this sounds like load balancing.

    By website I mean domain.

    For example if someone tries access the index on mydomain.com and someone else on business.mydomain.com the index is failing as described above.

    They are not different websites or machines, they are different domains on the same machine pointing to the same umbraco instance and thus using the same indexes, database and content.

    business.mydomain.com is for our memebrs who update their products which are in result available for purchase in the public facing mydomain.com. So naturally both access the same products content.

    I am not sure it falls under load balancing, that's why I'm asking the question whether this is potentially what are causing our issues.

  • Shannon Deminick 1498 posts 5073 karma points hq
    7 days ago
    Shannon Deminick
    0

    I'm still not clear on this

    They are not different websites or machines, they are different domains on the same machine pointing to the same umbraco instance and thus using the same indexes, database and content.

    Are you using the same IIS website for both domains or are you using 2 different IIS websites pointing to the same physical location?

  • nickornotto 279 posts 611 karma points
    7 days ago
    nickornotto
    0

    We are using the same server, same IIS, same web.config for main and subdomain website so to me it's not load balancing.

  • Shannon Deminick 1498 posts 5073 karma points hq
    6 days ago
    Shannon Deminick
    0

    Sure, i just need to know that there aren't more than one single process accessing the same files at the same time. The errors you are getting indicate exactly that - that there is more than one process accessing the files at the same time.

    This is what concerns me:

    Both domains have separate IIS site entries and pools (with the same pool settings).

    That to me says there is 2x IIS websites configured in your IIS with different application pools pointing to the same physical files.

  • nickornotto 279 posts 611 karma points
    6 days ago
    nickornotto
    0

    I meant there are 2 websites set upin IIS, eg. mydomain.com and business.mydomain.com so they have separate apppools as well.

    However since I started this thread we have changed our configuration:

    • we have discarded business.mydomain.com and now this website is served under mydomain.com/business. This means no separate websites or pools in the ISS (14 May)
    • we have added our custom member index so not to use InternalMember umbraco's index (17 May)

    So far the numbers of Lucene errors have been heavily reduced, still happening once a day though. In addition we are still getting cryptographic error once a day (I have described it elsewhere I think on github) but this improved as well after the above changed had been made.

    My main concern is though why our configuration with the same umbraco instance served by different domains cause any problems to indexes. I am pretty sure lots of umbraco site are running this way and not sure would it cause us issues in particular and how other such websites are handling that.

  • Shannon Deminick 1498 posts 5073 karma points hq
    6 days ago
    Shannon Deminick
    0

    So this is exactly the problem.

    enter image description here

    You cannot have more than 1 IIS website pointing to the same physical file system. Doing this is exactly the same as if each IIS website existed on different servers pointing to the same UNC file share. It really doesn't matter if they are using the same or different App pool. If they were in the same app pool they are still running under separate AppDomains with different IDs - basically segregated internal processes. Running in different AppPools just means even more segregation but ultimately for the sake of this explanation it doesn't matter. You cannot have multiple separate AppDomains/processes accessing the same file system at the same time.

    This setup is load balancing except in an unsupported way:

    • More than one process and AppDomain that have different application IDs are accessing the same files - cannot be done
    • More than one Umbraco website is being used as the master server - this setup means you have multiple master (read + write) servers writing to both the database and files - cannot be done

    It is possible to setup load balancing using a centralized file share, the docs for that are here https://our.umbraco.com/Documentation/Getting-Started/Setup/Server-Setup/Load-Balancing/files-shared-v7 but you still cannot have multiple master servers.

    You should have a single IIS site with all of your domains assigned. If you want to distribute the load, then you need to configure load balancing.

Please Sign in or register to post replies

Write your reply to:

Draft