Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • nickornotto 397 posts 900 karma points
    May 03, 2020 @ 07:18
    nickornotto
    0

    What's the best Umbraco websites, subdomains and indexes configuration to avoid Lucene errors?

    We are experiencing a series of application errors on Lucene indexes that result in the website crash and apppool restart.

    I suspect this might be caused by our websites, hostnames and indexes configuration. We have multiple root nodes in our Umbraco CMS (v 7.15.3 currently). The Umbraco content is structured in the following way:

    • Home -> site under mydomain.com hostname
    • Providers -> site under business.mydomain.com hostname
    • Reviews
    • Bookings
    • Settings etc.

    As per the above two first roots have separate hostnames: mydomain.com and business.mydomain.com. In practice both point to the same Umbraco instance on IIS of course.

    Both domains have separate IIS site entries and pools (with the same pool settings).

    Both domains though use the same InternalIndexSet. This is because the main website uses it for indexing the Umbraco content, and our members can access business.mydomain.com to update their content. They are able to see unpublished content and publish it.

    The error occurs normally when we are publishing a content in the CMS while a member is saving his content in the business area at the same time. We are able to replicate the issue while doing this (updating content on both sites).

    We had before the following index setup:

    <ExamineLuceneIndexSets>
      <IndexSet SetName="InternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/Internal/"/>
        <IndexSet SetName="MemberProductsIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/MemberProducts/" IndexParentId="1179">      </IndexSet>
    <IndexSet SetName="PublishedContentIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/PublishedContent/" IndexParentId="1179">
    </IndexSet>
    </ExamineLuceneIndexSets>
    

    where the IndexParentId is the id of the Home root (although I think it doesn't matter in this case). MemberProducts index was for both published & unpublished content, and PublishedContent is for only published content.

    And the errors were once related to the InternalIndex another time to MemberProducts index depending probably which one was in use first.

    We reduced the two above indexes to this:

    <ExamineLuceneIndexSets>
      <IndexSet SetName="InternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/Internal/"/>
    <IndexSet SetName="PublishedContentIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/PublishedContent/" IndexParentId="1179">
    </IndexSet>
    </ExamineLuceneIndexSets>
    

    So now both use the same InternalIndex but we are still experiencing the problems, just on InternalIndex only now since the other index has gone.

    The error we are getting can be one of this kind:

    Exception: System.IO.IOException
    
    Message: Cannot overwrite: C:\Inetpub\hosts\mydomain.com\httpdocs\App_Data\TEMP\ExamineIndexes\Internal\Index\_1.cfs
    
    StackTrace:    at Lucene.Net.Store.FSDirectory.InitOutput(String name)
       at Lucene.Net.Store.SimpleFSDirectory.CreateOutput(String name)
       at Lucene.Net.Index.CompoundFileWriter.Close()
       at Lucene.Net.Index.DocumentsWriter.CreateCompoundFile(String segment)
       at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes)
       at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes)
       at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary`2 commitUserData, Boolean internal_Renamed)
       at Lucene.Net.Index.IndexWriter.Commit(IDictionary`2 commitUserData)
       at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.TimerRelease() in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1711
       at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.<ScheduleCommit>b__7_0(Object _) in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1663
       at System.Threading.TimerQueueTimer.CallCallbackInContext(Object state)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.TimerQueueTimer.CallCallback()
       at System.Threading.TimerQueueTimer.Fire()
       at System.Threading.TimerQueue.FireNextTimers()
       at System.Threading.TimerQueue.AppDomainTimerCallback(Int32 id)
    

    or

    Exception: System.IO.IOException
    
    Message: Cannot overwrite: C:\Inetpub\hosts\mydomain.com\httpdocs\App_Data\TEMP\ExamineIndexes\Internal\Index\_52.cfs
    
    StackTrace:    at Lucene.Net.Store.FSDirectory.InitOutput(String name)
       at Lucene.Net.Store.SimpleFSDirectory.CreateOutput(String name)
       at Lucene.Net.Index.CompoundFileWriter.Close()
       at Lucene.Net.Index.DocumentsWriter.CreateCompoundFile(String segment)
       at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes)
       at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes)
       at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary`2 commitUserData, Boolean internal_Renamed)
       at Lucene.Net.Index.IndexWriter.Commit(IDictionary`2 commitUserData)
       at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.TimerRelease() in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1711
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.TimerQueueTimer.CallCallback()
       at System.Threading.TimerQueueTimer.Fire()
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()
    

    or

    Exception: System.IndexOutOfRangeException
    
    Message: Index of bound 1
    
    StackTrace:    at Lucene.Net.Util.BitVector.GetAndSet(Int32 bit)
       at Lucene.Net.Index.SegmentReader.DoDelete(Int32 docNum)
       at Lucene.Net.Index.IndexReader.DeleteDocument(Int32 docNum)
       at Lucene.Net.Index.DocumentsWriter.ApplyDeletes(IndexReader reader, Int32 docIDStart)
       at Lucene.Net.Index.DocumentsWriter.ApplyDeletes(SegmentInfos infos)
       at Lucene.Net.Index.IndexWriter.ApplyDeletes()
       at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes)
       at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes)
       at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary`2 commitUserData, Boolean internal_Renamed)
       at Lucene.Net.Index.IndexWriter.Commit(IDictionary`2 commitUserData)
       at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.TimerRelease() in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1711
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.TimerQueueTimer.CallCallback()
       at System.Threading.TimerQueueTimer.Fire()
       at System.Threading.TimerQueue.FireNextTimers()
    

    depending probably on the sequence of events, eg. which webiste tries to access the index first etc.

    As I mentioned the error happens when a content is being updated and saved/published at the same time on both mydomain.com and business.mydomain.com. And it doesn't need to be the same content. The same issue were happening on previous versions of Umbraco (7.6 & 7.15.1).

    I wonder if our settings are correct eg. the fact that we have different hostname using the same files and indexes but different pools is causing the issues?

    If so what would be a correct/ suggested configuration?

    We have the same configuration on localhost and stage site. The only difference is that they don't have a separate business subdomain but rather use stage.mydomain.com/members hostname instead of separate subdomain for members area.

    This issue does not occur in any of them.

    We are using a dedicated server with IIS10 on Windows Server 2016 Standard, no load balancing and no machine key.

  • nickornotto 397 posts 900 karma points
    May 06, 2020 @ 17:49
    nickornotto
    0

    Anyone can help?

  • Shannon Deminick 1524 posts 5269 karma points MVP 2x
    May 07, 2020 @ 04:58
    Shannon Deminick
    1

    I'm unsure about that ExamineLuceneIndexSets you've posted above since it shows that you have 2x indexes with the same name PublishedContentIndexSet pointing to different places which cannot be done and I'm surprised it lets you do that. Each index set name needs to be unique and correspond with the index name, see https://github.com/Shazwazza/Examine/wiki/NamingConventions

    Examine doesn't care about your domain configuration, it doesn't actually know anything about that.

    You should never use the internal index for your front-end since this will contain unpublished data (that is why the ExternalIndex is available for you to use), but maybe that what you want, not sure. .

    In the GH post https://github.com/umbraco/Umbraco-CMS/issues/3683 you've said you are not load balancing but here you say:

    depending probably on the sequence of events, eg. which webiste tries to access the index first etc.

    and

    We are able to replicate the issue while doing this (updating content on both sites).

    To me this sounds like load balancing.

    Then you say:

    eg. the fact that we have different hostname using the same files and indexes but different pools is causing the issues?

    Can you please explain this? This also sounds like multiple processes accessing the same lucene files at the same time which is not supported. Does this mean you have multiple IIS processes/sites pointing to the same website files?? If so, then this is not supported in this setup and is actually load balancing. If you have more than one process accessing your single database, you are load balancing and you will need to configure your sites accordingly.

    This is not an issue we've seen and there are a ton of sites running on 7.x and in load balanced scenarios so this will most certainly be down to a configuration/hosting setup issue.

  • nickornotto 397 posts 900 karma points
    May 08, 2020 @ 15:23
    nickornotto
    0

    Hi, sorry for the confusion. Yes, of course we cannot have two indexes with the same name, there was a mistake in my description above - I corrected it - and here's the correct set:

    <ExamineLuceneIndexSets>
      <IndexSet SetName="InternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/Internal/"/>
        <IndexSet SetName="MemberProductsIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/MemberProducts/" IndexParentId="1179">      </IndexSet>
    <IndexSet SetName="PublishedContentIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/PublishedContent/" IndexParentId="1179">
    </IndexSet>
    </ExamineLuceneIndexSets>
    

    In the GH post https://github.com/umbraco/Umbraco-CMS/issues/3683 you've said you are not load balancing but here you say:

    depending probably on the sequence of events, eg. which webiste tries to access the index first etc.

    and

    We are able to replicate the issue while doing this (updating content on both sites).

    To me this sounds like load balancing.

    By website I mean domain.

    For example if someone tries access the index on mydomain.com and someone else on business.mydomain.com the index is failing as described above.

    They are not different websites or machines, they are different domains on the same machine pointing to the same umbraco instance and thus using the same indexes, database and content.

    business.mydomain.com is for our memebrs who update their products which are in result available for purchase in the public facing mydomain.com. So naturally both access the same products content.

    I am not sure it falls under load balancing, that's why I'm asking the question whether this is potentially what are causing our issues.

  • Shannon Deminick 1524 posts 5269 karma points MVP 2x
    May 19, 2020 @ 23:34
    Shannon Deminick
    0

    I'm still not clear on this

    They are not different websites or machines, they are different domains on the same machine pointing to the same umbraco instance and thus using the same indexes, database and content.

    Are you using the same IIS website for both domains or are you using 2 different IIS websites pointing to the same physical location?

  • nickornotto 397 posts 900 karma points
    May 20, 2020 @ 06:37
    nickornotto
    0

    We are using the same server, same IIS, same web.config for main and subdomain website so to me it's not load balancing.

  • Shannon Deminick 1524 posts 5269 karma points MVP 2x
    May 20, 2020 @ 08:58
    Shannon Deminick
    0

    Sure, i just need to know that there aren't more than one single process accessing the same files at the same time. The errors you are getting indicate exactly that - that there is more than one process accessing the files at the same time.

    This is what concerns me:

    Both domains have separate IIS site entries and pools (with the same pool settings).

    That to me says there is 2x IIS websites configured in your IIS with different application pools pointing to the same physical files.

  • nickornotto 397 posts 900 karma points
    May 20, 2020 @ 12:47
    nickornotto
    0

    I meant there are 2 websites set upin IIS, eg. mydomain.com and business.mydomain.com so they have separate apppools as well.

    However since I started this thread we have changed our configuration:

    • we have discarded business.mydomain.com and now this website is served under mydomain.com/business. This means no separate websites or pools in the ISS (14 May)
    • we have added our custom member index so not to use InternalMember umbraco's index (17 May)

    So far the numbers of Lucene errors have been heavily reduced, still happening once a day though. In addition we are still getting cryptographic error once a day (I have described it elsewhere I think on github) but this improved as well after the above changed had been made.

    My main concern is though why our configuration with the same umbraco instance served by different domains cause any problems to indexes. I am pretty sure lots of umbraco site are running this way and not sure would it cause us issues in particular and how other such websites are handling that.

  • Shannon Deminick 1524 posts 5269 karma points MVP 2x
    May 20, 2020 @ 23:31
    Shannon Deminick
    0

    So this is exactly the problem.

    enter image description here

    You cannot have more than 1 IIS website pointing to the same physical file system. Doing this is exactly the same as if each IIS website existed on different servers pointing to the same UNC file share. It really doesn't matter if they are using the same or different App pool. If they were in the same app pool they are still running under separate AppDomains with different IDs - basically segregated internal processes. Running in different AppPools just means even more segregation but ultimately for the sake of this explanation it doesn't matter. You cannot have multiple separate AppDomains/processes accessing the same file system at the same time.

    This setup is load balancing except in an unsupported way:

    • More than one process and AppDomain that have different application IDs are accessing the same files - cannot be done
    • More than one Umbraco website is being used as the master server - this setup means you have multiple master (read + write) servers writing to both the database and files - cannot be done

    It is possible to setup load balancing using a centralized file share, the docs for that are here https://our.umbraco.com/Documentation/Getting-Started/Setup/Server-Setup/Load-Balancing/files-shared-v7 but you still cannot have multiple master servers.

    You should have a single IIS site with all of your domains assigned. If you want to distribute the load, then you need to configure load balancing.

  • nickornotto 397 posts 900 karma points
    Jun 08, 2020 @ 15:36
    nickornotto
    0

    Thanks for explaining this.

    We have changed our configuration so that business.mydomain.com has become www.mydomain.com/business so there is only one domain, one website pointing to the same app.

    It definately helped to the extend that we are able to update content via backoffice and our suppliers are able to update theirs.

    The cryptographic exception seems to be gone whatsoever (fingers crossed) which is not surprising without load-balacing-like setup.

    However we are still experiencing some issues with indexes. Roughly we are getting 1-2 of the above errors happening per day that cause the app pool crash.

  • Bo Jacobsen 590 posts 2366 karma points
    Jun 08, 2020 @ 18:49
    Bo Jacobsen
    0

    Hi nickornotto.

    If your business.mydomain.com and mydomain.com are running on the same Umbraco installation (as it seems). You can bind both the subdomain and the domain to the same application.

    When you click on the appropriate application, there should be a bind option in the right side, where you can bind both domains.

    In Umbraco you can then add a domain (business.mydomain.com) to your document at mydomain.com/business, and you will again have a two like website.

  • Amir Khan 1282 posts 2739 karma points
    Jun 08, 2020 @ 23:32
    Amir Khan
    0

    Exactly, we have a site for 1 client with ~40 domains and they're all one Umbraco instance, one site in IIS, one app pool, Umbraco can handle the routing with right click / culture and hostnames. Just need to add the bindings in IIS for the same site. Maybe useful to add this to the common pitfalls page in the docs?

  • nickornotto 397 posts 900 karma points
    Jun 09, 2020 @ 07:35
    nickornotto
    0

    Hi Bo, This is what we had adn we experienced those issues I initially outlined.

    We had business.mydomain.com & mydomain.com in the IIS as separate websites and the respective addresses bound to different roots within the same Umbraco app.

    Or they must be bound to the same root within the umbraco app for it to work?

  • nickornotto 397 posts 900 karma points
    Aug 15, 2020 @ 14:24
    nickornotto
    0

    After two months since the configuration change I can say that unfortunately it didn't solve the problem.

    The exception started to appear much less but over the time it raises gradually and now somerimes we are execting almost the same intensity of the issue as before.

    This is extremely frustrating and is a bad forecat for the application.

    In overall we were happy with umbraco but in these circumstances we are becoming very dissapointed sas it looks like it's not able to handle even small number of calls to API and indexes.

    The indexes crash (and crash the entire website) even if there seem not to be any activity on the site but upon saving a content in the CMS.

  • Shannon Deminick 1524 posts 5269 karma points MVP 2x
    Aug 17, 2020 @ 03:42
    Shannon Deminick
    0

    Hi Nick,

    I can see you've reported issues in multiple places and as there are multiple issues listed in all of these places it's hard to know what exact error(s) you are experiencing. Previously in this thread there were configuration problems with how things were hosted etc...

    Where do you wish to keep the conversation about what latest issues you have are? There are many many umbraco 7.x sites running in load balancing scenarios that don't exhibit issues. I don't know if we need to start from the beginning here since to help we'd need your exact hosting configuration again since I'm sure it has changed, versions you are using, exact errors and stack traces, etc... I'm happy to try to assist but I need specifics. Let me know where you want to continue that conversation.

  • nickornotto 397 posts 900 karma points
    Aug 21, 2020 @ 16:17
    nickornotto
    0

    Hi, Thanks for messaging and sorry for the delay, I have a very hectic week.

    The reason there are multiple issues reproted is because: 1. We are not (or were not initially) sure if various errors we are getting are connected so we seek advice for specific error messages. 2. Some threads like those on github were closed since I wrote in them so I couldn't continue the updates.

    I understand our solution is not unique but there is clearly a problem with indexes and I belive the problem is not unique. As you said somewhere - even if an index is getting corrupted it shouldn't bring the entire site down, this imo is just unacceptable.

    I also believe our current configuration is not load balancing unless accessing indexes by API is considered a load balancing factor? The cryptographic exception we were getting with the initial configuration has gone and now we 'only' have problems with indexes.

    I am happy to create a brand new thread here or on github with all our findings about the issue with indexes we are experiencing so that we could follow.

    I will post the link here once created.

  • nickornotto 397 posts 900 karma points
    Nov 18, 2020 @ 22:12
    nickornotto
    0

    This is the original issue we were getting and it is still happening: https://our.umbraco.com/forum/using-umbraco-and-getting-started/100682-unhandled-exception-in-appdomain-read-past-eof-error-in-lucene-indexing

    Also reported here: https://stackoverflow.com/questions/59725408/unhandled-exception-in-appdomain-read-past-eof-error-in-lucene-indexing

    Otherwise I was joining similar discussions but those above are my original reports of the issue we are experiencing.

  • Shannon Deminick 1524 posts 5269 karma points MVP 2x
    Nov 19, 2020 @ 01:37
    Shannon Deminick
    0

    Thanks nick, I've never seen the read past eof error but. Some of these are new stack traces too, I'm sure you saw my Examine report on a new issue which is pending release https://github.com/Shazwazza/Examine/issues/195 which is similar to another one but (like one of your stack traces) causes another odd error when committing. #195 will make sure the appdomain doesn't crash. I wonder if the eof error is a problem that occurs after this type of issue leaving the index in a very strange state since the appdomain is terminated.

  • Shannon Deminick 1524 posts 5269 karma points MVP 2x
    Nov 19, 2020 @ 01:40
    Shannon Deminick
    0

    Also note that this class is now used https://github.com/Shazwazza/Examine/commit/da998e9285417ecb8d0922990a44f832b6896127#diff-ef888fa730831bbe6920aae4049ab45cc04e79a33ee4c2b7189f1cbf93a9362d for the merge scheduler from 0.1.91 which means that eof error won't take down the appdomain

  • Thomsen 112 posts 335 karma points
    Aug 21, 2020 @ 12:59
    Thomsen
    0

    I have been struggeling with these problems since v8 was born, so I have created a new thread with a project and a demo site showcase with as much information about the setup and the hosting environment as possible. I have provided a login to the demo site and a download of the project.

    https://our.umbraco.com/forum/using-umbraco-and-getting-started//103281-unhandled-exception-in-appdomain-terminating-an-examine-index-and-lucene-errors-showcase

    I really hope this can help describe and show the problem and help to a solution.

    Thanks!

  • nickornotto 397 posts 900 karma points
    Aug 21, 2020 @ 16:19
    nickornotto
    0

    Thanks, I'll take a look. However our website is umbraco 7 based so your issues are probably a different story.

Please Sign in or register to post replies

Write your reply to:

Draft