I have faced with a problem and looking forward for a help from U.
Details:
The other day I got a message from Customer that our site is down. I checked the server and application itself and here is what I found:
CPU Usage almost 100% for my appPool
Memory usage almost 1.5GB (Enable 32-bit application : True)
"appcmd list requests" command for my AppPool showed hundreds
requests in a queue
Every other minute in Application Events : Warning ASP.NET with details:
Exception information:
Exception type: OutOfMemoryException
Exception message: Exception of type 'System.OutOfMemoryException' was thrown.
at System.String.CtorCharArrayStartLength(Char[] value, Int32 startIndex, Int32 length)
at Lucene.Net.Index.TermBuffer.ToTerm()
at Lucene.Net.Index.SegmentTermEnum.Term()
at Lucene.Net.Index.TermInfosReader.Get(Term term, Boolean useCache)
at Lucene.Net.Index.SegmentTermDocs.Seek(Term term)
at Lucene.Net.Search.MultiTermQueryWrapperFilter.TermGenerator.Generate(MultiTermQuery query, IndexReader reader, TermEnum enumerator)
at Lucene.Net.Search.MultiTermQueryWrapperFilter.GetDocIdSet(IndexReader reader)
at Lucene.Net.Search.ConstantScoreQuery.ConstantScorer..ctor(ConstantScoreQuery enclosingInstance, Similarity similarity, IndexReader reader, Weight w)...
As usual practice - I cleaned App_Data directory and restarted application... in a couple minutes of 100% CPU usage I got the same message with OutOfMemory ....
As quick solution I made ExamineIndex & ExamineSettings as disabled and restarted application - now it alive but without search possibility in backoffice (we don't use Examine in our code at all)
What is the most interesting that we got performance almost 4-5 times faster then it used to be with turned on Examine... and here is my main question - is there any possibility to get Examine back to application - but leave performance as it is now?
Additional details:
Umbraco 7.2.4 (I'm going to check the application performance and Examine availability after updating to the 7.4.3, but later)
Umbraco.config file is almost 30MB
I know that we have flaw of optimization in our code, it will be the next step, but even in current situation we got quite rare but heavy requests from 2-3 minutes to 20-30 seconds!
P.S. U are welcome with any kind of additional questions ...
Thanks in advance!
I just had the same issue on a new server today in fact - The main thing is that it's actually a server bug, which happens to affect Umbraco unfortunately.
It's very well described what should be done etc. on the issue itself.
Thank U for your time and advice...
My investigation in order:
server doesn't have mentioned updates installed (KB3000850 or
KB3007507) and I don't have any idea why :)
unfortunately I cleaned App_Data and I can't check Logs once again to
clarify if mentioned "Application shutdown" were there...
and while "OutOfMemory" exceptions are in Application log , which
wasn't mentioned in that issue ...
I think that it's certainly connected to Lucena. Especially when I already had this kind of issue before even with 7.1.4 on 2008 R2 server, with huge amount of nodes in content.
P.S. I feel like it's worth to check updated Umbraco with updated Examine... but again - performance increased considerably...
I have faced with the same performance issue caused my ExamineIndex on another server and this time mentioned updates are installed... but I can't see "Application shutdown. Reason: ConfigurationChange" in logs, as in the ticket.
Do U think it's worth to try to install even if not all conditions are satisfied?
ExamineIndex crashes application
Hello Dear EveryOne!
I have faced with a problem and looking forward for a help from U.
Details: The other day I got a message from Customer that our site is down. I checked the server and application itself and here is what I found:
Exception information: Exception type: OutOfMemoryException Exception message: Exception of type 'System.OutOfMemoryException' was thrown. at System.String.CtorCharArrayStartLength(Char[] value, Int32 startIndex, Int32 length) at Lucene.Net.Index.TermBuffer.ToTerm() at Lucene.Net.Index.SegmentTermEnum.Term() at Lucene.Net.Index.TermInfosReader.Get(Term term, Boolean useCache) at Lucene.Net.Index.SegmentTermDocs.Seek(Term term) at Lucene.Net.Search.MultiTermQueryWrapperFilter.TermGenerator.Generate(MultiTermQuery query, IndexReader reader, TermEnum enumerator) at Lucene.Net.Search.MultiTermQueryWrapperFilter.GetDocIdSet(IndexReader reader) at Lucene.Net.Search.ConstantScoreQuery.ConstantScorer..ctor(ConstantScoreQuery enclosingInstance, Similarity similarity, IndexReader reader, Weight w)...As usual practice - I cleaned App_Data directory and restarted application... in a couple minutes of 100% CPU usage I got the same message with OutOfMemory ....
As quick solution I made ExamineIndex & ExamineSettings as disabled and restarted application - now it alive but without search possibility in backoffice (we don't use Examine in our code at all)
What is the most interesting that we got performance almost 4-5 times faster then it used to be with turned on Examine... and here is my main question - is there any possibility to get Examine back to application - but leave performance as it is now?
Additional details:
P.S. U are welcome with any kind of additional questions ... Thanks in advance!
Hi StasN
I suspect that you might be suffering from this issue - Judging from what you describe above http://issues.umbraco.org/issue/U4-6338
I just had the same issue on a new server today in fact - The main thing is that it's actually a server bug, which happens to affect Umbraco unfortunately.
It's very well described what should be done etc. on the issue itself.
Hope this helps!
/Jan
Hello Jan,
Thank U for your time and advice... My investigation in order:
I think that it's certainly connected to Lucena. Especially when I already had this kind of issue before even with 7.1.4 on 2008 R2 server, with huge amount of nodes in content.
P.S. I feel like it's worth to check updated Umbraco with updated Examine... but again - performance increased considerably...
Hi Jan,
I have faced with the same performance issue caused my ExamineIndex on another server and this time mentioned updates are installed... but I can't see "Application shutdown. Reason: ConfigurationChange" in logs, as in the ticket.
Do U think it's worth to try to install even if not all conditions are satisfied?
Thanks in advance,
Stas
is working on a reply...