I'm working on project that now has ~50000 nodes and average publish rate is 5-20 nodes per hour from 6 AM till 10 PM. From almost two weeks I'm having big performance issues, the IIS takes 80% of RAM and 80-100% of CPU. All I could find till now is that umbraco.config is being recreated every 30-90 minutes. In log file I found entry that corresponds in time with disappearance of umbraco.config:
2015-12-30 12:56:36,053 [6] INFO Umbraco.Core.Publishing.PublishingStrategy - [P3496/T1/D2] Content 'My Article' with Id '80434' has been published.
2015-12-30 12:56:44,321 [6] INFO umbraco.content - [P3496/T1/D2] Save Xml to file...
2015-12-30 12:56:48,143 [6] ERROR umbraco.content - [P3496/T1/D2] Failed to save Xml to file.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
in System.Text.StringBuilder.ToString()
in umbraco.content.SaveXmlToString(XmlDocument xml)
in umbraco.content.<SaveXmlToFileAsync>d__b.MoveNext()
At first it happend only during big page load, 2-3 page views per second, now it's happening all around the clock. I've already set up cache on main pages, it helped only for few days. Anyone got a clue where to look?
Yes it does, size 40MB. And it stays for about 30-90 minutes, then it disappears for 30-40 minutes and the same thing again, Failed to save Xml to file -> OutOfMemoryException. Sometimes it won't even reach 100% of RAM and still the same info in log files. I've checked if IIS is running x86, it would explain not reaching 100% of RAM but no, it's running x64.
I'm wondering when Umbraco can trigger xml recreate and not write to existing one.
Which version of Umbraco? Some older versions had some memory leaks.
By the way, you can configure IIS to automatically recycle an application pool once the website reaches a certain amount of RAM usage. Not ideal, but could be a workaround for your situation.
Out of interest are you doing anything custom in your code? Nothing that is doing something on any same events? What surprises me is that you are saving one node and not a whole site publish? Is there anything else that could be causing it like lucene?
tags is string and content is node that's being published. When I first added this code, spike in resource consumption was about 2%.
And there's the second event ContentService_Saving used for logging IP adresses.
Nothing that is doing something on any same events? What surprises me is that you are saving one node and not a whole site publish?
I'm saving only one node because there can be up to 3 persons working at the same time on content but on different nodes.
Is there anything else that could be causing it like lucene?
I'm using Lucene for search page. From IIS logs I can see that it's being used 100-200 times per day. This page and Lucene were here almost from the start. Can Lucene indexing be so load heavy?
Have you got the code for those events? I am wondering if its in a foreach loop or something? Can we see the ContentService_Saving code? Where is it logging them to?
Sorry what i meant by "what surprises me" is that if it was a memory leak i would expect you to publishing 30000 nodes not 2. Which makes me think it probably is not that. And the fact Nicholas thinks it has been patched.
It could only be a problem if you were saving content manually i guess.
Do you have anything custom on the save events? Its odd it gone from part time to full time.
For last two days I could turn off my main domain for half an hour and do some test. When it was off the load was almost none... I've tried different publish scenarios and load was acceptable. I'm starting to think that publishing is not the problem.
I don't think its a problem with Umbraco. I do think its related to publishing. My gut is that the code you have in this publish event is trying to save some value in some format to the node and failing, thus giving you the xml error. I will have a look at the code tomorrow. Might be worth trying to get a member of core team to look at it and see if they have come across the issue before.
Could not debug on server but I did some logging, checked times for my code and all were below 1s, during big load it reached 5-8s.
I think I found the problem when started to turn off places where plugins - packages were used. Due to our client requirement my team had to use uCommentsy. When I turned off part that displays comments the server load dropped by 60%, I don't know yet if the recreating of umbraco.config stopped because it can only turned off for few minutes. I'm betting that inside it are old scripts that don't work good with my Umbraco version. Now I'll probably write my own comments mechanism. After farther testing I'll write if it helped.
And everything started to work, problem was with uCommentsy submit form. Made it from scratch and now it works fine. For future my team will replace whole uCommentsy with our own solution. Thanks for help everyone.
That makes sense. I never got on well with that package when I used it. Always used Discus. Glad you got it Fixed. What was it with the package. Did you reinstall or just created a new form?
I've downloaded the source for uCommentsy and started to trace which line could couse it but didn't find it. My best gues is that it opened for each document new session or meybe it looped on something and used all RAM.
The solution was to make my own form and use submit method from uCommentsy.
I've tried Disqus and could not use it on this project because of localizations and some rules that user must confirn that he/she will obey.
umbraco.config keeps recreating itself
I'm working on project that now has ~50000 nodes and average publish rate is 5-20 nodes per hour from 6 AM till 10 PM. From almost two weeks I'm having big performance issues, the IIS takes 80% of RAM and 80-100% of CPU. All I could find till now is that umbraco.config is being recreated every 30-90 minutes. In log file I found entry that corresponds in time with disappearance of umbraco.config:
At first it happend only during big page load, 2-3 page views per second, now it's happening all around the clock. I've already set up cache on main pages, it helped only for few days. Anyone got a clue where to look?
My Umbraco version is 7.2.8
My first thought is to ask how much RAM does your machine have? and is it a 64bit OS?
If you can easily increase the RAM, I'd try that first.
Sorry, forgot about it, its Windows Server 2008 R2 Foundation x64 with 8GB RAM and Intel Xeon processor.
I'm trying to setup a better one but for now 8GB must do...
I wouldn't have thought 8GB would be plenty, I kind of expected you to say 4GB or something smaller.
Does it ever successfully create the XML file?
Yes it does, size 40MB. And it stays for about 30-90 minutes, then it disappears for 30-40 minutes and the same thing again, Failed to save Xml to file -> OutOfMemoryException. Sometimes it won't even reach 100% of RAM and still the same info in log files. I've checked if IIS is running x86, it would explain not reaching 100% of RAM but no, it's running x64.
I'm wondering when Umbraco can trigger xml recreate and not write to existing one.
Which version of Umbraco? Some older versions had some memory leaks.
By the way, you can configure IIS to automatically recycle an application pool once the website reaches a certain amount of RAM usage. Not ideal, but could be a workaround for your situation.
Oops, I see that you actually mentioned the Umbraco version (7.2.8). I thought that one had the memory leak fixes.
Ok, I'll try pool recycling, thanks for tip.
Out of interest are you doing anything custom in your code? Nothing that is doing something on any same events? What surprises me is that you are saving one node and not a whole site publish? Is there anything else that could be causing it like lucene?
Custom code... there is one event ContentService_Published that I use for three things:
Two first were here from the start, third one is about two months old and the most resource greedy thing here is:
tags is string and content is node that's being published. When I first added this code, spike in resource consumption was about 2%.
And there's the second event ContentService_Saving used for logging IP adresses.
I'm saving only one node because there can be up to 3 persons working at the same time on content but on different nodes.
I'm using Lucene for search page. From IIS logs I can see that it's being used 100-200 times per day. This page and Lucene were here almost from the start. Can Lucene indexing be so load heavy?
Have you got the code for those events? I am wondering if its in a foreach loop or something? Can we see the ContentService_Saving code? Where is it logging them to?
Sorry what i meant by "what surprises me" is that if it was a memory leak i would expect you to publishing 30000 nodes not 2. Which makes me think it probably is not that. And the fact Nicholas thinks it has been patched.
It could only be a problem if you were saving content manually i guess.
Do you have anything custom on the save events? Its odd it gone from part time to full time.
This seems to be in the same ball park.
http://stackoverflow.com/questions/31270330/umbraco-log-error-failed-to-load-xml-from-file
Here are the triggers on Saving and Published:
For last two days I could turn off my main domain for half an hour and do some test. When it was off the load was almost none... I've tried different publish scenarios and load was acceptable. I'm starting to think that publishing is not the problem.
I don't think its a problem with Umbraco. I do think its related to publishing. My gut is that the code you have in this publish event is trying to save some value in some format to the node and failing, thus giving you the xml error. I will have a look at the code tomorrow. Might be worth trying to get a member of core team to look at it and see if they have come across the issue before.
Have you debugged the code?
Could not debug on server but I did some logging, checked times for my code and all were below 1s, during big load it reached 5-8s.
I think I found the problem when started to turn off places where plugins - packages were used. Due to our client requirement my team had to use uCommentsy. When I turned off part that displays comments the server load dropped by 60%, I don't know yet if the recreating of umbraco.config stopped because it can only turned off for few minutes. I'm betting that inside it are old scripts that don't work good with my Umbraco version. Now I'll probably write my own comments mechanism. After farther testing I'll write if it helped.
Could it be your content published event ?
I see you are republishing the content item again.
Maybe this is causing the publishing to end up in a loop ?
Dave
Republishing was there from the start, it is of course an error, I will change it to Publishing event, thanks for noticing.
And everything started to work, problem was with uCommentsy submit form. Made it from scratch and now it works fine. For future my team will replace whole uCommentsy with our own solution. Thanks for help everyone.
That makes sense. I never got on well with that package when I used it. Always used Discus. Glad you got it Fixed. What was it with the package. Did you reinstall or just created a new form?
Can you marked this as solved please :)
I've downloaded the source for uCommentsy and started to trace which line could couse it but didn't find it. My best gues is that it opened for each document new session or meybe it looped on something and used all RAM.
The solution was to make my own form and use submit method from uCommentsy.
I've tried Disqus and could not use it on this project because of localizations and some rules that user must confirn that he/she will obey.
Replacing uCommentsy is a good idea. And if you still have the uCommentsy dashboard that you see when you log in enabled, you may want to disable it: https://our.umbraco.org/projects/website-utilities/ucommentsy/general/56336-uCommentsy-Comments-Dashboard-Unusably-Slow
is working on a reply...