Small installation with almost no traffic (100 nodes, 20 users per day)
I'm experiencing the following issue on all of my websites deployed on Azure in shared mode (512MB RAM / hour). What happens is that every two-three days each site gets blocked because of an insufficient amount of memory. I've checked the monitor and have seen that the memory working set is constantly rising until it reaches the threshold and the entire site gets blocked for several minutes. (purple line on attached image of the monitor screenshot).
I've also checked the log files and found that every minute the following entries are being added:
I was looking for a Umbraco setting to disable this keep-alive task which is clearly running in the background but was unable to find one. ScheduledPublish/Index seems like something related to load balancing / distributed servers or is that the correct behavior of Umbraco?
I've investigated several possible reasons mentioned on this forum related to high memory usage:
Document and Content
App_Code
Linq
Preview function
XmlCacheEnabled
UmbracoLibraryCacheDuration
Nothing seems to help this, can anyone point me into the right direction? What else should I investigate in order to track this issue?
I'm still seeing this. Umbraco 7.2.1.
Around 400 nodes on the entire app. I'm the only user. Azure Websites Shared instance. Reaching 300megs as we speak.
Please keep an eye on the memory to see if it is actually a memory leak. Memory will always expand because IIS will optimize how it wants to manage memory allocations. 300mb isn't a significant amount of mem usage. If mem consumption grows per request, this is also normal, however if it keeps growing consistently until the app crashes, then that is a memory leak. Also note that memory leaks can easily be caused by any code - user code, packages, etc... so if you have determined that this is in fact a memory leak then the first thing is to determine what is causing the leak. Determining the cause is generally fairly easy so long as you can replicate the issue locally.
Right now it's static but every requests brings it up by a few megs. It's already at 350megs. I understand that IIS may do whatever it wants to manage the app memory. The problem is, once you go over 512, Azure suspends your app. It's not that your app crashed. It was still working. Azure shut it down because you went over quota. Also, I googled around to see what was an average memory consumption for this size of app (roughly 400 nodes in total) and it's never more than 200megs. My app is quite simple. 4 simple plugins and one Web API controller that I built. I profiled the process and came up with nothing strange. I'm no profiling expert so maybe I missed something.
PS the same behavior repeats itself on another app I have but this one is on the Basic azure plan so there's no quota limit.
Ok, well if Azure is shutting you down then that is probably not right since IIS won't consume empty memory that it isn't allowed to.
Can you replicate this memory consumption on your local machine ? If so, and if you are willing to share your project with me, you could email it over with steps to repliciate and I can run a profiler on it.
Alternatively (and maybe even better), is if you'd be willing to share a memory dump from your Azure website. I think there's a few ways in which you can create a dump file from Azure if you google around.
Another question: On the azure instance that you don't have a quota limit, does the memory keep rising or does it eventually stabilize?
Yes I can send you the project no problem. I'll do it tonight or tomorrow.
I can replicate this in my own IIS. I'll send you the Azure memory dump once I figure out how to do it :)
I just looked at the other Umbraco. It is now on 600megs and goes up on every request. However, I just noticed this one is 7.0.1 so I'll update it at some point this week and update you. But, both are behaving exactly the same so I doubt it's the Poco bug.
High Memory - Azure Websites
Hi All,
This is my first post so please go easy on me :)
Current setup:
I'm experiencing the following issue on all of my websites deployed on Azure in shared mode (512MB RAM / hour). What happens is that every two-three days each site gets blocked because of an insufficient amount of memory. I've checked the monitor and have seen that the memory working set is constantly rising until it reaches the threshold and the entire site gets blocked for several minutes. (purple line on attached image of the monitor screenshot).
I've also checked the log files and found that every minute the following entries are being added:
2014-09-22 22:41:45 HOMEPAGE GET /umbraco/ping.aspx 200 0 0 1207 527 62
2014-09-22 22:42:46 HOMEPAGE POST /umbraco/RestServices/ScheduledPublish/Index 200 0 0 1222 1229 359
2014-09-22 22:43:46 HOMEPAGE POST /umbraco/RestServices/ScheduledPublish/Index 200 0 0 1222 1229 109
2014-09-22 22:44:47 HOMEPAGE POST /umbraco/RestServices/ScheduledPublish/Index 200 0 0 1222 1229 671
2014-09-22 22:45:46 HOMEPAGE POST /umbraco/RestServices/ScheduledPublish/Index 200 0 0 1222 1229 156
2014-09-22 22:46:46 HOMEPAGE POST /umbraco/RestServices/ScheduledPublish/Index 200 0 0 1222 1229 124
2014-09-22 22:46:46 HOMEPAGE GET /umbraco/ping.aspx 200 0 0 1207 527 15
2014-09-22 22:47:45 HOMEPAGE POST /umbraco/RestServices/ScheduledPublish/Index 200 0 0 1222 1229 234
2014-09-22 22:48:45 HOMEPAGE POST /umbraco/RestServices/ScheduledPublish/Index 200 0 0 1222 1229 171
2014-09-22 22:49:45 HOMEPAGE POST /umbraco/RestServices/ScheduledPublish/Index 200 0 0 1222 1229 203
2014-09-22 22:50:45 HOMEPAGE POST /umbraco/RestServices/ScheduledPublish/Index 200 0 0 1222 1229 124
2014-09-22 22:51:46 HOMEPAGE POST /umbraco/RestServices/ScheduledPublish/Index 200 0 0 1222 1229 328
I was looking for a Umbraco setting to disable this keep-alive task which is clearly running in the background but was unable to find one. ScheduledPublish/Index seems like something related to load balancing / distributed servers or is that the correct behavior of Umbraco?
I've investigated several possible reasons mentioned on this forum related to high memory usage:
Nothing seems to help this, can anyone point me into the right direction? What else should I investigate in order to track this issue?
Any help will be highly appropriated,
Thx
I have exactly the same behaviour, also in shared site in Azure. The memory goes high and kills my website.
Does this mean we can't use the current version in Azure?
bleedow, have you found any answer?
The issue url http://issues.umbraco.org/issue/U4-5553 - core team is looking at it. Vote to help give it higher priority.
( I wouldn't say we can't use the current version for this reason. If necessary I would restart the site at periods (or on some limit). )
I think today's 7.1.7 release should fix this issue; We identified and fixed a memory leak in PetaPoco: http://umbra.co/1E4RFPH
I'm still seeing this. Umbraco 7.2.1. Around 400 nodes on the entire app. I'm the only user. Azure Websites Shared instance. Reaching 300megs as we speak.
Ideas?
Hi Eyal!
Does it keep growing? Since we released 7.1.7 we haven't seen memory leaks on the 500+ sites we run on umbraco.com - Umbraco as a Service.
Best, Niels...
Please keep an eye on the memory to see if it is actually a memory leak. Memory will always expand because IIS will optimize how it wants to manage memory allocations. 300mb isn't a significant amount of mem usage. If mem consumption grows per request, this is also normal, however if it keeps growing consistently until the app crashes, then that is a memory leak. Also note that memory leaks can easily be caused by any code - user code, packages, etc... so if you have determined that this is in fact a memory leak then the first thing is to determine what is causing the leak. Determining the cause is generally fairly easy so long as you can replicate the issue locally.
Hi,
Right now it's static but every requests brings it up by a few megs. It's already at 350megs. I understand that IIS may do whatever it wants to manage the app memory. The problem is, once you go over 512, Azure suspends your app. It's not that your app crashed. It was still working. Azure shut it down because you went over quota. Also, I googled around to see what was an average memory consumption for this size of app (roughly 400 nodes in total) and it's never more than 200megs. My app is quite simple. 4 simple plugins and one Web API controller that I built. I profiled the process and came up with nothing strange. I'm no profiling expert so maybe I missed something.
PS the same behavior repeats itself on another app I have but this one is on the Basic azure plan so there's no quota limit.
Eyal
Ok, well if Azure is shutting you down then that is probably not right since IIS won't consume empty memory that it isn't allowed to.
Can you replicate this memory consumption on your local machine ? If so, and if you are willing to share your project with me, you could email it over with steps to repliciate and I can run a profiler on it.
Alternatively (and maybe even better), is if you'd be willing to share a memory dump from your Azure website. I think there's a few ways in which you can create a dump file from Azure if you google around.
Another question: On the azure instance that you don't have a quota limit, does the memory keep rising or does it eventually stabilize?
My email addy is on this page: http://umbraco.com/about-us/team
Yes I can send you the project no problem. I'll do it tonight or tomorrow.
I can replicate this in my own IIS. I'll send you the Azure memory dump once I figure out how to do it :)
I just looked at the other Umbraco. It is now on 600megs and goes up on every request. However, I just noticed this one is 7.0.1 so I'll update it at some point this week and update you. But, both are behaving exactly the same so I doubt it's the Poco bug.
Thanks got all the help! E
Have you found the solution yet. I also have the same issue. I am using 7.2.2. It takes a lot of memory and sometimes CPU at 100%.
Do you know what "ScheduledPublish/Index" calling means. ? Or can we disable it somehow
I am encountering this, iis service keeps growing till halt; on 7.1.8
can you please share what was the remedy, how did you solve it?
It was never really resolved. I switch to Standard mode and the memory consumption dropped almost completely. This is all very weird.
If you are able to upgrade to at least 7.2.8. It will fix the problem. I had the same issue on 7.2.2 but 7.2.8 fixed this annoying.
And in IIS if you switch your App to 32 bit mode instead of 64 bit. It will use less memory but performance is still good enough.
is working on a reply...