Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Nicholas Westby 1912 posts 6371 karma points c-trib
    Feb 17, 2015 @ 18:54
    Nicholas Westby
    0

    Memory Climbs 2GB in 11 Hours in Umbraco 7.1.8

    Here's a graph from New Relic showing that the w3wp.exe process (IIS) is climbing from 0GB to 2GB in about 11 hours: enter image description here

    This is on a staging server with something like 5 users (probably more like 2-3 active users at a time). They are probably spending most of their time in Umbraco publishing things. Any idea of how exactly I could find the cause of this?

    I recall that the PetaPoco memory fix was supposed to fix some of this issue in a previous Umbraco version: http://umbraco.com/follow-us/blog-archive/2014/10/2/umbraco-717-and-623-saving-you-loads-of-memory

    Note that I'm not using any caching, aside from some minimal caching in static variables. I'm not using macro caching and I'm not using donut caching.

    Note that this staging server has only recently gone live. Here's a graph of memory usage over multiple days: enter image description here

    That first low spot is before the site was being used. The short dip in the middle is probably overnight when nobody was using it. And the large low spot to the right is over a weekend. The memory goes back down whenever the application pool restarts (which is set to happen periodically... the default for IIS). The high memory usage in the most recent time period is probably higher than the other two because there are more active users of the CMS (2-3 instead of 1-2).

    My workaround for the time being is to have IIS automatically restart the app pool whenever the memory approaches 1.5GB, though that is not ideal.

  • Nicholas Westby 1912 posts 6371 karma points c-trib
    Feb 17, 2015 @ 20:37
    Nicholas Westby
    0

    Just recycled the app pool a couple hours ago, and now the CPU is pegged. Can't publish anything and everything around the site is super slow.

    Any ideas? Maybe a locking issue that has been fixed in a newer version of Umbraco? Or something of that sort. Here are some recent items in the IIS log (scrubbed of sensitive info):

    2015-02-17 19:28:22 10.0.0.98 GET /umbraco/backoffice/UmbracoApi/Entity/GetByIds ids=7895&type=Media 80 redactedUser 1.2.3.4 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:35.0)+Gecko/20100101+Firefox/35.0 http://staging.redacted.com/umbraco 200 0 0 1177
    2015-02-17 19:28:22 10.0.0.98 GET /umbraco/backoffice/UmbracoApi/Entity/GetByIds ids=7891&type=Media 80 redactedUser 1.2.3.4 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:35.0)+Gecko/20100101+Firefox/35.0 http://staging.redacted.com/umbraco 200 0 0 1177
    2015-02-17 19:28:22 10.0.0.98 GET /umbraco/backoffice/UmbracoApi/Entity/GetByIds ids=7896&type=Media 80 redactedUser 1.2.3.4 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:35.0)+Gecko/20100101+Firefox/35.0 http://staging.redacted.com/umbraco 200 0 0 1177
    2015-02-17 19:28:26 10.0.0.98 GET /umbraco/backoffice/UmbracoApi/Images/GetBigThumbnail originalImagePath=%2Fmedia%2F3924%2Fsomeimage.png 80 redactedUser 1.2.3.4 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:35.0)+Gecko/20100101+Firefox/35.0 http://staging.redacted.com/umbraco 200 0 0 4400
    2015-02-17 19:28:30 10.0.0.98 GET /umbraco/backoffice/UmbracoApi/Images/GetBigThumbnail originalImagePath=%2Fmedia%2F3926%2Fsomeimage-2.png 80 redactedUser 1.2.3.4 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:35.0)+Gecko/20100101+Firefox/35.0 http://staging.redacted.com/umbraco 200 0 0 7376
    2015-02-17 19:28:30 10.0.0.98 GET /umbraco/backoffice/UmbracoApi/Images/GetBigThumbnail originalImagePath=%2Fmedia%2F3925%2Fsomeimage-3.png 80 redactedUser 1.2.3.4 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:35.0)+Gecko/20100101+Firefox/35.0 http://staging.redacted.com/umbraco 200 0 0 7563
    2015-02-17 19:28:30 10.0.0.98 GET /umbraco/backoffice/UmbracoApi/Images/GetBigThumbnail originalImagePath=%2Fmedia%2F3907%2Fsome-image-4.png 80 redactedUser 1.2.3.4 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:35.0)+Gecko/20100101+Firefox/35.0 http://staging.redacted.com/umbraco 200 0 0 8347
    2015-02-17 19:28:34 10.0.0.98 GET /umbraco/backoffice/UmbracoApi/Images/GetBigThumbnail originalImagePath=%2Fmedia%2F3921%2Fsome-image-5.png 80 redactedUser 1.2.3.4 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:35.0)+Gecko/20100101+Firefox/35.0 http://staging.redacted.com/umbraco 200 0 0 11231
    2015-02-17 19:28:36 10.0.0.98 GET /umbraco/backoffice/UmbracoApi/Authentication/GetRemainingTimeoutSeconds - 80 redactedUser2 2.3.4.5 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_9_5)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/40.0.2214.111+Safari/537.36 http://staging.redacted.com/umbraco 200 0 0 942
    

    Notice that it's taking up to 10 seconds to do very simple operations. I could fix this with an application pool restart, but I'm sure the issue would just resurface again.

  • Nicholas Westby 1912 posts 6371 karma points c-trib
    Feb 17, 2015 @ 20:56
    Nicholas Westby
    0

    I wonder if it has to do with the fact that I changed a password recently (there is a bug in my version of Umbraco that requires the app pool to restart before the new password will take effect). I did a New Relic thread profile and here are a couple hot spots:

    enter image description here

    Seems like there is some thread locking happening and some weirdness with the handling of users.

  • Nicholas Westby 1912 posts 6371 karma points c-trib
    Feb 19, 2015 @ 01:06
    Nicholas Westby
    0

    I'm thinking the memory may be a non-issue (perhaps just IIS avoiding garbage collection when it's not necessary). However, the CPU issue is still a problem.

    I'm trying to profile the CPU with dotTrace, but keep running into issues (the latest being an apparent incompatibility between dotTrace and New Relic). Will update if I figure anything out, but if anybody has dealt with this before, I'd like to hear about your experience.

  • Shannon Deminick 1494 posts 5045 karma points hq
    Mar 03, 2015 @ 07:58
    Shannon Deminick
    0

    For memory, if you use Ants memory profiler it will tell you the type of memory used. Depending on how much memory your server has will depend on how IIS feels like hanging on to memory. For example, I have 16GB of RAM on my machine and for some larger sites when I debug, ANTS might show me something like this:

    ants memory profiling

    Notice that the vast majority is just empty/unused/reserved memory. The managed memory is actually only consuming 120mb

    ANTs was the only mem profiler that seem to be able to give me this info when I was debugging

  • Nicholas Westby 1912 posts 6371 karma points c-trib
    Mar 03, 2015 @ 08:03
    Nicholas Westby
    0

    Yeah, IIRC that's basically what dotMemory said. dotMemory also has a garbage collection feature, which when I tried it reduced the amount of memory used.

    I was only worried about the memory because I was thinking it may be causing the CPU issue, but now I'm pretty sure that's not the case.

    For anybody interested, here's the YouTrack issue for the CPU issue: http://issues.umbraco.org/issue/U4-6292

  • Shannon Deminick 1494 posts 5045 karma points hq
    Mar 03, 2015 @ 08:10
    Shannon Deminick
    1

    yup debugging that now

  • Nicholas Westby 1912 posts 6371 karma points c-trib
    Mar 03, 2015 @ 08:13
    Nicholas Westby
    0

    Cool, let me know if I can help in some way. I'm off to be now, but I'll be deploying that site affected by the issue when I wake up.

  • Nicholas Westby 1912 posts 6371 karma points c-trib
    Apr 11, 2015 @ 05:01
    Nicholas Westby
    100

    For anybody curious, I believe the CPU issue has likely been resolved. Haven't tested it thoroughly, but I did upgrade to Umbraco 7.2.4 and haven't seen the issue since. Refer to this issue for the cause/resolution (basically, a bug with the change tracking when multiple people were logged in with the same user): http://issues.umbraco.org/issue/U4-6292

    Again, the memory issue is not really a problem. Just IIS avoiding garbage collection as much as possible (for performance reasons, from what I gather).

Please Sign in or register to post replies

Write your reply to:

Draft