Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Anthony Dang 1404 posts 2558 karma points MVP c-trib
    Aug 03, 2017 @ 20:33
    Anthony Dang
    0

    We are getting a weird issue with a live site we inherited. Its running on 7.2.6.

    Every hour on the hour the cpu spikes to max 100% for about 1min making the site unresponsive. The 100% CPU is on the w3wp.exe process on an AWS EC2 virtual machine.

    There is nothing in windows logs / umbraco logs / elmah logs.

    The VM running the site is brand new with nothing else running on it just this website. The VM is running Windows Server 2016 Datacenter.

    We are using SQL Server 2016 in AWS RDS. The instance size is db.m4.xlarge.

  • Nicholas Westby 1895 posts 6334 karma points c-trib
    Aug 03, 2017 @ 20:38
    Nicholas Westby
    0

    A bunch of stuff can cause something like this. This thread has some general guidance: https://our.umbraco.org/forum/using-umbraco-and-getting-started/85487-unexplained-spike-in-iis-server-cpu-activity-nearly-100-causing-outages

    The two easiest things are to do Windows updates and set fcnMode to disabled.

    I've also seen some weird stuff if HTTPS is used by isn't configured exactly right in a number of config files.

  • Anthony Dang 1404 posts 2558 karma points MVP c-trib
    Aug 03, 2017 @ 21:25
    Anthony Dang
    0

    Changing fcnMode to disabled seems like a very drastic thing to do. I've never needed to do that before.

    We have done fcnMode="Single" for one site previously.

    The site does serve a lot of images. I might try fcnMode="Single".

  • Sebastiaan Janssen 4843 posts 14347 karma points MVP admin hq
    Aug 04, 2017 @ 15:14
    Sebastiaan Janssen
    0

    I recommend to always use fcnMode="Single"! That's how Umbraco ships these days.

    See: http://issues.umbraco.org/issue/U4-7712

    https://shazwazza.com/post/all-about-aspnet-file-change-notification-fcn/

  • Anthony Dang 1404 posts 2558 karma points MVP c-trib
    Aug 03, 2017 @ 22:14
    Anthony Dang
    0

    Also, just wondering what could be causing the CPU spike on the hour?

    It's like clock work. eg. 12pm, 1pm, 2pm.

  • Nicholas Westby 1895 posts 6334 karma points c-trib
    Aug 03, 2017 @ 22:31
    Nicholas Westby
    0

    Off the top of my head, a scheduled task that happens to be hourly or maybe if IIS were for some reason configured to recycle the application pool every hour (the default is every 29 hours). Could also be if IIS were configured to restart if there are some number of errors in an hour (I think the default is to check for some number of errors in a 5-minute timespan). Could also be some external tool (e.g., a heartbeat/ping tool, such as New Relic) that is hitting a specific page that has an error that causes the site to restart.

    Probably a bunch of other scenarios.

  • Anthony Dang 1404 posts 2558 karma points MVP c-trib
    Aug 04, 2017 @ 09:11
    Anthony Dang
    0

    IIS is just default settings, so yes it's the 29hr app restart.

    There are no external tools on the server. It's all fresh out of the box. Strange.

    Turns out that we already had fcnMode="Single".

  • Anthony Dang 1404 posts 2558 karma points MVP c-trib
    Aug 04, 2017 @ 12:54
    Anthony Dang
    0

    Update:

    fcnMode="Disabled" didnt work.

  • Anthony Dang 1404 posts 2558 karma points MVP c-trib
    Aug 04, 2017 @ 14:35
    Anthony Dang
    1

    Update:

    So we found the problem. The site is experiencing an hourly DDOS attack.

    The thing that threw us off was that it wasnt happening every hour.

  • Nicholas Westby 1895 posts 6334 karma points c-trib
    Aug 04, 2017 @ 15:07
    Nicholas Westby
    0

    Glad you figured it out! FYI, Cloudflare or another WAF can help to avoid DDOS attacks.

    BTW, you might want to confirm that it's a DDOS attack and not an overly aggressive search engine. I've had search engines cause me headaches in the past.

Please Sign in or register to post replies

Write your reply to:

Draft