Suddenly down with 502 error - appears related to cache activity
Hi All, Our primary site is down, showing a 502 error: Web server received an invalid response while acting as a gateway or proxy server. Azure tech support helped obtain a memory dump, which shows a loop of:
2015-12-16 11:41:26,792 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26
2015-12-16 11:41:26,839 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26
2015-12-16 11:41:26,871 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26
2015-12-16 11:41:26,917 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26
2015-12-16 11:41:26,964 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26
2015-12-16 11:41:26,996 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26
2015-12-16 11:41:27,042 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26
2015-12-16 11:41:27,074 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:27
2015-12-16 11:41:27,121 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:27
2015-12-16 11:41:27,152 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:27
2015-12-16 11:41:27,199 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:27
It appears to be caught in some kind of loop of editing the cache. We haven't changed anything in the past 5-7 days. Most recent change 8 days ago was to update a web api controller to provide data to a page through a javascript call. Previous version of the web api had been working well for several months, but was updated to improve performance. Any help or suggestions would be appreciated. Steve
and many other site URLs truncated and ending in various symbols, most of which are followed by an entry:
/umbraco/surface/noticesurface/setNoticeoff
I saw that there is a database exploit which uses magmi/web/magmi.php, but otherwise have never heard of it. Is it possible the site the victim of some type of exploit or attack? Thanks - Steve
Suddenly down with 502 error - appears related to cache activity
Hi All,
Our primary site is down, showing a 502 error: Web server received an invalid response while acting as a gateway or proxy server.
Azure tech support helped obtain a memory dump, which shows a loop of:
Umbraco.Web.LegacyScheduledTasks.CacheItemRemoved(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)+3c SystemWebni!System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)+a7 SystemWebni!System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)+1c1 SystemWebni!System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)+905 SystemWebni!System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)+74 Umbraco.Web.LegacyScheduledTasks.AddTask(System.String, Int32)+f8 Umbraco.Web.LegacyScheduledTasks.CacheItemRemoved(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)+3c SystemWebni!System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)+a7 SystemWebni!System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)+1c1 SystemWebni!System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)+905 SystemWebni!System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)+74 Umbraco.Web.LegacyScheduledTasks.AddTask(System.String, Int32)+f8 Umbraco.Web.LegacyScheduledTasks.ApplicationStarting(Umbraco.Core.UmbracoApplicationBase, Umbraco.Core.ApplicationContext)+d0 Umbraco.Core.ApplicationEventHandler.OnApplicationStarting(Umbraco.Core.UmbracoApplicationBase, Umbraco.Core.ApplicationContext)+25
And Umbraco log file shows a repeating list of:
2015-12-16 11:41:26,792 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26 2015-12-16 11:41:26,839 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26 2015-12-16 11:41:26,871 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26 2015-12-16 11:41:26,917 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26 2015-12-16 11:41:26,964 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26 2015-12-16 11:41:26,996 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26 2015-12-16 11:41:27,042 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:26 2015-12-16 11:41:27,074 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:27 2015-12-16 11:41:27,121 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:27 2015-12-16 11:41:27,152 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:27 2015-12-16 11:41:27,199 [7] INFO umbraco.BusinessLogic.Log - [Thread 1] Log scrubbed. Removed all items older than 2015-10-17 11:41:27
It appears to be caught in some kind of loop of editing the cache. We haven't changed anything in the past 5-7 days. Most recent change 8 days ago was to update a web api controller to provide data to a page through a javascript call. Previous version of the web api had been working well for several months, but was updated to improve performance.
Any help or suggestions would be appreciated.
Steve
Additionally, there is a series of strange entries in the server event log (from the LogFiles folder at the azure application root), including:
ourdomain.com/reviews/&sa=U&ved=0ahUKEwjl-u62wNbJAhUFXBQKHfEqDW84lgEQFghBMAg&usg=AFQjCNFbAIP3az29iJ_iaG05GpTF5ICIYA/magmi/web/magmi.php
and many other site URLs truncated and ending in various symbols, most of which are followed by an entry:
/umbraco/surface/noticesurface/setNoticeoff
I saw that there is a database exploit which uses magmi/web/magmi.php, but otherwise have never heard of it.
Is it possible the site the victim of some type of exploit or attack?
Thanks - Steve
is working on a reply...