Last week I ran into an interesting edge case error!
We have an Umbraco website, with a very large amount of media. Due to an Umbraco bug related to running migrations, the website failed to recycle. Once the website had failed, it could not be restarted, until we had both cleared the migrations and also disabled AzureCdnToolkit.
I'm not exactly sure why AzureCdnToolkit was erroring, but I wondered if the errors had been cached at application level, or in the CDN?
The errors would loop, and then the application would crash and recycle.
Thought I'd share, as it might be useful ;-) Cheers! Laurie
2016-11-18 09:43:48,450 [P10096/D2/T1] INFO Umbraco.Core.CoreBootManager - Umbraco 7.5.4 application starting on RD0003FF59D32E
2016-11-18 09:43:48,591 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Determining hash of code files on disk
2016-11-18 09:43:48,654 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Hash determined (took 65ms)
2016-11-18 09:43:48,919 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of umbraco.interfaces.IApplicationStartupHandler
2016-11-18 09:43:48,986 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of umbraco.interfaces.IApplicationStartupHandler, found 0 (took 56ms)
2016-11-18 09:43:49,076 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.PropertyEditors.IPropertyEditorValueConverter
2016-11-18 09:43:49,091 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.PropertyEditors.IPropertyEditorValueConverter, found 0 (took 1ms)
2016-11-18 09:43:49,091 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.PropertyEditors.IPropertyValueConverter
2016-11-18 09:43:49,091 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.PropertyEditors.IPropertyValueConverter, found 0 (took 1ms)
2016-11-18 09:43:49,169 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Web.Mvc.SurfaceController
2016-11-18 09:43:49,185 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Web.Mvc.SurfaceController, found 0 (took 1ms)
2016-11-18 09:43:49,185 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Web.WebApi.UmbracoApiController
2016-11-18 09:43:49,200 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Web.WebApi.UmbracoApiController, found 0 (took 20ms)
2016-11-18 09:43:49,482 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.Media.IThumbnailProvider
2016-11-18 09:43:49,499 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.Media.IThumbnailProvider, found 0 (took 7ms)
2016-11-18 09:43:49,499 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.Media.IImageUrlProvider
2016-11-18 09:43:49,499 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.Media.IImageUrlProvider, found 0 (took 0ms)
2016-11-18 09:43:55,187 [P10096/D2/T1] INFO Umbraco.Core.DatabaseContext - CanConnect = True
2016-11-18 09:43:56,482 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.Models.PublishedContent.PublishedContentModel
2016-11-18 09:43:56,482 [P10096/D2/T1] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.Models.PublishedContent.PublishedContentModel, found 0 (took 8ms)
2016-11-18 09:43:58,685 [P10096/D2/T1] INFO Umbraco.Core.DatabaseContext - CanConnect = True
2016-11-18 09:43:58,685 [P10096/D2/T1] INFO Umbraco.Core.DatabaseContext - CanConnect = True
2016-11-18 09:43:58,796 [P10096/D2/T1] INFO Umbraco.Web.Cache.CacheRefresherEventHandler - Initializing Umbraco internal event handlers for cache refreshing
2016-11-18 09:43:59,657 [P10096/D2/T1] INFO UmbracoExamine.LocalStorage.LocalTempStorageIndexer - Successfully sync'd main index to local temp storage for index: ~/App_Data/TEMP/ExamineIndexes/RD0003FF59D32E/Internal/
2016-11-18 09:43:59,798 [P10096/D2/T1] INFO UmbracoExamine.LocalStorage.LocalTempStorageIndexer - Successfully sync'd main index to local temp storage for index: ~/App_Data/TEMP/ExamineIndexes/RD0003FF59D32E/InternalMember/
2016-11-18 09:43:59,860 [P10096/D2/T1] INFO Umbraco.Web.Search.ExamineEvents - Initializing Examine and binding to business logic events
2016-11-18 09:43:59,860 [P10096/D2/T1] INFO Umbraco.Web.Search.ExamineEvents - Adding examine event handlers for index providers: 3
2016-11-18 09:44:00,032 [P10096/D2/T1] INFO Umbraco.Core.CoreBootManager - Umbraco application startup complete (took 12118ms)
2016-11-18 09:44:01,141 [P10096/D2/T8] INFO Umbraco.Core.Sync.ApplicationUrlHelper - ApplicationUrl: http://www.-removed-s.com/umbraco (using scheduledTasks/@baseUrl)
2016-11-18 09:44:04,272 [P10096/D2/T8] INFO umbraco.content - Load Xml from file...
2016-11-18 09:44:04,506 [P10096/D2/T8] INFO umbraco.content - Loaded Xml from file.
2016-11-18 09:44:08,558 [P10096/D2/T8] INFO Umbraco.Core.PluginManager - Starting resolution types of Umbraco.Core.PropertyEditors.IParameterEditor
2016-11-18 09:44:08,558 [P10096/D2/T8] INFO Umbraco.Core.PluginManager - Completed resolution of types of Umbraco.Core.PropertyEditors.IParameterEditor, found 0 (took 4ms)
2016-11-18 09:45:04,860 [P10096/D2/T15] INFO umbraco.BusinessLogic.Log - Log scrubbed. Removed all items older than 2016-11-17 09:45:04
2016-11-18 09:46:44,849 [P10096/D2/T12] ERROR Umbraco.Web.Scheduling.KeepAlive - Failed (at "http://www.-removed-s.com/umbraco").
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Umbraco.Web.Scheduling.KeepAlive.<PerformRunAsync>d__4.MoveNext()
2016-11-18 09:46:45,318 [P10096/D2/T12] ERROR Umbraco.Web.Scheduling.ScheduledPublishing - Failed (at "http://www.-removed-s.com/umbraco").
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Umbraco.Web.Scheduling.ScheduledPublishing.<PerformRunAsync>d__7.MoveNext()
2016-11-18 09:49:25,336 [P10096/D2/T17] ERROR Umbraco.Web.Scheduling.ScheduledPublishing - Failed (at "http://www.-removed-s.com/umbraco").
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Umbraco.Web.Scheduling.ScheduledPublishing.<PerformRunAsync>d__7.MoveNext()
2016-11-18 09:51:30,608 [P10096/D2/T22] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [LogScrubber] Terminating
2016-11-18 09:51:30,608 [P10096/D2/T22] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [LogScrubber] Waiting for tasks to complete
2016-11-18 09:51:30,624 [P10096/D2/T22] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [LogScrubber] Tasks completed, terminated
2016-11-18 09:51:30,749 [P10096/D2/T22] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [KeepAlive] Terminating
2016-11-18 09:51:30,749 [P10096/D2/T22] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [KeepAlive] Waiting for tasks to complete
2016-11-18 09:51:30,749 [P10096/D2/T22] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledTasks] Terminating
2016-11-18 09:51:30,749 [P10096/D2/T22] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledTasks] Waiting for tasks to complete
2016-11-18 09:51:30,749 [P10096/D2/T19] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [KeepAlive] Tasks completed, terminated
2016-11-18 09:51:30,749 [P10096/D2/T12] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledTasks] Tasks completed, terminated
2016-11-18 09:51:30,780 [P10096/D2/T22] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledPublishing] Terminating
2016-11-18 09:51:30,780 [P10096/D2/T22] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledPublishing] Waiting for tasks to complete
2016-11-18 09:51:30,780 [P10096/D2/T22] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [Umbraco.Web.Scheduling.IBackgroundTask] Terminating
2016-11-18 09:51:30,780 [P10096/D2/T22] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [Umbraco.Web.Scheduling.IBackgroundTask] Waiting for tasks to complete
2016-11-18 09:51:30,780 [P10096/D2/T18] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [Umbraco.Web.Scheduling.IBackgroundTask] Tasks completed, terminated
2016-11-18 09:51:40,526 [P10096/D2/T8] ERROR Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions - Error resolving media url from the CDN
System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponse()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.<>c__DisplayClass6_1.<UrlToCdnUrl>b__0()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.TryFiveTimes(Action delegateAction)
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.UrlToCdnUrl(String cropUrl, Boolean htmlEncode, String currentDomain)
2016-11-18 09:51:40,583 [P10096/D2/T17] INFO Umbraco.Web.Scheduling.BackgroundTaskRunner - [ScheduledPublishing] Tasks completed, terminated
2016-11-18 09:51:40,771 [P10096/D2/T8] ERROR Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions - Error resolving media url from the CDN
System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponse()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.<>c__DisplayClass6_1.<UrlToCdnUrl>b__0()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.TryFiveTimes(Action delegateAction)
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.UrlToCdnUrl(String cropUrl, Boolean htmlEncode, String currentDomain)
2016-11-18 09:51:40,850 [P10096/D2/T8] ERROR Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions - Error resolving media url from the CDN
System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponse()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.<>c__DisplayClass6_1.<UrlToCdnUrl>b__0()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.TryFiveTimes(Action delegateAction)
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.UrlToCdnUrl(String cropUrl, Boolean htmlEncode, String currentDomain)
2016-11-18 09:51:40,880 [P10096/D2/T8] ERROR Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions - Error resolving media url from the CDN
System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponse()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.<>c__DisplayClass6_1.<UrlToCdnUrl>b__0()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.TryFiveTimes(Action delegateAction)
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.UrlToCdnUrl(String cropUrl, Boolean htmlEncode, String currentDomain)
2016-11-18 09:51:40,896 [P10096/D2/T8] ERROR Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions - Error resolving media url from the CDN
System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponse()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.<>c__DisplayClass6_1.<UrlToCdnUrl>b__0()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.TryFiveTimes(Action delegateAction)
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.UrlToCdnUrl(String cropUrl, Boolean htmlEncode, String currentDomain)
2016-11-18 09:51:40,912 [P10096/D2/T8] ERROR Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions - Error resolving media url from the CDN
System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponse()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.<>c__DisplayClass6_1.<UrlToCdnUrl>b__0()
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.TryFiveTimes(Action delegateAction)
at Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions.UrlToCdnUrl(String cropUrl, Boolean htmlEncode, String currentDomain)
2016-11-18 09:51:40,927 [P10096/D2/T8] ERROR Our.Umbraco.AzureCDNToolkit.UrlHelperRenderExtensions - Error resolving media url from the CDN
System.Net.WebException: The remote server returned an error: (403) Forbidden.
This is really interesting as I have seen this happen once recently too. I have have a ongoing support ticket with Azure about it as the route of the problem appears to be that randomly my Web App (and yours) appear to not be able to resolve a azureedge.net domain but it seems to be perfectly healthy externally.
If you still have the issue, it's likely that restarting the Web App will resolve it.
I'm thinking about adding a error count type system so that the toolkit will disable resolution if it's receiving 400/500 status codes from the CDN until the count comes back down or something like that.
Hello Jeavon,
We are getting this issue too. The interesting part is that restarting the app did not solve the issue. We have to restart it more than one and it starts to answer requests. This happens quite frequent (almost every week) we consider to not use the toolkit until we found the answer to this issue.
I'm working with:
No it doesn't matter although I would always recommend that you are using the latest ImageProcessor.
I don't know why you are seeing localhost requests, the requests should always be using the host specified in the AzureCDNToolkit:Domain app setting...
Application Startup Errors
Hey Jeavon,
Hope you've had a good week.
Last week I ran into an interesting edge case error!
We have an Umbraco website, with a very large amount of media. Due to an Umbraco bug related to running migrations, the website failed to recycle. Once the website had failed, it could not be restarted, until we had both cleared the migrations and also disabled AzureCdnToolkit.
I'm not exactly sure why AzureCdnToolkit was erroring, but I wondered if the errors had been cached at application level, or in the CDN?
The errors would loop, and then the application would crash and recycle.
Thought I'd share, as it might be useful ;-) Cheers! Laurie
Thanks Laurence,
This is really interesting as I have seen this happen once recently too. I have have a ongoing support ticket with Azure about it as the route of the problem appears to be that randomly my Web App (and yours) appear to not be able to resolve a azureedge.net domain but it seems to be perfectly healthy externally.
If you still have the issue, it's likely that restarting the Web App will resolve it.
I'm thinking about adding a error count type system so that the toolkit will disable resolution if it's receiving 400/500 status codes from the CDN until the count comes back down or something like that.
I'll keep you posted.
Jeavon
Hello Jeavon, We are getting this issue too. The interesting part is that restarting the app did not solve the issue. We have to restart it more than one and it starts to answer requests. This happens quite frequent (almost every week) we consider to not use the toolkit until we found the answer to this issue. I'm working with:
You might want to try the pre-release, we recently merged a PR that solves a issue when crops are being requested that don't exist. See https://github.com/CrumpledDog/Umbraco-AzureCDNToolkit/pull/14
This doesn't matter what version of ImageProcessor or ImageResizer I'm using?
Because it keeps me sending Exceptions to AppInsights like this:
Why the media element is requested to localhost? Is that correct? Have you used stress tests on sites that uses this library?
I ask because some other requests have the correct domain on it.
Thank you for your time.
No it doesn't matter although I would always recommend that you are using the latest ImageProcessor.
I don't know why you are seeing localhost requests, the requests should always be using the host specified in the
AzureCDNToolkit:Domain
app setting...Hello Jeavon, this really concerns me because this are part of my configs for the toolkit:
but if you can see on dependencies some comes from the Domain but most of them comes from localhost :
I really don't know what happen here or why it keeps requesting from localhost.
Can be the app config Domain be override from the webconfig or some other place?
Greetings.
Is it possible that you could be (re)setting the value somewhere else? It might be something like this:
Negative, in all code the only place that is in we web.config with a default value and in the app settings on azure.
We have this in one of our clients sites and it brought the site down. Restarting the web app didn't resolve it.
is working on a reply...