Homepage regularly hangs after error from ImageProcessor
Hello all. First time posting, so thank you first to all who've come before. I've solved a lot of issues from reading these forums. This, however I cannot...
Our site was previously built and hosted in Umbraco v7.14. I recently rebuilt the site fresh in v8.5 as it was such a large change between versions. As well as created a new database for it.
It has two cultures: gb & au
Articulate v4.1.1 & Umbraco Forms v8.2.0 (both installed with NuGet)
Built within Visual Studio v16.4.4.
.NET v4.7.2
IIS v10.0.14393.0
Windows server 2016
All was fine for the first few weeks after the new site went up, now I keep receiving these errors in the backend that line up with the homepage (for both cultures) hanging until I sign in and republish the homepage node (all other pages on the site display correctly, it just seems to be affecting the home page).
Example:
ImageProcessor.Common.Exceptions.ImageProcessingException: ProcessImageAsync 610 : No image exists at E:\[root]\media\1491\image01.jpg
The “[root]” is just me replacing the physical path on the server for this post.
They’re all image path related errors. The links are still trying to access the images with the old Umbraco way of numbered folders inside the media folder, rather than the new way of random characters.
Each image mentioned does actually exist in the site, however they were all manually re-inputted when the v8 rebuild happened, so their paths in the backend are correct and display correctly in the front-end.
Example:
ImageProcessor.Common.Exceptions.ImageProcessingException: ProcessImageAsync 610 : No image exists at E:\[root]\media\1hzdc0pk\image01.jpg
I’ve refreshed my server cache as well as my Umbraco caches, so I have no idea where it could be coming from. Any thoughts?
Thanks for the response Thomas. I hadn't thought of that. I'll give that idea a go in IIS and see if it can help stop these errors from appearing.
I've also just tried actually giving (Google, or whatever's looking for the old images) the images themselves... I got fed up trying to find the solution to the problem this morning, so re-uploaded all of the old images in their old folders (in addition to their newly uploaded Umb8 counterparts) to sit there and be found by whatever process is looking for them. Haven't had an image path related error since...Although I did get a new one not long after:
ImageProcessor.Common.Exceptions.ImageProcessingException: ProcessImageAsync 559 : The request /media/1521/image01.jpg?crop=0 could not be understood by the server due to malformed syntax.
Still, I'd prefer to clean the old images out, so I'll give your suggestion a try.
Update:
After uploading all of the old images to their old folders and implementing the 404 rule in IIS, we went for about 10 hours without an error (which is a record right now!). However around 1am this morning we got:
ImageProcessor.Common.Exceptions.ImageProcessingException: ProcessImageAsync 610 : No image exists at E:\[root]\media\1506\lendingmetrics-takes-home-two-awards-at-the-lending-awards-2019.jpg
Which unfortunately left the homepage hanging until I got into work this morning.
Here's the rule I wrote in the web.config based on your advice Thomas:
Provided it's written correctly, that must mean that something else other than Google is calling to the images. Maybe something from my database?
As a note - this old image that the process was looking for wasn't in that folder - so I've now uploaded it to it. Sort of "plugging holes in a sinking ship" method...Not the best, but there weren't any more errors related to images that had been uploaded yesterday.
I didn't actually... I tested the pattern within the IIS "Test Pattern..." and it came out correctly, so I didn't think to test the problematic path in the browser itself yesterday to verify.
Just went to do it now and saw that it didn't actually kick to the 404 (have also now changed it to a Redirect - was a Rewrite before). Also had to remove the leading / to get it to work.
Testing further - Once corrected, it did work for the old image link:
(kicked correctly to the 404), however checking through the rest of the site with the rule enabled, it also blocked a number of the newly uploaded (correct) Umbraco 8 images, so I just disabled it.
I'm not overly familiar with patterns for Rewrites and Redirects. Only what I've learned on my own...is there a pattern which only targets sequential random numbers rather than just numbers within a string?
Yeah :-/ Same here. I've worked with Umbraco for three years now on the same site and never encountered this problem. Though at the same time, the way Umbraco organises its Media folder - with sequential random numbers vs. the new random numbers and characters - never changed until now.
It's got to be some sort of legacy thing somewhere that I just can't reach.
As I already rebuilt the site from the ground up, I'm hesitant to just go ahead and do it again, but as this error drags on and consumes more of my time, I may as well have by now...
Also, as I don't know what's causing the errors to happen, I could very well rebuild the whole thing and run into the same problem again for all I know..
Yep - that's right actually. Each one of those old folders has an image in it now so is technically valid as well - as I'd uploaded them all before your reply yesterday to give the errors something to find.
I'll give a look to that IsFile/IsDirectory idea and see if it can work and test on image paths that are guaranteed to be bogus. Hopefully should do the trick for that angle anyway. That is, of course, if this error doesn't now mutate and start happening on images paths that do exist!
I think something in the lines of this could work if you are not using Azure Blob Storage for images (remember to verify)
<rule name="Block none existing image files before ImageProcessor" stopProcessing="true">
<match url="^media/" />
<conditions>
<add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
</conditions>
<action type="CustomResponse" statusCode="404" statusReason="File or Directory not found." statusDescription="The resource you are looking for might have been removed, had its name changed, or is temporarily unavailable." />
</rule>
Right! That seems to have done the trick for unavailable image paths. All present images are appearing correctly (both on the site and if navigated to directly) and all bogus image paths are correctly kicking to the 404 page. Only thing I altered in your rule was to point to our own 404 page instead of the generic server error.
Now to sit and wait to see if any more errors come through and/or if the homepage hangs.
Thanks for the help so far Thomas! I'll post back with any updates.
Interesting - my experience is just the opposite. Was absolutely fine for 3 years on v7.14, but now plagued with these errors on the change to v8. With yours and mine, makes me think it's not as much to do with Umbraco per se, but something that's interacting with it...or how it's set up (server, build...etc).
Good news: the homepage didn't hang at all from our work on it yesterday to today.
Bad news, still got two of these through:
ImageProcessor.Common.Exceptions.ImageProcessingException: ProcessImageAsync 610 : No image exists at E:\[root]\blog\media\1009\depositphotos_42232545_m-2015.jpg
As well as two of these:
ImageProcessor.Common.Exceptions.ImageProcessingException: ProcessImageAsync 559 : The request /media/1248/secret-to-a-happy-workplace.jpg?crop=0 could not be understood by the server due to malformed syntax.
Hi all - updates since:
Updated to the latest version of ImageProcessor (v2.8.0) and was fine in regards to website not hanging for 24 hours, though still received the odd syntax error in the logs (mentioned below). However, have just received another path related error which made the homepage front-end hang again. The old image itself wasn't physically there.
ImageProcessor.Common.Exceptions.ImageProcessingException: ProcessImageAsync 609 : No image exists at E:\[root]\blog\media\1009\depositphotos_42232545_m-2015.jpg
Also, since the update I have also still received occasional errors for malformed syntax.
ImageProcessor.Common.Exceptions.ImageProcessingException: ProcessImageAsync 558 : The request /media/1247/doodly-blog-post.jpg?crop=0 could not be understood by the server due to malformed syntax.
These syntax ones happen to old images that physically exist in the place that they're being looked for (as I put them back to attempt to stop the image path errors happening) in addition to newly uploaded Umbraco 8 ones as well (with the random character paths).
So unfortunately, this must mean the IIS trick isn't blocking what's happening, and the update to the latest ImageProcessor didn't solve the errors either.
You should look into why your application hangs when errors occur as that isn't normal, is there something special in your error handling that can cause recursion or other strange senarios?
I would recommend using a static html error page as a start for 500+ errors. as that eliminates all custom code.
Eg. if your 500 error page links to a missing image it would cause a loop that could hang the server.
I've been pouring through our server Event Viewer and logs around the time of the hangs. We actually had one this morning, however for the first time - there were no errors, or anything for that matter, in the Umbraco logs around the time the homepage was discovered hanging. Just an HttpException in the Windows Event Viewer where a request had timed out.
So yes, it must be something else happening here than first thought. The image path issues might've aggravated it, but the fact a hang has just happened on its own shows there's something different causing or at least adding to the hanging issue.
I think I've found the reason why the homepage was hanging. It was due to third-party javascript - which existed across the site, but for some reason hated living on the homepage (all other code there was the same as the rest of the site, which is why it didn't cross my mind until the hangs started happening when no errors were present in the backend, so I started stripping out all non-essential scripts for testing).
The initial ImageProcessor errors in the backend that timed with the homepage hanging (and subsequent "malformed syntax" ones) seemed to have been a red herring. Not sure why the malformed syntax ones are happening still, but at least the site isn't hanging because of them, so will investigate later.
Annoyingly enough, aside from everything we tried, putting the old images back in their original folders and locations was the only thing that quieted the missing image errors. I'll try again in the future to remove them and see if the error still persists.
We had our first weekend without a hang in months this weekend, so I'll give it another week before I mark the solution.
Hope you're all well and safe (during these strange times).
Nope, was clear for almost a week then homepage hangs picked up again. No errors anywhere this time in the console or Umbraco log. Only on our Event Viewer on our server marking "HttpException Request timed out" around the times of the hangs. All other pages on the site remain up and running. Still just affecting the homepage for both cultures.
Republishing the home node still does the trick to get the homepage back up, but this is getting really tiring.
Will keep digging and testing until a solution is found.
Hi Maurice,
Sadly, no. I've been republishing the home node this whole time. Hoping in our next update to a fresh site build with v8.13 it stops the homepage hanging issue.
The only updates since are that 1. The error doesn't seem to be tied directly to the Image Processor errors as stated in the title of this thread as they have stopped and the hanging has continued and 2. we found we could get the page back up by also recycling the app pool in IIS, so we've set an automated recycle every 30mins....not ideal, but it's helping with the regular hangs.
One of my running theories is that our server isn't playing nice with .NET 4.7.2. The site was originally built in .NET Core, then switched to .NET 4.7.2 for this latest build (which has had the hanging issues), so I'm hoping as well when the .NET Core version of Umbraco is released, when we update to that, it'll help stop the hanging.
Thanks Andrew! We too have set our IIS app pool to be recycled every 4 hours. That helped with the hangs for now, but isn't a solution, of course.
We suppose there's a problem with a custom index, though: The hangs affect only pages that call the IndexSearcher in the background. All other pages, including the backend, are still working. We didn't notice that earlier, because we only checked the home page which also used the searcher for the custom index :-/
Our custom index gets reindexed regularly with external data every hour. Reindexing takes about 1 minute. Everytime a "hang" occurs, we can see heavy file activity in the folder of our custom index that won't stop until we recycle the app pool. However, as we can see from our log files, these hangs don't match the reindexing schedule at all ...
It seems it's a completely different issue after all :(
Darnit. Thanks for the indexer information - I'll take a look at that at some point on our end and see if it matches up. Fingers crossed a solution can be found. After more than a year of this, I'll be more than happy to see it go. Best of luck. Will check back in if we find anything new.
Homepage regularly hangs after error from ImageProcessor
Hello all. First time posting, so thank you first to all who've come before. I've solved a lot of issues from reading these forums. This, however I cannot...
Our site was previously built and hosted in Umbraco v7.14. I recently rebuilt the site fresh in v8.5 as it was such a large change between versions. As well as created a new database for it.
All was fine for the first few weeks after the new site went up, now I keep receiving these errors in the backend that line up with the homepage (for both cultures) hanging until I sign in and republish the homepage node (all other pages on the site display correctly, it just seems to be affecting the home page).
Example:
The “[root]” is just me replacing the physical path on the server for this post.
They’re all image path related errors. The links are still trying to access the images with the old Umbraco way of numbered folders inside the media folder, rather than the new way of random characters.
Each image mentioned does actually exist in the site, however they were all manually re-inputted when the v8 rebuild happened, so their paths in the backend are correct and display correctly in the front-end.
Example:
I’ve refreshed my server cache as well as my Umbraco caches, so I have no idea where it could be coming from. Any thoughts?
Thanks in advance.
It isn't google that still holds old paths for images, and try to index the images again?
normally you would either have to redirect the old image url's to the new url.
or or create a rewrite rule that responds with 404 all the /media/[0-9]+/* paths to get google to accept that the images are gone.
if google gets a 500 error they will retry the request for a period of time, until they accept it's gone.
Thanks for the response Thomas. I hadn't thought of that. I'll give that idea a go in IIS and see if it can help stop these errors from appearing.
I've also just tried actually giving (Google, or whatever's looking for the old images) the images themselves... I got fed up trying to find the solution to the problem this morning, so re-uploaded all of the old images in their old folders (in addition to their newly uploaded Umb8 counterparts) to sit there and be found by whatever process is looking for them. Haven't had an image path related error since...Although I did get a new one not long after:
Still, I'd prefer to clean the old images out, so I'll give your suggestion a try.
Update: After uploading all of the old images to their old folders and implementing the 404 rule in IIS, we went for about 10 hours without an error (which is a record right now!). However around 1am this morning we got:
Which unfortunately left the homepage hanging until I got into work this morning.
Here's the rule I wrote in the web.config based on your advice Thomas:
Provided it's written correctly, that must mean that something else other than Google is calling to the images. Maybe something from my database?
As a note - this old image that the process was looking for wasn't in that folder - so I've now uploaded it to it. Sort of "plugging holes in a sinking ship" method...Not the best, but there weren't any more errors related to images that had been uploaded yesterday.
Did you test the rule by accessing one of the problemetic url's and see the redirect kick in?
I didn't actually... I tested the pattern within the IIS "Test Pattern..." and it came out correctly, so I didn't think to test the problematic path in the browser itself yesterday to verify.
Just went to do it now and saw that it didn't actually kick to the 404 (have also now changed it to a Redirect - was a Rewrite before). Also had to remove the leading / to get it to work.
Testing further - Once corrected, it did work for the old image link:
https://www.lendingmetrics.com/media/1506/lendingmetrics-takes-home-two-awards-at-the-lending-awards-2019.jpg
(kicked correctly to the 404), however checking through the rest of the site with the rule enabled, it also blocked a number of the newly uploaded (correct) Umbraco 8 images, so I just disabled it.
I'm not overly familiar with patterns for Rewrites and Redirects. Only what I've learned on my own...is there a pattern which only targets sequential random numbers rather than just numbers within a string?
Great, its always a good idea to test that things work in reality.
But I still don't get why the image loading causes the site to hang, I have never experienced that myself.
Yeah :-/ Same here. I've worked with Umbraco for three years now on the same site and never encountered this problem. Though at the same time, the way Umbraco organises its Media folder - with sequential random numbers vs. the new random numbers and characters - never changed until now.
It's got to be some sort of legacy thing somewhere that I just can't reach.
As I already rebuilt the site from the ground up, I'm hesitant to just go ahead and do it again, but as this error drags on and consumes more of my time, I may as well have by now...
Also, as I don't know what's causing the errors to happen, I could very well rebuild the whole thing and run into the same problem again for all I know..
the ending slash should keep it from blocking the other images
or are the folders without letters that are actually valid?
you could also negate the rule with the IsFile or IsDirectory, do't remember the exact name.
That would keep the rule from kicking in if there is an actual file on disk with that name.
Yep - that's right actually. Each one of those old folders has an image in it now so is technically valid as well - as I'd uploaded them all before your reply yesterday to give the errors something to find.
I'll give a look to that IsFile/IsDirectory idea and see if it can work and test on image paths that are guaranteed to be bogus. Hopefully should do the trick for that angle anyway. That is, of course, if this error doesn't now mutate and start happening on images paths that do exist!
I think something in the lines of this could work if you are not using Azure Blob Storage for images (remember to verify)
Awesome. Thanks! Will give it a go.
Right! That seems to have done the trick for unavailable image paths. All present images are appearing correctly (both on the site and if navigated to directly) and all bogus image paths are correctly kicking to the 404 page. Only thing I altered in your rule was to point to our own 404 page instead of the generic server error.
Now to sit and wait to see if any more errors come through and/or if the homepage hangs.
Thanks for the help so far Thomas! I'll post back with any updates.
Seen this error from an earlier version of image processor after updating it haven't seen the issue since. Happy running the latest version on V8
Interesting - my experience is just the opposite. Was absolutely fine for 3 years on v7.14, but now plagued with these errors on the change to v8. With yours and mine, makes me think it's not as much to do with Umbraco per se, but something that's interacting with it...or how it's set up (server, build...etc).
Have you updated image processor to version 2?
Yeah, as it was fresh install from just over a month ago (via NuGet) it came with version 2.7.0.100.
Just checked and v2.8.0 is out - could try updating to that?
I would suggest so. I tweeted James asking what had changed he said it had some bug fixes. Not sure if it's other fixes unrelated though
Cool. Thanks for that. Appreciate it. I'll update to v2.8 and see if it helps.
Update from yesterday:
Good news: the homepage didn't hang at all from our work on it yesterday to today.
Bad news, still got two of these through:
As well as two of these:
Hi all - updates since: Updated to the latest version of ImageProcessor (v2.8.0) and was fine in regards to website not hanging for 24 hours, though still received the odd syntax error in the logs (mentioned below). However, have just received another path related error which made the homepage front-end hang again. The old image itself wasn't physically there.
Also, since the update I have also still received occasional errors for malformed syntax.
These syntax ones happen to old images that physically exist in the place that they're being looked for (as I put them back to attempt to stop the image path errors happening) in addition to newly uploaded Umbraco 8 ones as well (with the random character paths).
So unfortunately, this must mean the IIS trick isn't blocking what's happening, and the update to the latest ImageProcessor didn't solve the errors either.
You should look into why your application hangs when errors occur as that isn't normal, is there something special in your error handling that can cause recursion or other strange senarios?
I would recommend using a static html error page as a start for 500+ errors. as that eliminates all custom code.
Eg. if your 500 error page links to a missing image it would cause a loop that could hang the server.
Oky doke - will give that a go.
I've been pouring through our server Event Viewer and logs around the time of the hangs. We actually had one this morning, however for the first time - there were no errors, or anything for that matter, in the Umbraco logs around the time the homepage was discovered hanging. Just an HttpException in the Windows Event Viewer where a request had timed out.
So yes, it must be something else happening here than first thought. The image path issues might've aggravated it, but the fact a hang has just happened on its own shows there's something different causing or at least adding to the hanging issue.
Will keep digging..
thanks for the solution. will give it a go !!
Hi all. Thank you for your time on this.
I think I've found the reason why the homepage was hanging. It was due to third-party javascript - which existed across the site, but for some reason hated living on the homepage (all other code there was the same as the rest of the site, which is why it didn't cross my mind until the hangs started happening when no errors were present in the backend, so I started stripping out all non-essential scripts for testing).
The initial ImageProcessor errors in the backend that timed with the homepage hanging (and subsequent "malformed syntax" ones) seemed to have been a red herring. Not sure why the malformed syntax ones are happening still, but at least the site isn't hanging because of them, so will investigate later.
Annoyingly enough, aside from everything we tried, putting the old images back in their original folders and locations was the only thing that quieted the missing image errors. I'll try again in the future to remove them and see if the error still persists.
We had our first weekend without a hang in months this weekend, so I'll give it another week before I mark the solution.
Hope you're all well and safe (during these strange times).
Nope, was clear for almost a week then homepage hangs picked up again. No errors anywhere this time in the console or Umbraco log. Only on our Event Viewer on our server marking "HttpException Request timed out" around the times of the hangs. All other pages on the site remain up and running. Still just affecting the homepage for both cultures.
Republishing the home node still does the trick to get the homepage back up, but this is getting really tiring.
Will keep digging and testing until a solution is found.
Hi Andrew, did you find a solution?
Hi Maurice, Sadly, no. I've been republishing the home node this whole time. Hoping in our next update to a fresh site build with v8.13 it stops the homepage hanging issue.
The only updates since are that 1. The error doesn't seem to be tied directly to the Image Processor errors as stated in the title of this thread as they have stopped and the hanging has continued and 2. we found we could get the page back up by also recycling the app pool in IIS, so we've set an automated recycle every 30mins....not ideal, but it's helping with the regular hangs.
One of my running theories is that our server isn't playing nice with .NET 4.7.2. The site was originally built in .NET Core, then switched to .NET 4.7.2 for this latest build (which has had the hanging issues), so I'm hoping as well when the .NET Core version of Umbraco is released, when we update to that, it'll help stop the hanging.
Thanks Andrew! We too have set our IIS app pool to be recycled every 4 hours. That helped with the hangs for now, but isn't a solution, of course.
We suppose there's a problem with a custom index, though: The hangs affect only pages that call the IndexSearcher in the background. All other pages, including the backend, are still working. We didn't notice that earlier, because we only checked the home page which also used the searcher for the custom index :-/
Our custom index gets reindexed regularly with external data every hour. Reindexing takes about 1 minute. Everytime a "hang" occurs, we can see heavy file activity in the folder of our custom index that won't stop until we recycle the app pool. However, as we can see from our log files, these hangs don't match the reindexing schedule at all ...
It seems it's a completely different issue after all :(
Darnit. Thanks for the indexer information - I'll take a look at that at some point on our end and see if it matches up. Fingers crossed a solution can be found. After more than a year of this, I'll be more than happy to see it go. Best of luck. Will check back in if we find anything new.
is working on a reply...