We have a website which receives a lot of trafic in these days, and now we have some performance issues with the server.
When diving in to Worker Process -> Request in IIS 7, I find that there is a lot of requests, runned by ImageProcessorModule which takes a very long time. Time elapsed as high as 3.661.329 msec
Did you resolve this? We are running a 4 server load balanced environment and this is killing performance over a certain number of users. the processor hits 100% and the site dies and it seems to be due to imageprocessor requests.
At work we recently ran a test isolating ImageProcessor requests to a single box on a set up of 3 servers with 13 Umbraco sites. That box showed no significant change in CPU.
Some of those sites get 1500-2000 simultaneous users at a time.
I can't prove it and to be honest I'm sceptical that this is the root of the problem. I suspect there is another config issue underlying this but the imageprocessor module is showing lots of requests in IIS.
I was really just looking some advice on where else to look.
The ImageProcessor module is called on every request as RAMMFAR will be enabled so that's likely why it shows up a lot.
You can test whether image requests are getting cached though by isolating a single request and checking it in Chrome with the network pane open in developer tools. That should show the correct response headers with expires etc and also give you an indicator of the processing time. If there is any logs you could post I could have a look to see if I can see anything fishy.
Oh and what version of Umbraco and ImageProcessor.Web are you using with what caching mechanism?
We are running U7.2.8 and I upgraded ImageProcessor and ImageProcessor.Web to the latest versions yesterday. There is nothing in the Umbraco logs that I can see.
A bit of further info. I replaced using ImageProcessor (https://our.umbraco.org/projects/website-utilities/slimsy) with ImageGen on one page and performance has improved considerably.
Sorry, I don't follow. You replaced Slimsy with ImageGen?
Slimsy isn't ImageProcessor and you definitely do not want to replace ImageProcessor. The caching mechanism in ImageGen will not cope over time and a httphandler rather than a module simply isn't scalable.
I think Umbraco requires RAMMFAR to run. I can't think why it would be enabled otherwise.
That request time of 33ms isn't bad. you'll find multiple images within a page will not add to that time significantly. The browser is being set to cache subsequent requests so for repeating users there will be no request.
Realistically if you are load balancing you should be using a cloud caching mechanism unless you are only using one of your machines to process image requests. Otherwise you are creating multiple caches.
We are using Slimsy to display images. My understanding is that it uses the built in Umbraco image cropper which uses ImageProcessor?
As an experiment because we were getting such poor results I replaced Slimsy with ImageGen and it improved things. I know it's not ideal and it's not a permanent solution but it seems to work.
The request time for a single image is fine. It's when we ramp up the load test that things spin out.
Perhaps you could point me at some code you would use to generate images using ImageProcessor and I can try that?
Yeah, Slimsy uses the built in cropper but the underlying JavaScript library it uses creates a lot more calls than I personally would like.
I've been using Picturefill to polyfill the picture element and set up a series of crops for what I need. That works well and is efficient.
When I get a moment I'll see if I can run some updated load tests on the ImageProcessor solution to ensure that there isn't an issue. (It could be something funky going on within Umbraco, it could be something I've introduced.)
If you have the chance yourself there is actually a Test Website within the ImageProcessor solution that you could run load tests against.
Thanks James. Yes I suspected the problem was more to do with Slimsy than ImageProcessor. I've replaced most Slimsy calls with standard GetCropUrl calls instead and will run some more tests to see what happens.
We set up ImageProcessor on a cookieless domain in IIS for a high traffic site, and it worked beautifully in staging.
Under load though it literally died. Completely. Even requests that weren't using the querystring parameters would get stuck waiting. We had interceptAllRequests="false".
There is nothing else running on the site than ImageProcessor.
Edit: To be more specific, the only things installed on the site are ImageProcessor, ImageProcessor.Web and ImageProcessor.Web.Config
I'm going to need more info than this. if you have interceptAllRequests="false" enabled ImageProcessor.Web is doing very, very little. Everything will be offloaded to the static file handler.
Thanks for getting back. I wrote that post in a slightly panicked rush so it didn't make much sense! We had load tested a new site prior to pushing live but we didn't load test the image serving performance.
The umbraco site contains products each of which have something like 60 images (including a spinner). We serve these from a cookieless domain via a separate site in IIS.
This separate cookieless site was running ImageProcessor and ImageProcessor.Web, as well as the postprocessor (which we used as it improved Pageview Insights scores).
The cookieless site used virtual directories for the product images (stored separately and nothing to do with Umbraco) as well as virtual directories for umbraco media and other assets. Again, as far as the cookieless site is concerned, just folders full of assess and nothing else.
What we found was when we went live, although the umbraco site didn't blink under load, the image site completely fell over. It would have been processing a large number of simultaneous requests for images that weren't cropped yet.
The worker process went up to 2000 threads and 3gb and was totally unresponsive. ALL images served through the cookieless site either failed to load or took minutes. We tried using 6 worker processes for the site but this didn't help. We then got to a stage where we couldn't terminate the original massive worker process and IIS reset did nothing, we had to reboot the box. Basically it was an absolute mess! What really caught me off guard was that it wasn't processing requests that didn't even use any querystring parameters. I expected a performance hit but not a total blockage.
Eventually after some hell we switched out the whole thing for ImageResizer which seems to handle everything without much sweat. Of course it's "expensive" and closed source. :(
I would love to know what went so catastrophically wrong with Imageprocessor though. I really don't know what went wrong as you said you've used it on sites with thousands of concurrent users. But perhaps we were loading more images?
I know ImageProcessor is undergoing major rewriting right now, and you need assistance with that and please don't think I'm complaining about it, it's worked great for me in other sites in the past and I'm sure in the future. I just want to know what was causing the issue for next time and maybe identify what it's limitations are.
Ps. I'll post my settings and configuration as it's perfectly possible that I screwed up configuration.
It's super disappointing to hear you've had issues. I'm only able to test so much and am not able to personally load test high volume sites during debugging with my laptop so maybe there is an unknown problem.
I'm wondering, did you try it without the postprocessor? That would have been my first thing to check. There's some interestingcode in there where I have to asynchronously spin up an instance of a command window to run the tool for each image. I can't imagine that will scale well at all.
The actual processing/caching itself though should be fast.
I'm hoping in the future with the rewrite to avoid the need for postprocessing whatsoever since my encoders and decoders are all open source and should be thoroughly optimizable. I've never done anything as difficult as writing a 2D imaging API though so it's a slow process fraught with issue.
Share your config files if you can though I doubt there's an issue with them.
Sorry to interject, but we developed a (fairly) old site now in Umbraco 6 that uses an early version of ImageProcessor (1.9.4) and ImageProcess.Web (3.2.9). These date back to mid-2015. This site is load-balanced and can have a high-load on it as it sells tickets for major events and when a ticket goes on sale the load can be high. It's been through and passed professional load-testing tests. We've not add any issues with performance I'm aware of.
I'm guessing this old version didn't have Postprocessor? When was this added as a default and is it still active? As I'd bet this would more likely be the culprit.
Also, and this is a bit tangential, I did something of note. On our custom error handling page (defined like this in web.config)
we log all exceptions. One a new site I noticed a lot of requests for non-existent images (they had existed on the old site) that was causing the following exception to be thrown:
System.Web.HttpException (0x80004005): No image exists at C:\inetpub\wwwroot\MyWebsite\images\email\headerbg.gif
at ImageProcessor.Web.Services.LocalFileImageService.<GetImage>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at ImageProcessor.Web.HttpModules.ImageProcessingModule.<ProcessImageAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar)
at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
So it seems that every time any image is requested and not found - even if there are no query params - ImageProcessor (or a component of) is throwing an exception. Can this be avoided and if an image is not found or it has no image params it is left to IIS to handle?
Dan, PostProcessor is not on by default, you have to install it (which just drops the DLL in the bin folder, as far as I can tell, I didn't have to touch the configuration).
I did pull the PostProcessor DLL out pretty fast from the site, and the application was restarted a number of times later but we still had the severe performance issues. It's hard to say how much better that made it since we were not in a controlled test (!) but it was essentially still not serving any images for long periods of time. CPU usage wasn't that high, it just sat there using memory and filling threads.
For a bit of context, the site has around 3000 products, each of which will need about 60 images processing - 1 in listings, 40 on loading the product page, and around 20 on client-side request for a gallery. So that's around 180,000 images that need to be processed.
We only had up to 300 concurrent users, which I thought would be easily handled, but it still just seemed to grind to a halt.
Zac, sounds like your use case is very different to the site I mentioned - whilst it had a lot of visits it didn't have anything like that number of images. Plus most images would have been created very early on and then just served cached - there wouldn't have been lots of variations.
I only mention post processing because I thought (and I may be wrong) that it was incorporated into ImageProcessor by default at one point.
Okay, good to know about the use case. In our case the products change a lot every day too, maybe 5%-10% turnover so it can't rely too much on everything already being "done".
I think you're right on the post processing, as the docs currently indicate (I believe incorrectly) that you don't need to install the plugin anymore, but I saw a tweet a while back by James (IIRC) that said this has been moved to a separate package again.
I'm intrigued by the numbers 300 concurrent users is nothing. As I noted above we have 1500-2000 concurrent users at a time on one of our sites. It seems to handle that no bother.
I really need to get some load testing sorted but I simply don't have the equipment.
Dan, the latest versions ignore 404's and let whatever normally handles them take over (usually the static file handler).
Indeed James the 300 concurrent users isn't much and that's why I wasn't too concerned with load-testing the image processing before launch, and it's quite tricky to set that up.
But I think the sheer concurrency of image cropping, even with only 300 users, was too much for it.
If 200 users are hitting product pages in the same time period, ImageProcessor would be hit with upwards of 8,000 tasks at the same time, and I suppose that if it can't keep up then the queue just grows. After 36 hours the site has now made nearly 250,000 crops (using ImageResizer) so I suppose it's more about number of requests than number of users.
Out of interest, your big site - roughly how many images do you have?
URGENT: Performance issue with ImageProcessor
Hi
We have a website which receives a lot of trafic in these days, and now we have some performance issues with the server.
When diving in to Worker Process -> Request in IIS 7, I find that there is a lot of requests, runned by ImageProcessorModule which takes a very long time. Time elapsed as high as 3.661.329 msec
Is the images not cached?
What can we do?
It is an Umbraco 7.2.1
Hi Carsten,
I'm going to need a little bit more information from you in order to determine what is going on.
Hopefully we can get to the bottom of this.
Cheers
James
Did you resolve this? We are running a 4 server load balanced environment and this is killing performance over a certain number of users. the processor hits 100% and the site dies and it seems to be due to imageprocessor requests.
Can you please provide proof it's ImageProcessor?
At work we recently ran a test isolating ImageProcessor requests to a single box on a set up of 3 servers with 13 Umbraco sites. That box showed no significant change in CPU.
Some of those sites get 1500-2000 simultaneous users at a time.
I can't prove it and to be honest I'm sceptical that this is the root of the problem. I suspect there is another config issue underlying this but the imageprocessor module is showing lots of requests in IIS.
I was really just looking some advice on where else to look.
Thanks for your response.
No worries,
I'm not sure where else to look to be honest.
The ImageProcessor module is called on every request as RAMMFAR will be enabled so that's likely why it shows up a lot.
You can test whether image requests are getting cached though by isolating a single request and checking it in Chrome with the network pane open in developer tools. That should show the correct response headers with expires etc and also give you an indicator of the processing time. If there is any logs you could post I could have a look to see if I can see anything fishy.
Oh and what version of Umbraco and ImageProcessor.Web are you using with what caching mechanism?
So can RAMMFAR be turned off? Or can we restrict requests to those required by Umbraco
For a single image: Response time: 33ms Response headers: HTTP/1.1 304 Not Modified Cache-Control: public, must-revalidate, max-age=31536000 Expires: Fri, 21 Oct 2016 06:21:18 GMT Accept-Ranges: bytes ETag: "454120bacbfad01:0" Vary: Accept-Encoding ImageProcessedBy: ImageProcessor.Web/4.4.0.0 Date: Thu, 22 Oct 2015 06:21:17 GMT
parameters: anchor:center mode:crop quality:80 width:2048 heightratio:0.2991452991452991452991452991 format:jpg slimmage:true rnd:130822305520000000
We are running U7.2.8 and I upgraded ImageProcessor and ImageProcessor.Web to the latest versions yesterday. There is nothing in the Umbraco logs that I can see.
A bit of further info. I replaced using ImageProcessor (https://our.umbraco.org/projects/website-utilities/slimsy) with ImageGen on one page and performance has improved considerably.
Sorry, I don't follow. You replaced Slimsy with ImageGen?
Slimsy isn't ImageProcessor and you definitely do not want to replace ImageProcessor. The caching mechanism in ImageGen will not cope over time and a httphandler rather than a module simply isn't scalable.
I think Umbraco requires RAMMFAR to run. I can't think why it would be enabled otherwise.
That request time of 33ms isn't bad. you'll find multiple images within a page will not add to that time significantly. The browser is being set to cache subsequent requests so for repeating users there will be no request.
Realistically if you are load balancing you should be using a cloud caching mechanism unless you are only using one of your machines to process image requests. Otherwise you are creating multiple caches.
We are using Slimsy to display images. My understanding is that it uses the built in Umbraco image cropper which uses ImageProcessor?
As an experiment because we were getting such poor results I replaced Slimsy with ImageGen and it improved things. I know it's not ideal and it's not a permanent solution but it seems to work.
The request time for a single image is fine. It's when we ramp up the load test that things spin out.
Perhaps you could point me at some code you would use to generate images using ImageProcessor and I can try that?
Thanks for all your help, really useful.
Dan
Happy to help...
Yeah, Slimsy uses the built in cropper but the underlying JavaScript library it uses creates a lot more calls than I personally would like.
I've been using Picturefill to polyfill the picture element and set up a series of crops for what I need. That works well and is efficient.
When I get a moment I'll see if I can run some updated load tests on the ImageProcessor solution to ensure that there isn't an issue. (It could be something funky going on within Umbraco, it could be something I've introduced.)
If you have the chance yourself there is actually a Test Website within the ImageProcessor solution that you could run load tests against.
Maybe something might come up.
Thanks James. Yes I suspected the problem was more to do with Slimsy than ImageProcessor. I've replaced most Slimsy calls with standard GetCropUrl calls instead and will run some more tests to see what happens.
Hi Dan, Did you ever get anywhere with this?
Hi James,
We set up ImageProcessor on a cookieless domain in IIS for a high traffic site, and it worked beautifully in staging.
Under load though it literally died. Completely. Even requests that weren't using the querystring parameters would get stuck waiting. We had interceptAllRequests="false".
There is nothing else running on the site than ImageProcessor.
Edit: To be more specific, the only things installed on the site are ImageProcessor, ImageProcessor.Web and ImageProcessor.Web.Config
All are the latest up-to-date versions.
Hi Zac,
I'm going to need more info than this. if you have
interceptAllRequests="false"enabled ImageProcessor.Web is doing very, very little. Everything will be offloaded to the static file handler.https://github.com/JimBobSquarePants/ImageProcessor/blob/Framework/src/ImageProcessor.Web/HttpModules/ImageProcessingModule.cs#L489
What do you mean by this? Is it an Umbraco build? if so, are there any known performance issues with that. Do you have any packages installed etc.
A debug log would be useful.
https://blogs.msdn.microsoft.com/tess/2008/02/04/net-debugging-demos-information-and-setup-instructions/
Cheers
James
Hi James,
Thanks for getting back. I wrote that post in a slightly panicked rush so it didn't make much sense! We had load tested a new site prior to pushing live but we didn't load test the image serving performance.
The umbraco site contains products each of which have something like 60 images (including a spinner). We serve these from a cookieless domain via a separate site in IIS.
This separate cookieless site was running ImageProcessor and ImageProcessor.Web, as well as the postprocessor (which we used as it improved Pageview Insights scores).
The cookieless site used virtual directories for the product images (stored separately and nothing to do with Umbraco) as well as virtual directories for umbraco media and other assets. Again, as far as the cookieless site is concerned, just folders full of assess and nothing else.
What we found was when we went live, although the umbraco site didn't blink under load, the image site completely fell over. It would have been processing a large number of simultaneous requests for images that weren't cropped yet.
The worker process went up to 2000 threads and 3gb and was totally unresponsive. ALL images served through the cookieless site either failed to load or took minutes. We tried using 6 worker processes for the site but this didn't help. We then got to a stage where we couldn't terminate the original massive worker process and IIS reset did nothing, we had to reboot the box. Basically it was an absolute mess! What really caught me off guard was that it wasn't processing requests that didn't even use any querystring parameters. I expected a performance hit but not a total blockage.
Eventually after some hell we switched out the whole thing for ImageResizer which seems to handle everything without much sweat. Of course it's "expensive" and closed source. :(
I would love to know what went so catastrophically wrong with Imageprocessor though. I really don't know what went wrong as you said you've used it on sites with thousands of concurrent users. But perhaps we were loading more images?
I know ImageProcessor is undergoing major rewriting right now, and you need assistance with that and please don't think I'm complaining about it, it's worked great for me in other sites in the past and I'm sure in the future. I just want to know what was causing the issue for next time and maybe identify what it's limitations are.
Ps. I'll post my settings and configuration as it's perfectly possible that I screwed up configuration.
Hi Zac,
It's super disappointing to hear you've had issues. I'm only able to test so much and am not able to personally load test high volume sites during debugging with my laptop so maybe there is an unknown problem.
I'm wondering, did you try it without the postprocessor? That would have been my first thing to check. There's some interesting code in there where I have to asynchronously spin up an instance of a command window to run the tool for each image. I can't imagine that will scale well at all.
The actual processing/caching itself though should be fast.
I'm hoping in the future with the rewrite to avoid the need for postprocessing whatsoever since my encoders and decoders are all open source and should be thoroughly optimizable. I've never done anything as difficult as writing a 2D imaging API though so it's a slow process fraught with issue.
Share your config files if you can though I doubt there's an issue with them.
Cheers
James
Sorry to interject, but we developed a (fairly) old site now in Umbraco 6 that uses an early version of ImageProcessor (1.9.4) and ImageProcess.Web (3.2.9). These date back to mid-2015. This site is load-balanced and can have a high-load on it as it sells tickets for major events and when a ticket goes on sale the load can be high. It's been through and passed professional load-testing tests. We've not add any issues with performance I'm aware of.
I'm guessing this old version didn't have Postprocessor? When was this added as a default and is it still active? As I'd bet this would more likely be the culprit.
Also, and this is a bit tangential, I did something of note. On our custom error handling page (defined like this in web.config)
we log all exceptions. One a new site I noticed a lot of requests for non-existent images (they had existed on the old site) that was causing the following exception to be thrown:
So it seems that every time any image is requested and not found - even if there are no query params - ImageProcessor (or a component of) is throwing an exception. Can this be avoided and if an image is not found or it has no image params it is left to IIS to handle?
Hi James, Dan
Dan, PostProcessor is not on by default, you have to install it (which just drops the DLL in the bin folder, as far as I can tell, I didn't have to touch the configuration).
I did pull the PostProcessor DLL out pretty fast from the site, and the application was restarted a number of times later but we still had the severe performance issues. It's hard to say how much better that made it since we were not in a controlled test (!) but it was essentially still not serving any images for long periods of time. CPU usage wasn't that high, it just sat there using memory and filling threads.
For a bit of context, the site has around 3000 products, each of which will need about 60 images processing - 1 in listings, 40 on loading the product page, and around 20 on client-side request for a gallery. So that's around 180,000 images that need to be processed.
We only had up to 300 concurrent users, which I thought would be easily handled, but it still just seemed to grind to a halt.
Zac, sounds like your use case is very different to the site I mentioned - whilst it had a lot of visits it didn't have anything like that number of images. Plus most images would have been created very early on and then just served cached - there wouldn't have been lots of variations.
I only mention post processing because I thought (and I may be wrong) that it was incorporated into ImageProcessor by default at one point.
Okay, good to know about the use case. In our case the products change a lot every day too, maybe 5%-10% turnover so it can't rely too much on everything already being "done".
I think you're right on the post processing, as the docs currently indicate (I believe incorrectly) that you don't need to install the plugin anymore, but I saw a tweet a while back by James (IIRC) that said this has been moved to a separate package again.
Zac Bugger. Have I not updated the docs yet...
Yeah you need to install it separately.
I'm intrigued by the numbers 300 concurrent users is nothing. As I noted above we have 1500-2000 concurrent users at a time on one of our sites. It seems to handle that no bother.
I really need to get some load testing sorted but I simply don't have the equipment.
Dan, the latest versions ignore 404's and let whatever normally handles them take over (usually the static file handler).
Indeed James the 300 concurrent users isn't much and that's why I wasn't too concerned with load-testing the image processing before launch, and it's quite tricky to set that up.
But I think the sheer concurrency of image cropping, even with only 300 users, was too much for it.
If 200 users are hitting product pages in the same time period, ImageProcessor would be hit with upwards of 8,000 tasks at the same time, and I suppose that if it can't keep up then the queue just grows. After 36 hours the site has now made nearly 250,000 crops (using ImageResizer) so I suppose it's more about number of requests than number of users.
Out of interest, your big site - roughly how many images do you have?
is working on a reply...