7.4.1 Performance issues between staging and production
I've got a site that was in 7.2.8 that I migrated to 7.4.1. There are quite a few host names I have to support given our development, test, staging and production environments.
I was hosting the site on Windows Azure but it was too slow for our client's liking. We moved it to their data center. Their production site runs great now, but we're looking at deploying updates and their staging site runs DOG slow. The code is nearly identical asside from some web.config updates (SSL cookies, X-Frame-Options host header, and a few other minor things).
What could be causing this problem? I've noticed that Umbraco seems to need some time to "do it's thing" before speeding up but I can't make heads or tails of it. Any recommendations would be greatly appreciated!
It's hard to say how long it take for Umbraco to do it's thing. It just seems like it's slow with new domains at first, which makes sense, but sometimes there seems to be no rhyme or reason.
The slowdown is constant right now. We made some slight modifications, but nothing major. The database is the same and the IIS settings are the same.
The live version of the site is very fast, but this staging version is suddenly slow. I've tried adding the new code and settings one at a time and cannot isolate what the issue could be.
I'll have to check with my client to see what version of Windows he's running on as well as updates, etc.
fcnMode is set to single in the web.config.
I will have to check with my client to see if the CPU is getting pegged. I am headed there tomorrow and will check.
No load balancing that I know of, but there is a WAF in front of the site(s).
/umbraco runs much faster than the main site.
I see a ton of the following error in the trace logs:
13:24:58 WARN Umbraco.Web.UmbracoModule
Status code is 404 yet TrySkipIisCustomErrors is false - IIS will take over.
Wouldn't worry too much about the 404 warning. If you want to get rid of it, you might want to set trySkipIssCustomErrors to true in your umbracoSettings.config file (that will hopefully cause Umbraco to handle the 404 error). You could also try comparing your staging files to that of a freshly manually installed Umbraco site to see if any files didn't get transferred. You could also look at your IIS logs to see which URL's are causing the most 404 errors, which could give you some extra hints of where you should focus your efforts.
I would make sure the staging server has all the latest Windows updates. There was a Windows bug (in IIS or ASP.NET... I forget) that would incorrectly detect some file changes as ones that should initiate an application pool recycle, which lead to repeated application pool recycles. Installing that fix should resolve that issue if it is occurring.
One thing I also tend to do is reconfigure rapid fail protection in IIS (you can configure this on the application pool):
The default is shown in that screenshot. If you increase the maximum failures (to, say, 100), it's less likely that enough errors will occur quickly enough to cause IIS to recycle the application pool.
We are consistently seeing the Umbraco site wait for 4-6 seconds before loading. The problem appears to be on the server side, but we're at a loss as to what could be slowing Umbraco from returning a request in a reasonable amount of time.
We experienced an issue using basic authentication on staging sites - do you by any chance have basic auth on IIS enabled with it being a staging site?
I haven't tried removing the templates yet, but the issue appears to be with the Umbraco.Web.Scheduling.BackgroundTaskRunner. When running debuggers on our staging server it shows it waiting for 6.281 seconds before releasing. This is consistent on every request. Something appears to be wrong with the task runner.
We disabled ModelBuilder and it sped up the site considerably. Apparently something in the update to 7.4.1 did not work properly. I have not tried removing the MySqlSiteMapProvider yet.
7.4.1 Performance issues between staging and production
I've got a site that was in 7.2.8 that I migrated to 7.4.1. There are quite a few host names I have to support given our development, test, staging and production environments.
I was hosting the site on Windows Azure but it was too slow for our client's liking. We moved it to their data center. Their production site runs great now, but we're looking at deploying updates and their staging site runs DOG slow. The code is nearly identical asside from some web.config updates (SSL cookies, X-Frame-Options host header, and a few other minor things).
What could be causing this problem? I've noticed that Umbraco seems to need some time to "do it's thing" before speeding up but I can't make heads or tails of it. Any recommendations would be greatly appreciated!
Thank you!
I'd need some more info to help diagnose this. Here are some questions that knowing the answers to would help:
Thanks Nicholas.
It's hard to say how long it take for Umbraco to do it's thing. It just seems like it's slow with new domains at first, which makes sense, but sometimes there seems to be no rhyme or reason.
The slowdown is constant right now. We made some slight modifications, but nothing major. The database is the same and the IIS settings are the same.
The live version of the site is very fast, but this staging version is suddenly slow. I've tried adding the new code and settings one at a time and cannot isolate what the issue could be.
I'll have to check with my client to see what version of Windows he's running on as well as updates, etc.
fcnMode is set to single in the web.config.
I will have to check with my client to see if the CPU is getting pegged. I am headed there tomorrow and will check.
No load balancing that I know of, but there is a WAF in front of the site(s).
/umbraco runs much faster than the main site.
I see a ton of the following error in the trace logs:
13:24:58 WARN Umbraco.Web.UmbracoModule
Status code is 404 yet TrySkipIisCustomErrors is false - IIS will take over.
I'm not sure what to think about that error.
I'm wondering if this post (possible rewrite solution near the bottom) could help.
https://our.umbraco.org/forum/umbraco-7/using-umbraco-7/72516-umbracowebumbracomodule-returns-404
Wouldn't worry too much about the 404 warning. If you want to get rid of it, you might want to set trySkipIssCustomErrors to true in your umbracoSettings.config file (that will hopefully cause Umbraco to handle the 404 error). You could also try comparing your staging files to that of a freshly manually installed Umbraco site to see if any files didn't get transferred. You could also look at your IIS logs to see which URL's are causing the most 404 errors, which could give you some extra hints of where you should focus your efforts.
I would make sure the staging server has all the latest Windows updates. There was a Windows bug (in IIS or ASP.NET... I forget) that would incorrectly detect some file changes as ones that should initiate an application pool recycle, which lead to repeated application pool recycles. Installing that fix should resolve that issue if it is occurring.
One thing I also tend to do is reconfigure rapid fail protection in IIS (you can configure this on the application pool):
The default is shown in that screenshot. If you increase the maximum failures (to, say, 100), it's less likely that enough errors will occur quickly enough to cause IIS to recycle the application pool.
We are consistently seeing the Umbraco site wait for 4-6 seconds before loading. The problem appears to be on the server side, but we're at a loss as to what could be slowing Umbraco from returning a request in a reasonable amount of time.
HI Joel.
We experienced an issue using basic authentication on staging sites - do you by any chance have basic auth on IIS enabled with it being a staging site?
Just a thought.
Carl
What happens if you delete the entire contents of your templates (i.e., your CSHTML views)? Does the site load any faster?
I haven't tried removing the templates yet, but the issue appears to be with the Umbraco.Web.Scheduling.BackgroundTaskRunner. When running debuggers on our staging server it shows it waiting for 6.281 seconds before releasing. This is consistent on every request. Something appears to be wrong with the task runner.
I wish I could just turn it off.
What if you try this:
From here: https://our.umbraco.org/projects/starter-kits/ublogsy/ublogsy-bugs/59408-Publish-Content-not-working-properly-on-live-site#comment-200906
Which I found from here: https://our.umbraco.org/forum/umbraco-7/using-umbraco-7/67762-moved-site-to-new-host-now-publishing-is-very-slow-and-non-responsive#comment-225067
Thanks Nicholas.
We disabled ModelBuilder and it sped up the site considerably. Apparently something in the update to 7.4.1 did not work properly. I have not tried removing the MySqlSiteMapProvider yet.
Joel
Glad to hear you found an improvement! #h5yr
Curious though that there is (or was) a difference in speed between servers. I wonder what that's about.
is working on a reply...