We have run soak tests (using WebLoad) against a site we are developing and found consistently that three hours into the test, CPU on all web servers, SQL servers and search servers goes to 100% for several minutes, during which time, many 500 server errors are logged.
This does not coincide with application pool recycling.
The site under test is being served from two load-balanced web servers.
Assuming the observed spikes are nothing to do with routine maintenance of the underlying server platform (such as backups starting, etc.), is there an Umbraco activity that would explain this, such as (thinking out loud) SQL flushing or re-indexing, a scheduled system Umbraco task, etc.?
For info, the errors relate to SQL: "Exception message: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding." This does not necessarily mean SQL itself is the cause; SQL could merely be another victim of the same underlying issue.
have you set up any monitoring to see what services and what processes the database server and web server is using. it may be a specific query or other element that is causing the issue, out of the box it is not something I have seen when testing on a load balanced or mirrored web site.
The fact you are getting the issue makes it seem like a sceduled taks or it could be the result of a cumulative process that is slowing using up resources. I feel a infinite loop coming one
However It's not something I have seen working out the box, do you have any custom cursors or other looping search code that could be eating up resources?
Umbraco performance test showing 100% CPU three hours into test
We have run soak tests (using WebLoad) against a site we are developing and found consistently that three hours into the test, CPU on all web servers, SQL servers and search servers goes to 100% for several minutes, during which time, many 500 server errors are logged.
This does not coincide with application pool recycling.
The site under test is being served from two load-balanced web servers.
Assuming the observed spikes are nothing to do with routine maintenance of the underlying server platform (such as backups starting, etc.), is there an Umbraco activity that would explain this, such as (thinking out loud) SQL flushing or re-indexing, a scheduled system Umbraco task, etc.?
For info, the errors relate to SQL: "Exception message: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding." This does not necessarily mean SQL itself is the cause; SQL could merely be another victim of the same underlying issue.
Thanks in advance for any help.
Neil
have you set up any monitoring to see what services and what processes the database server and web server is using. it may be a specific query or other element that is causing the issue, out of the box it is not something I have seen when testing on a load balanced or mirrored web site.
especially if the site is relatively simple.
Thanks Ravi. We are using resource monitor and will be analysing the results over the next day or so.
The fact you are getting the issue makes it seem like a sceduled taks or it could be the result of a cumulative process that is slowing using up resources. I feel a infinite loop coming one
However It's not something I have seen working out the box, do you have any custom cursors or other looping search code that could be eating up resources?
Turned out to be a scheduled virus scan.
Thanks for your suggestions.
Neil
is working on a reply...