We've not altered our codebase or released a new version of Umbraco for the last few weeks however today I've been informed Umbraco is running exceptionally slowly.
We're hosted in Azure on the app service and use the Azure SQL database
I've traced the cause back to the database suddently hitting 100% DTU usage and it's been pinned there almost all day by the following query:
(@0 nvarchar(40),@1 nvarchar(4000),@2 int)SELECT cmsPropertyData.*
FROM cmsPropertyData
INNER JOIN cmsPropertyType
ON cmsPropertyData.propertytypeid = cmsPropertyType.id
INNER JOIN
(SELECT cmsContent.nodeId, cmsContentVersion.VersionId FROM [cmsContentVersion]
INNER JOIN [cmsContent]
ON [cmsContentVersion].[ContentId] = [cmsContent].[nodeId]
INNER JOIN [umbracoNode]
ON [cmsContent].[nodeId] = [umbracoNode].[id]
WHERE (([umbracoNode].[nodeObjectType] = @0))
AND ((upper([umbracoNode].[path]) LIKE upper(@1) AND ([umbracoNode].[id] <> @2)))
) as docData
ON cmsPropertyData.versionId = docData.VersionId AND cmsPropertyData.contentNodeId = docData.nodeId
LEFT OUTER JOIN cmsDataTypePreValues
ON cmsPropertyType.dataTypeId = cmsDataTypePreValues.datatypeNodeId
From looking at the source I suspect this is related to the VersionableRepositoryBase but I can't track down what could be calling it or how to fix it - any assistance or help would be appreciated.
It calmed down yesterday afternoon but it started again as soon as our first content editor logged in this morning looking at the logs.
Before yesterday our average was 3-5% DTU, only getting close to 100% when running a backup / copying data to another database so it's really suprising to be seeing it.
We also had a lot of database timeouts yesterday in the logs but we're not seeing anything like that today.
We're still facing these problems we're not sure if we've hit a content cliff or it's a change in 7.5.x
I'm going to scale the database it's just strange that it's somewhat random as to when we're seeing the spikes. We had almost a full week without any issues.
We've started to encounter this problem recently after upgrading to 7.5.11 from 7.4.3.
Running in azure we're seeing an S3 database being pinned to max dtus all day where previously were only occasionally spiking to 100%.
What's suspicious are the very high numbers of calls for 4 specific queries.
Each is called nearly a million times in 24 hours, and this is on a single instance edit server in a load balanced environment including courier. There is rarely more than a single user editing content.
Any ideas or hints what activity these 4 queries represent before I go digging in the 7.5.11 source to try to diagnose this myself?
Dan, did you get anywhere resolving this in your case? I'm suspicious that scaling is just papering over an Umbraco bug.
Yes and no, we did scale up to S1 (we were running S0 and 7.5.4 at the time if memory serves) but it still seems intermittent, we occasionally get spikes and the main culprits for us are the same queries (now running 7.5.11).
If I perform admin tasks in a staging slot to rebuild indexes and XML cache integrity before switching it into wider use, 'warming up' it does seem to reduce the occurence, we also increased the database timeouts in the connection string up from the default to 900 seconds as I had a feeling that it was these longer running tasks attempting and failing to complete repeating over and over.
I can't say we've fixed it but those are a couple of steps I tried and they have helped, we do get spikes to 100% but it's not as sustained and doesn't seem to have any performance impact now on the front-end or our editors.
Umbraco Azure sudden 100% database/DTU usage
Hi all,
We've not altered our codebase or released a new version of Umbraco for the last few weeks however today I've been informed Umbraco is running exceptionally slowly.
We're hosted in Azure on the app service and use the Azure SQL database
I've traced the cause back to the database suddently hitting 100% DTU usage and it's been pinned there almost all day by the following query:
From looking at the source I suspect this is related to the VersionableRepositoryBase but I can't track down what could be calling it or how to fix it - any assistance or help would be appreciated.
Kind regards,
Dan
It calmed down yesterday afternoon but it started again as soon as our first content editor logged in this morning looking at the logs.
Before yesterday our average was 3-5% DTU, only getting close to 100% when running a backup / copying data to another database so it's really suprising to be seeing it.
We also had a lot of database timeouts yesterday in the logs but we're not seeing anything like that today.
It's calmed down again now but to give you some perspective: our DTU usage graph looks like Monument Valley
We're still facing these problems we're not sure if we've hit a content cliff or it's a change in 7.5.x
I'm going to scale the database it's just strange that it's somewhat random as to when we're seeing the spikes. We had almost a full week without any issues.
We've started to encounter this problem recently after upgrading to 7.5.11 from 7.4.3.
Running in azure we're seeing an S3 database being pinned to max dtus all day where previously were only occasionally spiking to 100%.
What's suspicious are the very high numbers of calls for 4 specific queries. Each is called nearly a million times in 24 hours, and this is on a single instance edit server in a load balanced environment including courier. There is rarely more than a single user editing content.
Any ideas or hints what activity these 4 queries represent before I go digging in the 7.5.11 source to try to diagnose this myself?
Dan, did you get anywhere resolving this in your case? I'm suspicious that scaling is just papering over an Umbraco bug.
Hi Brett,
Yes and no, we did scale up to S1 (we were running S0 and 7.5.4 at the time if memory serves) but it still seems intermittent, we occasionally get spikes and the main culprits for us are the same queries (now running 7.5.11).
If I perform admin tasks in a staging slot to rebuild indexes and XML cache integrity before switching it into wider use, 'warming up' it does seem to reduce the occurence, we also increased the database timeouts in the connection string up from the default to 900 seconds as I had a feeling that it was these longer running tasks attempting and failing to complete repeating over and over.
I can't say we've fixed it but those are a couple of steps I tried and they have helped, we do get spikes to 100% but it's not as sustained and doesn't seem to have any performance impact now on the front-end or our editors.
Thank you for the suggestions Dan.
We're investigating our case further and think it may be related to the Umbraco permissions model and some work we had to do to 'extend' them.
is working on a reply...