Concurrency - deadlock in umbraco.content.LoadContentFromDatabase
Hi people!
Using Umbraco 4.0.4.2 for a large site (18MB XML cache) and experiencing the following problem: Our customer works with several people in the Umbraco back-end @ the same time. From time to time we are experiencing deadlocks (like 6 times a day) and some of these times, the website shows the "There's still some work to do" page (No nodes).
That's because the umbraco.config xml cache was corrupt (just one node inside of it, without decent <?xml ?> headers and stuff...
The root of the problem lies in loading the contentXml from the database, which takes 16 seconds at some moments (busy server). In these 16 seconds the chances for deadlocks to occur are pretty high. That's why we're getting the following exception in the umbracoLog table:
Error Republishing: System.Data.SqlClient.SqlException: Transaction (Process ID 73) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) at System.Data.SqlClient.SqlDataReader.HasMoreRows() at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout) at umbraco.DataLayer.RecordsReaderAdapter`1.Read() at umbraco.content.LoadContentFromDatabase(Boolean loadDocuments)
So the problem lies inside the "umbraco.content.LoadContentFromDatabase(bool)" method, where there's no locking at all (private object instead of private static object).
We're also getting this exception a lot (while permissions are perfect, I'm sure!), which also points at concurrency I guess:
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/2/ROOT
Process ID: 134828
Exception: System.UnauthorizedAccessException
Message: Access to the path 'C:\inetpub\wwwroot\<website_folder>\data\umbraco.config' is denied.
StackTrace: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.File.SetAttributes(String path, FileAttributes fileAttributes) at umbraco.content.ClearDiskCache() at umbraco.content.SaveContentToDisk(XmlDocument xmlDoc) at umbraco.content.<>c__DisplayClass10.<SaveContentToDiskAsync>b__f(Object ) at System.Threading.ExecutionContext.runTryCode(Object userData) at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack) at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)
This is partially fixed in 4.5.X, because the lock objects are static, but the SqlHelper doesn't implement anything like transactions or locking.
We can't update to 4.5.X yet, because of some important packages which are not compatible (yet).
I guess this is a bug and should be fixed, meanwhile I turned off the settings "XmlCacheEnabled" and "ContinouslyUpdateXmlDiskCache", hoping the site won't shut down again.
Anyway, does anyone know what's happening, is it a bug and is there something we can do about it?
No I'm afraid not... I ended up editing the umbraco source, so when a deadlock occurs, the current method get's called again recursively (up to 5 times). But this is not a great fix, it's more like a dirty hack.
Linked to this (https://our.umbraco.org/forum/umbraco-as-a-service/issues/66369-loadcontentfromdatabase-taking-site-down) as it sounds like the same issue, and I did manage to work around it with a SQL change.
Concurrency - deadlock in umbraco.content.LoadContentFromDatabase
Hi people!
Using Umbraco 4.0.4.2 for a large site (18MB XML cache) and experiencing the following problem:
Our customer works with several people in the Umbraco back-end @ the same time.
From time to time we are experiencing deadlocks (like 6 times a day) and some of these times, the website shows the "There's still some work to do" page (No nodes).
That's because the umbraco.config xml cache was corrupt (just one node inside of it, without decent <?xml ?> headers and stuff...
The root of the problem lies in loading the contentXml from the database, which takes 16 seconds at some moments (busy server). In these 16 seconds the chances for deadlocks to occur are pretty high. That's why we're getting the following exception in the umbracoLog table:
So the problem lies inside the "umbraco.content.LoadContentFromDatabase(bool)" method, where there's no locking at all (private object instead of private static object).
We're also getting this exception a lot (while permissions are perfect, I'm sure!), which also points at concurrency I guess:
This is partially fixed in 4.5.X, because the lock objects are static, but the SqlHelper doesn't implement anything like transactions or locking.
We can't update to 4.5.X yet, because of some important packages which are not compatible (yet).
I guess this is a bug and should be fixed, meanwhile I turned off the settings "XmlCacheEnabled" and "ContinouslyUpdateXmlDiskCache", hoping the site won't shut down again.
Anyway, does anyone know what's happening, is it a bug and is there something we can do about it?
Hi
Did you find out what the problem was here?
Thanks,
Alex
No I'm afraid not :-(
Doh!
Did turning off the settings "XmlCacheEnabled" and "ContinouslyUpdateXmlDiskCache" help at all?
No I'm afraid not... I ended up editing the umbraco source, so when a deadlock occurs, the current method get's called again recursively (up to 5 times). But this is not a great fix, it's more like a dirty hack.
Hi Stefan,
Do remember if you ever got to the bottom of this? I have a similar issue here:
MultiNode Tree Picker - TreeDataService Deadlock
Thanks, Simon
Nah I don't remember. The website this was about is now on v4.11.8, so guess it's fixed in the meantime...
Linked to this (https://our.umbraco.org/forum/umbraco-as-a-service/issues/66369-loadcontentfromdatabase-taking-site-down) as it sounds like the same issue, and I did manage to work around it with a SQL change.
is working on a reply...