Since yesterday we have been seeing deadlocks in our log which are coming from Vendr.Core.Services.OrderService.SaveOrder(Order entity).
We do not have much information on it, but we would like to know what possibly could be wrong or where we could look to prevent this.
Here is all the information I have so far:
Umbraco version 8.8.0
Vendr version v1.3.2
This is the code:
[HttpPost]
public ActionResult AddToCart(AddToCartDto postModel)
{
try
{
using (var uow = _uowProvider.Create())
{
var store = CurrentPage.GetStore();
var order = _sessionManager.GetOrCreateCurrentOrder(store.Id)
.AsWritable(uow)
.AddProduct(postModel.ProductReference, postModel.Quantity);
_orderService.SaveOrder(order);
uow.Complete();
}
}
This is the error message:
Exception
System.Data.SqlClient.SqlException (0x80131904): Transaction (Process ID 86) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) in D:\a\1\s\src\Umbraco.Core\Persistence\FaultHandling\RetryPolicy.cs:line 172
at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
at NPoco.Database.<Update>b__214_0(String sql, Object[] args, Func`2 next)
at NPoco.Database.UpdateImp[TRet](String tableName, String primaryKeyName, Object poco, Object primaryKeyValue, IEnumerable`1 columns, Func`4 executeFunc, TRet defaultId)
at NPoco.Database.Update(String tableName, String primaryKeyName, Object poco, Object primaryKeyValue, IEnumerable`1 columns)
at NPoco.Database.Save[T](T poco)
at Vendr.Core.Persistence.Repositories.OrderRepository.Save(OrderState state, OrderState originalState)
at Vendr.Core.Services.OrderService.SaveOrder(Order entity)
at Website.Store.Web.Controllers.CartSurfaceController.AddToCart(AddToCartDto postModel) in D:\a\1\s\Website.store\Web\Controllers\CartSurfaceController.cs:line 54
at lambda_method(Closure , ControllerBase , Object[] )
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c.<BeginInvokeSynchronousActionMethod>b__9_0(IAsyncResult asyncResult, ActionInvocation innerInvokeState)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`2.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_0.<InvokeActionMethodFilterAsynchronouslyRecursive>b__0()
at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_2.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2()
at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_2.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2()
at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_2.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2()
at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_2.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2()
at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3_6.<BeginInvokeAction>b__4()
at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3_1.<BeginInvokeAction>b__1(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult)
at System.Web.Mvc.Controller.<>c.<BeginExecuteCore>b__152_1(IAsyncResult asyncResult, ExecuteCoreState innerState)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult)
at System.Web.Mvc.MvcHandler.<>c.<BeginProcessRequest>b__20_1(IAsyncResult asyncResult, ProcessRequestState innerState)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
ClientConnectionId:3f050f57-34ed-4c8f-b732-17ad9c23e9de
Error Number:1205,State:45,Class:13
Hmmm, deadlock debugging is no fun, but we'll see what we can find.
Few questions then off the bat
What is your server architecture?
Are you using anything like loadbalancing?
What kind of load is the site under when these deadlocks happen?
Do you have any custom code at play here? Custom event handlers or the likes?
I'd also suggest trying to perform some SQL trace logging on the deadlocks to see if you can get Deadlock Graph for the deadlocks so that we can try and figure out where exactly the deadlock is occuring.
It did sounds like some concurrency thing with entities being saved multiple times (hence the custom code question), but really glad you were able to hunt it down.
Deadlock on OrderService.SaveOrder(Order entity)
Hi,
Since yesterday we have been seeing deadlocks in our log which are coming from Vendr.Core.Services.OrderService.SaveOrder(Order entity). We do not have much information on it, but we would like to know what possibly could be wrong or where we could look to prevent this.
Here is all the information I have so far:
This is the code:
This is the error message:
Hey Michel,
Hmmm, deadlock debugging is no fun, but we'll see what we can find.
Few questions then off the bat
I'd also suggest trying to perform some SQL trace logging on the deadlocks to see if you can get Deadlock Graph for the deadlocks so that we can try and figure out where exactly the deadlock is occuring.
Matt
Hi Matt,
Thanks a lot for answering this fast! Sorry for bothering you with the questions and the very bad info.
Both this and my other question are solved. We had an issue with the product page which posted to the AddToCart method twice.
It was a pain in the .... to find, but we did it!
Ahhh, fantastic.
It did sounds like some concurrency thing with entities being saved multiple times (hence the custom code question), but really glad you were able to hunt it down.
Matt
is working on a reply...