I am seeing tons of "Lock request time out period exceeded The statement has been terminated." when using Azure Web Apps and trying to publish descendants. There are a lot of publish issues with Umbraco when trying to publish more than one thing even with all the recommended Azure Web App settings.
I've been getting this more and more since 8.9 and it's progressively getting worse (even after upgrading to 8.11 today). Setup is Dedicated server running SQL Server 2017 (so no azure).
Always happens after doing a few publish events (updating content) - only way to fix it is to restart SQL Server which is far from ideal, god knows how i'd do this if it was Azure.
Log entry:
2021-01-28 11:49:02,112 [P15776/D2/T17] ERROR Umbraco.Web.Editors.ContentController - Unhandled controller exception occurred for request 'http://site.com.local/umbraco/backoffice/UmbracoApi/Content/PostSave'
System.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded.
The statement has been terminated.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func1 func) in D:\a\1\s\src\Umbraco.Core\Persistence\FaultHandling\RetryPolicy.cs:line 172
at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
at Umbraco.Core.Persistence.SqlSyntax.SqlServerSyntaxProvider.WriteLock(IDatabase db, TimeSpan timeout, Int32[] lockIds) in D:\a\1\s\src\Umbraco.Core\Persistence\SqlSyntax\SqlServerSyntaxProvider.cs:line 272
at Umbraco.Core.Services.Implement.ContentService.Save(IContent content, Int32 userId, Boolean raiseEvents) in D:\a\1\s\src\Umbraco.Core\Services\Implement\ContentService.cs:line 782
at Umbraco.Web.Editors.ContentController.SaveAndNotify(ContentItemSave contentItem, Func2 saveMethod, Int32 variantCount, Dictionary2 notifications, SimpleNotificationModel globalNotifications, String invariantSavedLocalizationKey, String variantSavedLocalizationKey, String cultureForInvariantErrors, Boolean& wasCancelled) in D:\a\1\s\src\Umbraco.Web\Editors\ContentController.cs:line 924
at Umbraco.Web.Editors.ContentController.PostSaveInternal(ContentItemSave contentItem, Func2 saveMethod, Func2 mapToDisplay) in D:\a\1\s\src\Umbraco.Web\Editors\ContentController.cs:line 712
at lambdamethod(Closure , Object , Object[] )
at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>cDisplayClass62.
@Umbraco people, what is happening with this, we have started seeing this happening on a number of sites using 8.6+ Essentially it is like the database is in single user mode, so if one person is deleting a node no one can do anything else. This needs sorted as sites are going live.
@Umbraco people this needs looked at asap, we are seeing several clients with this problem that are not on azure and are running 8.6+ we have upgrade to 8.12 but that has not resolved the problem.
Cool - let us know how you get on. Since I originally reported it, i've had very few, if any recurrences of it, even though i'm still on 8.11.1 which is really bizarre!
OK, we have already seen some odd results with timeout set at 5000. At one point all content on the front end of the site dissappeared and only re-appeared once republished or memory cache was reloaded.
Not sure what to suggest tbh, can't get simpler than that. In my case, I'm running IIS on one server, Azure for DB and development is on another machine but shared db.
Could be worth re-opening that github issue and tagging in Shannon to see if he can shed any light on it
Thanks, the problem isn't the same, we are seeing allsorts now, all seem related to caching, we have regenerated in memory cache 3 times now as we started seeing content dissappear without warning. I will re-open the GitHub issue.
Just my two cents on this issue seeing as I was one of the initial reporters of it on the github issue page.
So over the months for us and most people this solution when added to the web.config worked well and resulted in far less frequent occurrences of the issue
If you are running on azure like most of our sites are then add this as well in the app services configuration
With our particular use case, we have external applications that can create content in umbraco via an api fairly straight forward stuff. However we also have a bunch of editors in different time zones that also create content. The bulk of new content is created or rather ingested via the api into umbraco.
What I noticed was that the lock would usually 90% of the time occur whenever the api was ingesting content while an editor was trying to also do something with content in umbraco (save, save and publish, delete etc)
I finally managed to get rid of the lock error eventually by changing the models builder mode from purelive to dll. This required me to first install the full models builder version in umbraco as it doesn't come with the full version any longer.
Once that was done I cleared the temp folder stuff in App_Data and restarted the site and that resolved the lock issue.
Another issue that resolved was the endless wait after making a change to a document type to save that change on said document type. I think this is most noticeable when you have a gazillion nodes that use that document type.
Anyway... I haven't gotten around to figuring out why though as yet but maybe give that a try.
Oh I must also say that disabling models builder all together didn't make a difference.
Another thing to note, I am not using the timeout setting as I believe that addresses something else completely
Please also note that the causes of this are a bit complex and vary depending on the setup. Our setup is simple no load balancing (no other installations pointing to the same db). Take some time a read this thread here to get a better understanding of the causes and scenarios.
https://github.com/umbraco/Umbraco-CMS/issues/8006#issuecomment-620328022
SQL lock request timeout
Hi We are currently using Umbraco for our self service portal.
We are currently experience lots of sql lock request errors :
System.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded
Users have to register and sign in to use the site so we have a lot of read/write to Members.
I have read various posts and we have added the following line to config and upgraded Umbraco to 8.6.5
Umbraco.Core.MainDom.Lock = SqlMainDomLock
Anyone sugestions?
Cheers
Umbraco 8.9.
I am seeing tons of "Lock request time out period exceeded The statement has been terminated." when using Azure Web Apps and trying to publish descendants. There are a lot of publish issues with Umbraco when trying to publish more than one thing even with all the recommended Azure Web App settings.
I've been getting this more and more since 8.9 and it's progressively getting worse (even after upgrading to 8.11 today). Setup is Dedicated server running SQL Server 2017 (so no azure).
Always happens after doing a few publish events (updating content) - only way to fix it is to restart SQL Server which is far from ideal, god knows how i'd do this if it was Azure.
Log entry:
2021-01-28 11:49:02,112 [P15776/D2/T17] ERROR Umbraco.Web.Editors.ContentController - Unhandled controller exception occurred for request 'http://site.com.local/umbraco/backoffice/UmbracoApi/Content/PostSave' System.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded. The statement has been terminated. at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action
1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource
1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func
1 func) in D:\a\1\s\src\Umbraco.Core\Persistence\FaultHandling\RetryPolicy.cs:line 172 at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd) at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args) at Umbraco.Core.Persistence.SqlSyntax.SqlServerSyntaxProvider.WriteLock(IDatabase db, TimeSpan timeout, Int32[] lockIds) in D:\a\1\s\src\Umbraco.Core\Persistence\SqlSyntax\SqlServerSyntaxProvider.cs:line 272 at Umbraco.Core.Services.Implement.ContentService.Save(IContent content, Int32 userId, Boolean raiseEvents) in D:\a\1\s\src\Umbraco.Core\Services\Implement\ContentService.cs:line 782 at Umbraco.Web.Editors.ContentController.SaveAndNotify(ContentItemSave contentItem, Func2 saveMethod, Int32 variantCount, Dictionary
2 notifications, SimpleNotificationModel globalNotifications, String invariantSavedLocalizationKey, String variantSavedLocalizationKey, String cultureForInvariantErrors, Boolean& wasCancelled) in D:\a\1\s\src\Umbraco.Web\Editors\ContentController.cs:line 924 at Umbraco.Web.Editors.ContentController.PostSaveInternal(ContentItemSave contentItem, Func2 saveMethod, Func
2 mapToDisplay) in D:\a\1\s\src\Umbraco.Web\Editors\ContentController.cs:line 712 at lambdamethod(Closure , Object , Object[] ) at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>cDisplayClass62.@Umbraco people, what is happening with this, we have started seeing this happening on a number of sites using 8.6+ Essentially it is like the database is in single user mode, so if one person is deleting a node no one can do anything else. This needs sorted as sites are going live.
@Umbraco people this needs looked at asap, we are seeing several clients with this problem that are not on azure and are running 8.6+ we have upgrade to 8.12 but that has not resolved the problem.
Similar problems: https://our.umbraco.com/forum/using-umbraco-and-getting-started/102325-lock-request-time-out-period-exceeded
Related?
https://our.umbraco.com/forum/using-umbraco-and-getting-started/102070-examine-index-error-members-can-not-login
Have you reviewed the fix that was applied and configured the timeout value?
https://github.com/umbraco/Umbraco-CMS/pull/9744
Thanks for pointing that out Simon. Looking at that I have added the following to our web.config:
Hopefully that will help.
Cool - let us know how you get on. Since I originally reported it, i've had very few, if any recurrences of it, even though i'm still on 8.11.1 which is really bizarre!
OK, we have already seen some odd results with timeout set at 5000. At one point all content on the front end of the site dissappeared and only re-appeared once republished or memory cache was reloaded.
Problem has shifted some what but still recieving errors, customers reporting "impossible to use".
What is the setup of the site? Load balanced, multi server, db, azure etc?
I have tried adding the writelocktimeout line in to config in the last hour. I will let you know if it makes any difference for me.
Super simple, not load balanced, single server (same server for db and iis).
Not sure what to suggest tbh, can't get simpler than that. In my case, I'm running IIS on one server, Azure for DB and development is on another machine but shared db.
Could be worth re-opening that github issue and tagging in Shannon to see if he can shed any light on it
Thanks, the problem isn't the same, we are seeing allsorts now, all seem related to caching, we have regenerated in memory cache 3 times now as we started seeing content dissappear without warning. I will re-open the GitHub issue.
Hi All
Just my two cents on this issue seeing as I was one of the initial reporters of it on the github issue page.
So over the months for us and most people this solution when added to the web.config worked well and resulted in far less frequent occurrences of the issue
If you are running on azure like most of our sites are then add this as well in the app services configuration
With our particular use case, we have external applications that can create content in umbraco via an api fairly straight forward stuff. However we also have a bunch of editors in different time zones that also create content. The bulk of new content is created or rather ingested via the api into umbraco.
What I noticed was that the lock would usually 90% of the time occur whenever the api was ingesting content while an editor was trying to also do something with content in umbraco (save, save and publish, delete etc)
I finally managed to get rid of the lock error eventually by changing the models builder mode from purelive to dll. This required me to first install the full models builder version in umbraco as it doesn't come with the full version any longer.
Once that was done I cleared the temp folder stuff in App_Data and restarted the site and that resolved the lock issue.
Another issue that resolved was the endless wait after making a change to a document type to save that change on said document type. I think this is most noticeable when you have a gazillion nodes that use that document type.
Anyway... I haven't gotten around to figuring out why though as yet but maybe give that a try.
Oh I must also say that disabling models builder all together didn't make a difference.
Another thing to note, I am not using the timeout setting as I believe that addresses something else completely
Umbraco versions : 8.7, 8.11, 8.15.2
Please also note that the causes of this are a bit complex and vary depending on the setup. Our setup is simple no load balancing (no other installations pointing to the same db). Take some time a read this thread here to get a better understanding of the causes and scenarios. https://github.com/umbraco/Umbraco-CMS/issues/8006#issuecomment-620328022
I encountered the following issues in load-balanced environment ( 1 master and 2 front end in Umbraco 8.15.1 )
NuCache.Content.db because it is being used by another process when restarting one of front ends
Lock request time out when publishing contents in Umbraco back office.
I was able to resolve the issue by applying your solutions
Thanks!
Is that all you did? I'm seeing this on a non load-balanced environment. Getting sql timeouts and the same NuCache error you got.
I've already tried the MainDom.Lock app setting, and disabling the overlapped recycling on the application pool.
https://our.umbraco.com/forum/using-umbraco-and-getting-started/107789-site-suddenly-starts-exceed-pool-limits-and-fail-to-boot
Thanks will have a look at this.
Did you ever solve the problem? I'm experiencing similar issues: https://our.umbraco.com/forum/using-umbraco-and-getting-started/107789-site-suddenly-starts-exceed-pool-limits-and-fail-to-boot
is working on a reply...