Umbraco.Cms.Core.DistributedLocking.Exceptions.DistributedWriteLockTimeoutException: Failed to acquire write lock for id: -333.
Microsoft.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded.
I've been trying to track this one down for a while. Since we upgraded to Umbraco 10.5.1 for our latest site, this error has been causing us severe issues and we're about to go live so needs fixed.
As I said, we are running 10.5.1, on Azure but without any sort of load balancing. When the error happens there is sometimes a developer doing things on localhost:port but not today, no one is working on the site and we are still getting the error.
I've added some more info to the associated issue. If there is an official umbraco rep who can contact me I can provide a db where the issue happens. It seems to happen when the db has a LOT of pages - my site has over 4000.
Thanks Russell for adding to the voices. It is really frustrating with this build of Umbraco and worse still our site is hardly 30 pages but it is very custom and with developers distributed around Scotland because of remote working, we are all developing in different ways but all need to connect to the same database.
From everything I have read, it feels like the person who wrote the code for the database has tried to be super diligent to ensure that we don't get 2 or more folks saving things at the same time so has put in resource Locks all throughout the code, but it feels heavy handed in some ways. From other bits I have read this issue may have been in older versions of Umbraco but the worst we saw was Examine indexes out of alignment and some caching not working properly, but nothing a quick restart didn't fix.
This -333 error just stops things from working entirely.
But I get it, however we should have the ability to turn Locks and Caching on and off, especially in the development cycle of a website when these things can actual slow processes down.
(Umbraco v10 is still the best version of Umbraco I have ever used and I can't wait to get all my clients moved over to it so I can run on even cheaper Linux servers :'D)
Has there been any progress on this issue, as we're getting the same error intermittently on our 10.5.1 installation?
We've changed the setup so the admin is only served from one server, but we still see this error occasionally. Sometimes the user is able to publish the content item again right away, sometimes they need to wait a few minutes before it's successful.
Any help would be grateful as we're due to go live with this site shortly.
Any update on this? We have the same problem after updating to version 12.0.1 from version 8.X.X. It's not only on the local environment but also on the test slot on Azure where we are testing the new version.
If someone from Umbraco can take a look that would be nice. Here is my stack trace:
Received an error from the server
An error occurred
Failed to acquire read lock for id: -333.
Exception Details
Umbraco.Cms.Core.DistributedLocking.Exceptions.DistributedReadLockTimeoutException, Umbraco.Core, Version=12.0.1.0, Culture=neutral, PublicKeyToken=null: Failed to acquire read lock for id: -333.
Stacktrace
at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock..ctor(SqlServerDistributedLockingMechanism parent, Int32 lockId, DistributedLockType lockType, TimeSpan timeout)
at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.ReadLock(Int32 lockId, Nullable`1 obtainLockTimeout)
at Umbraco.Cms.Core.Scoping.LockingMechanism.ObtainReadLock(Int32 lockId, Nullable`1 timeout)
at Umbraco.Cms.Core.Scoping.LockingMechanism.LockInner(Guid instanceId, Dictionary`2& locks, HashSet`1& locksSet, Action`2 obtainLock, Nullable`1 timeout, Int32 lockId)
at Umbraco.Cms.Core.Scoping.LockingMechanism.EagerReadLockInner(Guid instanceId, Nullable`1 timeout, Int32[] lockIds)
at Umbraco.Cms.Core.Scoping.LockingMechanism.EnsureLocks(Guid scopeInstanceId)
at Umbraco.Cms.Infrastructure.Scoping.Scope.get_Database()
at Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.DocumentRepository.GetContentSchedule(Int32 contentId)
at Umbraco.Cms.Core.Services.ContentService.GetContentScheduleByContentId(Int32 contentId)
at Umbraco.Cms.Web.BackOffice.Controllers.ContentController.<>c__DisplayClass91_0.<MapToDisplayWithSchedule>b__0(MapperContext context)
at Umbraco.Cms.Core.Mapping.UmbracoMapper.Map[TTarget](Object source, Action`1 f)
at Umbraco.Cms.Web.BackOffice.Controllers.ContentController.MapToDisplayWithSchedule(IContent content)
at Umbraco.Cms.Web.BackOffice.Controllers.ContentController.GetById(Int32 id)
at lambda_method2372(Closure, Object, Object[])
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
Sadly no further forward. We have had to limit all devs to use the staging domain when they are adding content and creating doctypes. And only 1 person building models locally. :(
When you go live, you should be fine. I have found the issue was because of Azure and multiple "servers" connecting to the database.
I've forced the front end Devs to only use the staging URL to create empty components or create pages, leaving me as the only one using localhost, or at least me being the only one using localhost making changes to the database...as it seems Reads are fine but the -333 happens only on updates.
If you are having the issues on live in back office on Azure, back office does not work well on a load balanced setup, or it doesn't work at all really.
So if you have something load balanced even for your 20-30 pages, then look at a sub domain for /Umbraco only.
It's still a pain. I don't understand why they've written back office in the way they have. Most Back offices are rarely used by many many users at the same time. But it feels like they've tried to code it in a way to be Enterprise-y ala Sitecore. They maybe should have that type of code split off in a Umbraco.Enterprise edition while the rest of us small Dev shops get one that works 🤪 imho of course - Umbraco is too good these days for everything else, so I truly don't want to bring it down.
Yep, our live is just a normal web and sql server setup so not azure. Like you, we just have a number of devs using localhost and thats what seems to cause the problem. Very painful, sometimes feels like 1 step forward 2 steps back but hopefully they get on it and sort it. I really don't care about the odd situation caches screw up slightly.
Just came across this thread and we have been struggling this with our update to the .NET core versions of Umbraco (11 in to 12).
There is also a bug report on the repo GitHub about this and the devs have not responded in months. It would be nice to at least know someone is looking at this because it renders the backoffice completely unusable.
FYI for those interested, the only way I have found to remedy/workaround this issue is to explicitly go in the the database, find the blocking process (which was spun up by Umbraco), and kill it. This will allow the backoffice users to continue to be able to save without shutting down or restarting the database (which we don't have the luxury of doing on SQL Azure).
I have only seen it where the whole row was missing. I have not done a deep dive and found out what this value actually does, but you should be able to change it to 1 and see if it has any good affect. Otherwise change it back.
Any updates? We are experiencing this problem as well, but with our MemberTree (-335), even with latest Umbraco version... It's having a big impact on our platform when there are a lot of users on the website as we often have to get Member information. Making it nearly unmanageable.
Almost a year since I originally posted this, using Umbraco 13 bleeding edge on an Azure setup - database and hosting - with only 1 person using the site, and only 1 domain... I am getting this error when I try to unpublish multiple items in a single go.
It seems to be able to do 20 unpublishes from a Child Items grid view but then hits -333.
As I am a developer, I mostly shrug it off.
As a client, I'd be royally annoyed!!
Why is it Umbraco's own team never answer these Forum questions and anything that is asked on github get's ignored? We are paying £000s a year in partnership fees and other things, yet these issues are still happening with no reasoning?
Failed to acquire write lock for id: -333
Help :D
I've been trying to track this one down for a while. Since we upgraded to Umbraco 10.5.1 for our latest site, this error has been causing us severe issues and we're about to go live so needs fixed.
As I said, we are running 10.5.1, on Azure but without any sort of load balancing. When the error happens there is sometimes a developer doing things on localhost:port but not today, no one is working on the site and we are still getting the error.
appsettings:
Is there anything I should be considering? I have another site that is on 10.3 and never has any issues with -333.
Help :(
More information. When no one is on the site, I can save a page twice but on the third hit of the save button I get -333 Failed To Acquire Lock.
I've added some more info to the associated issue. If there is an official umbraco rep who can contact me I can provide a db where the issue happens. It seems to happen when the db has a LOT of pages - my site has over 4000.
Thanks Russell for adding to the voices. It is really frustrating with this build of Umbraco and worse still our site is hardly 30 pages but it is very custom and with developers distributed around Scotland because of remote working, we are all developing in different ways but all need to connect to the same database.
From everything I have read, it feels like the person who wrote the code for the database has tried to be super diligent to ensure that we don't get 2 or more folks saving things at the same time so has put in resource Locks all throughout the code, but it feels heavy handed in some ways. From other bits I have read this issue may have been in older versions of Umbraco but the worst we saw was Examine indexes out of alignment and some caching not working properly, but nothing a quick restart didn't fix.
This -333 error just stops things from working entirely.
But I get it, however we should have the ability to turn Locks and Caching on and off, especially in the development cycle of a website when these things can actual slow processes down.
(Umbraco v10 is still the best version of Umbraco I have ever used and I can't wait to get all my clients moved over to it so I can run on even cheaper Linux servers :'D)
Any update on this? Er are having the same problem after the update.
Hi,
Has there been any progress on this issue, as we're getting the same error intermittently on our 10.5.1 installation?
We've changed the setup so the admin is only served from one server, but we still see this error occasionally. Sometimes the user is able to publish the content item again right away, sometimes they need to wait a few minutes before it's successful.
Any help would be grateful as we're due to go live with this site shortly.
Regards, Mike
Any update on this? We have the same problem after updating to version 12.0.1 from version 8.X.X. It's not only on the local environment but also on the test slot on Azure where we are testing the new version.
If someone from Umbraco can take a look that would be nice. Here is my stack trace:
Sadly no further forward. We have had to limit all devs to use the staging domain when they are adding content and creating doctypes. And only 1 person building models locally. :(
Any update on this issue? It is a huge problem and has been for some time now.
Any update on this, we have a site going live soon, only 20 - 30 pages and we are having problems already?
When you go live, you should be fine. I have found the issue was because of Azure and multiple "servers" connecting to the database.
I've forced the front end Devs to only use the staging URL to create empty components or create pages, leaving me as the only one using localhost, or at least me being the only one using localhost making changes to the database...as it seems Reads are fine but the -333 happens only on updates.
If you are having the issues on live in back office on Azure, back office does not work well on a load balanced setup, or it doesn't work at all really.
So if you have something load balanced even for your 20-30 pages, then look at a sub domain for /Umbraco only.
It's still a pain. I don't understand why they've written back office in the way they have. Most Back offices are rarely used by many many users at the same time. But it feels like they've tried to code it in a way to be Enterprise-y ala Sitecore. They maybe should have that type of code split off in a Umbraco.Enterprise edition while the rest of us small Dev shops get one that works 🤪 imho of course - Umbraco is too good these days for everything else, so I truly don't want to bring it down.
Yep, our live is just a normal web and sql server setup so not azure. Like you, we just have a number of devs using localhost and thats what seems to cause the problem. Very painful, sometimes feels like 1 step forward 2 steps back but hopefully they get on it and sort it. I really don't care about the odd situation caches screw up slightly.
Just came across this thread and we have been struggling this with our update to the .NET core versions of Umbraco (11 in to 12).
There is also a bug report on the repo GitHub about this and the devs have not responded in months. It would be nice to at least know someone is looking at this because it renders the backoffice completely unusable.
FYI for those interested, the only way I have found to remedy/workaround this issue is to explicitly go in the the database, find the blocking process (which was spun up by Umbraco), and kill it. This will allow the backoffice users to continue to be able to save without shutting down or restarting the database (which we don't have the luxury of doing on SQL Azure).
https://github.com/umbraco/Umbraco-CMS/issues/14195
Too bad there still isn't a solution for this.. still struggeling with this one in latest v13 version...
Hi,
Could you make sure that inside of [umbracoLock] table is a row with Id -333 and Value 1 and Name ContentTree ?
I have seen on upgrades sometimes these values are missing for the table and that gives these errors
It's there, but actually with value -1...
Wat does that mean? Should I change it to 1?
I have only seen it where the whole row was missing. I have not done a deep dive and found out what this value actually does, but you should be able to change it to 1 and see if it has any good affect. Otherwise change it back.
This wasn't the solution unfortunately...
As you can see here there are many more people who are struggeling with this one: https://github.com/umbraco/Umbraco-CMS/issues/14195
But the Umbraco crew doesn't seem to matter that much :(
Any updates? We are experiencing this problem as well, but with our MemberTree (-335), even with latest Umbraco version... It's having a big impact on our platform when there are a lot of users on the website as we often have to get Member information. Making it nearly unmanageable.
Are all your users accessing the database/code/site on their own machines, or are the all connecting via the 1 domain on 1 server via a browser?
Almost a year since I originally posted this, using Umbraco 13 bleeding edge on an Azure setup - database and hosting - with only 1 person using the site, and only 1 domain... I am getting this error when I try to unpublish multiple items in a single go.
It seems to be able to do 20 unpublishes from a Child Items grid view but then hits -333.
As I am a developer, I mostly shrug it off. As a client, I'd be royally annoyed!!
Why is it Umbraco's own team never answer these Forum questions and anything that is asked on github get's ignored? We are paying £000s a year in partnership fees and other things, yet these issues are still happening with no reasoning?
As per https://github.com/umbraco/Umbraco-CMS/issues/14195 discussion, the problem is described here: https://umbracare.net/blog/addressing-umbraco-performance-issues-a-critical-update-for-version-13-and-14/ and PR that attempts to fix the issue: https://github.com/umbraco/Umbraco-CMS/pull/16837
Not tried myself so cannot say whether it's working or not.
Ah this is brilliant and thank you for posting all the info. Hopefully this comes very soon to an Umbraco upgrade path :D
is working on a reply...