Currently, we have an issue with one of our sites that runs on the latest Umbraco version (8.18.0) that introduced content version cleanup.
The scheduled task fails to execute due to a time-out on the SQL database. We suspect that it's caused by a large number of records that the script is trying to delete and it's timing out because of it. In Umbraco we see this in the logging:
System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
The statement has been terminated. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at StackExchange.Profiling.Data.ProfiledDbCommand.ExecuteNonQuery() in C:\projects\dotnet\src\MiniProfiler.Shared\Data\ProfiledDbCommand.cs:line 272
at Umbraco.Core.Persistence.FaultHandling.FaultHandlingDbCommand.
Has anyone run into this issue? The project is around for quite some time, so there is a lot of page history present in the database. Maybe that's causing issues?
I'm also having trouble getting the content cleanup to work on v8.18.3. Have you had any luck since you posted this?
I turned on debug logging to see if I could find more info, but I'm not having much luck determining where the timeout is occurring. I think it may be timing out trying to obtain a write lock, but I'm not very confident that I understand how the locking works. I added the logs below in case someone sees something obvious that I missed.
2022-07-06 18:26:26,119 [P21744/D12/T38] INFO Umbraco.Core.Runtime.CoreRuntime - Booting Umbraco 8.18.3. [Timing ff6c0f8]
...
2022-07-06 18:26:45,977 [P21744/D12/T12] INFO Umbraco.Web.Search.BackgroundIndexRebuilder - Starting initialize async background thread.
2022-07-06 18:29:46,183 [P21744/D12/T66] DEBUG Umbraco.Core.Services.Implement.ContentService - Discovered 1243 candidate(s) for ContentVersion cleanup.
2022-07-06 18:29:46,213 [P21744/D12/T66] DEBUG Umbraco.Core.Services.Implement.ContentService - Removing 1218 ContentVersion(s).
2022-07-06 18:29:51,857 [P21744/D12/T36] ERROR Umbraco.Core.Persistence.UmbracoDatabase - Exception (0ab4b232).
System.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded.
The statement has been terminated.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) in D:\a\1\s\src\Umbraco.Core\Persistence\FaultHandling\RetryPolicy.cs:line 172
at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
ClientConnectionId:b7c242a9-021f-45de-a3f8-7011d80ae625
Error Number:1222,State:51,Class:16
2022-07-06 18:29:52,286 [P21744/D12/T36] DEBUG Umbraco.Core.Persistence.UmbracoDatabase - At:
at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at Umbraco.Core.Persistence.UmbracoDatabase.OnException(Exception ex) in D:\a\1\s\src\Umbraco.Core\Persistence\UmbracoDatabase.cs:line 211
at NPoco.Database.OnExceptionInternal(Exception exception)
at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
at Umbraco.Core.Persistence.SqlSyntax.SqlServerSyntaxProvider.ObtainWriteLock(IDatabase db, TimeSpan timeout, Int32 lockId) in D:\a\1\s\src\Umbraco.Core\Persistence\SqlSyntax\SqlServerSyntaxProvider.cs:line 285
at Umbraco.Core.Persistence.SqlSyntax.SqlServerSyntaxProvider.WriteLock(IDatabase db, Int32[] lockIds) in D:\a\1\s\src\Umbraco.Core\Persistence\SqlSyntax\SqlServerSyntaxProvider.cs:line 278
at Umbraco.Core.Scoping.Scope.LockInner(IUmbracoDatabase db, Guid instanceId, Dictionary`2& locks, HashSet`1& locksSet, Action`2 obtainLock, Action`3 obtainLockTimeout, Nullable`1 timeout, Int32[] lockIds) in D:\a\1\s\src\Umbraco.Core\Scoping\Scope.cs:line 941
at Umbraco.Core.Scoping.Scope.EagerWriteLockInner(IUmbracoDatabase db, Guid instanceId, Nullable`1 timeout, Int32[] lockIds) in D:\a\1\s\src\Umbraco.Core\Scoping\Scope.cs:line 910
at Umbraco.Core.Scoping.Scope.EnsureDbLocks() in D:\a\1\s\src\Umbraco.Core\Scoping\Scope.cs:line 487
at Umbraco.Core.Scoping.Scope.get_Database() in D:\a\1\s\src\Umbraco.Core\Scoping\Scope.cs:line 324
at Umbraco.Core.Scoping.Scope.get_Database() in D:\a\1\s\src\Umbraco.Core\Scoping\Scope.cs:line 332
at Umbraco.Core.Persistence.Repositories.Implement.ContentRepositoryBase`3.CheckDataIntegrity(ContentDataIntegrityReportOptions options) in D:\a\1\s\src\Umbraco.Core\Persistence\Repositories\Implement\ContentRepositoryBase.cs:line 495
at Umbraco.Core.Services.Implement.ContentService.CheckDataIntegrity(ContentDataIntegrityReportOptions options) in D:\a\1\s\src\Umbraco.Core\Services\Implement\ContentService.cs:line 2450
at Umbraco.Web.HealthCheck.Checks.Data.DatabaseIntegrityCheck.CheckPaths(String actionAlias, String actionName, String entityType, Boolean detailedReport, Func`1 doCheck) in D:\a\1\s\src\Umbraco.Web\HealthCheck\Checks\Data\DatabaseIntegrityCheck.cs:line 60
at Umbraco.Web.HealthCheck.Checks.Data.DatabaseIntegrityCheck.CheckDocuments(Boolean fix) in D:\a\1\s\src\Umbraco.Web\HealthCheck\Checks\Data\DatabaseIntegrityCheck.cs:line 54
at Umbraco.Web.HealthCheck.Checks.Data.DatabaseIntegrityCheck.GetStatus() in D:\a\1\s\src\Umbraco.Web\HealthCheck\Checks\Data\DatabaseIntegrityCheck.cs:line 39
at Umbraco.Web.HealthCheck.HealthCheckResults.<.ctor>b__4_1(HealthCheck t) in D:\a\1\s\src\Umbraco.Web\HealthCheck\HealthCheckResults.cs:line 26
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
at Umbraco.Web.HealthCheck.HealthCheckResults..ctor(IEnumerable`1 checks) in D:\a\1\s\src\Umbraco.Web\HealthCheck\HealthCheckResults.cs:line 19
at Umbraco.Web.Scheduling.HealthCheckNotifier.<PerformRunAsync>d__6.MoveNext() in D:\a\1\s\src\Umbraco.Web\Scheduling\HealthCheckNotifier.cs:line 76
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
at Umbraco.Web.Scheduling.HealthCheckNotifier.PerformRunAsync(CancellationToken token)
at Umbraco.Web.Scheduling.RecurringTaskBase.<RunAsync>d__6.MoveNext() in D:\a\1\s\src\Umbraco.Web\Scheduling\RecurringTaskBase.cs:line 77
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine)
at Umbraco.Web.Scheduling.RecurringTaskBase.RunAsync(CancellationToken token)
at Umbraco.Web.Scheduling.BackgroundTaskRunner`1.<RunAsync>d__40.MoveNext() in D:\a\1\s\src\Umbraco.Web\Scheduling\BackgroundTaskRunner.cs:line 552
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine)
at Umbraco.Web.Scheduling.BackgroundTaskRunner`1.RunAsync(T bgTask, CancellationToken token)
at Umbraco.Web.Scheduling.BackgroundTaskRunner`1.<Pump>d__36.MoveNext() in D:\a\1\s\src\Umbraco.Web\Scheduling\BackgroundTaskRunner.cs:line 410
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
at Umbraco.Web.Scheduling.BackgroundTaskRunner`1.<WaitForLatch>d__39.MoveNext() in D:\a\1\s\src\Umbraco.Web\Scheduling\BackgroundTaskRunner.cs:line 535
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Threading.Tasks.TaskFactory.CompleteOnInvokePromise.Invoke(Task completingTask)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Threading.Tasks.TaskCompletionSource`1.TrySetResult(TResult result)
at System.Threading.Tasks.TaskCompletionSource`1.SetResult(TResult result)
at Umbraco.Web.Scheduling.RecurringTaskBase.<.ctor>b__4_0(Object _) in D:\a\1\s\src\Umbraco.Web\Scheduling\RecurringTaskBase.cs:line 57
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.TimerQueueTimer.CallCallback()
at System.Threading.TimerQueueTimer.Fire()
at System.Threading.TimerQueue.FireNextTimers()
2022-07-06 18:29:52,290 [P21744/D12/T36] DEBUG Umbraco.Core.Persistence.UmbracoDatabase - Sql:
UPDATE umbracoLock WITH (REPEATABLEREAD) SET value = (CASE WHEN (value=1) THEN -1 ELSE 1 END) WHERE id=@0
-- @0:-333
2022-07-06 18:29:52,294 [P21744/D12/T36] ERROR Umbraco.Web.HealthCheck.HealthCheckResults - Error running scheduled health check: Database data integrity check
System.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded.
The statement has been terminated.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) in D:\a\1\s\src\Umbraco.Core\Persistence\FaultHandling\RetryPolicy.cs:line 172
at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
at Umbraco.Core.Persistence.SqlSyntax.SqlServerSyntaxProvider.ObtainWriteLock(IDatabase db, TimeSpan timeout, Int32 lockId) in D:\a\1\s\src\Umbraco.Core\Persistence\SqlSyntax\SqlServerSyntaxProvider.cs:line 285
at Umbraco.Core.Persistence.SqlSyntax.SqlServerSyntaxProvider.WriteLock(IDatabase db, Int32[] lockIds) in D:\a\1\s\src\Umbraco.Core\Persistence\SqlSyntax\SqlServerSyntaxProvider.cs:line 278
at Umbraco.Core.Scoping.Scope.LockInner(IUmbracoDatabase db, Guid instanceId, Dictionary`2& locks, HashSet`1& locksSet, Action`2 obtainLock, Action`3 obtainLockTimeout, Nullable`1 timeout, Int32[] lockIds) in D:\a\1\s\src\Umbraco.Core\Scoping\Scope.cs:line 956
at Umbraco.Core.Scoping.Scope.EagerWriteLockInner(IUmbracoDatabase db, Guid instanceId, Nullable`1 timeout, Int32[] lockIds) in D:\a\1\s\src\Umbraco.Core\Scoping\Scope.cs:line 910
at Umbraco.Core.Scoping.Scope.EnsureDbLocks() in D:\a\1\s\src\Umbraco.Core\Scoping\Scope.cs:line 487
at Umbraco.Core.Scoping.Scope.get_Database() in D:\a\1\s\src\Umbraco.Core\Scoping\Scope.cs:line 330
at Umbraco.Core.Scoping.Scope.get_Database() in D:\a\1\s\src\Umbraco.Core\Scoping\Scope.cs:line 332
at Umbraco.Core.Persistence.Repositories.Implement.ContentRepositoryBase`3.CheckDataIntegrity(ContentDataIntegrityReportOptions options) in D:\a\1\s\src\Umbraco.Core\Persistence\Repositories\Implement\ContentRepositoryBase.cs:line 495
at Umbraco.Core.Services.Implement.ContentService.CheckDataIntegrity(ContentDataIntegrityReportOptions options) in D:\a\1\s\src\Umbraco.Core\Services\Implement\ContentService.cs:line 2450
at Umbraco.Web.HealthCheck.Checks.Data.DatabaseIntegrityCheck.CheckPaths(String actionAlias, String actionName, String entityType, Boolean detailedReport, Func`1 doCheck) in D:\a\1\s\src\Umbraco.Web\HealthCheck\Checks\Data\DatabaseIntegrityCheck.cs:line 60
at Umbraco.Web.HealthCheck.Checks.Data.DatabaseIntegrityCheck.CheckDocuments(Boolean fix) in D:\a\1\s\src\Umbraco.Web\HealthCheck\Checks\Data\DatabaseIntegrityCheck.cs:line 54
at Umbraco.Web.HealthCheck.Checks.Data.DatabaseIntegrityCheck.GetStatus() in D:\a\1\s\src\Umbraco.Web\HealthCheck\Checks\Data\DatabaseIntegrityCheck.cs:line 39
at Umbraco.Web.HealthCheck.HealthCheckResults.<.ctor>b__4_1(HealthCheck t) in D:\a\1\s\src\Umbraco.Web\HealthCheck\HealthCheckResults.cs:line 26
ClientConnectionId:b7c242a9-021f-45de-a3f8-7011d80ae625
Error Number:1222,State:51,Class:16
2022-07-06 18:29:52,330 [P21744/D12/T36] INFO Umbraco.Web.HealthCheck.HealthCheckResults - Scheduled health check results:
...
2022-07-06 18:30:16,279 [P21744/D12/T66] ERROR Umbraco.Core.Persistence.UmbracoDatabase - Exception (ca76e2a2).
System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
The statement has been terminated. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) in D:\a\1\s\src\Umbraco.Core\Persistence\FaultHandling\RetryPolicy.cs:line 172
at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
ClientConnectionId:37583191-6192-411a-b537-5f5d1dcb8958
Error Number:-2,State:0,Class:11
2022-07-06 18:30:16,280 [P21744/D12/T66] DEBUG Umbraco.Core.Persistence.UmbracoDatabase - At:
at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at Umbraco.Core.Persistence.UmbracoDatabase.OnException(Exception ex) in D:\a\1\s\src\Umbraco.Core\Persistence\UmbracoDatabase.cs:line 211
at NPoco.Database.OnExceptionInternal(Exception exception)
at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
at Umbraco.Core.Persistence.Repositories.Implement.DocumentVersionRepository.DeleteVersions(IEnumerable`1 versionIds) in D:\a\1\s\src\Umbraco.Core\Persistence\Repositories\Implement\DocumentVersionRepository.cs:line 123
at Umbraco.Core.Services.Implement.ContentService.CleanupDocumentVersions(DateTime asAtDate) in D:\a\1\s\src\Umbraco.Core\Services\Implement\ContentService.cs:line 3355
at Umbraco.Core.Services.Implement.ContentService.PerformContentVersionCleanup(DateTime asAtDate) in D:\a\1\s\src\Umbraco.Core\Services\Implement\ContentService.cs:line 3232
at Umbraco.Web.Scheduling.ContentVersionCleanup.PerformRun() in D:\a\1\s\src\Umbraco.Web\Scheduling\ContentVersionCleanup.cs:line 68
at Umbraco.Web.Scheduling.RecurringTaskBase.Run() in D:\a\1\s\src\Umbraco.Web\Scheduling\RecurringTaskBase.cs:line 67
at Umbraco.Web.Scheduling.BackgroundTaskRunner`1.<RunAsync>d__40.MoveNext() in D:\a\1\s\src\Umbraco.Web\Scheduling\BackgroundTaskRunner.cs:line 556
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine)
at Umbraco.Web.Scheduling.BackgroundTaskRunner`1.RunAsync(T bgTask, CancellationToken token)
at Umbraco.Web.Scheduling.BackgroundTaskRunner`1.<Pump>d__36.MoveNext() in D:\a\1\s\src\Umbraco.Web\Scheduling\BackgroundTaskRunner.cs:line 410
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
at Umbraco.Web.Scheduling.BackgroundTaskRunner`1.<WaitForLatch>d__39.MoveNext() in D:\a\1\s\src\Umbraco.Web\Scheduling\BackgroundTaskRunner.cs:line 535
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Threading.Tasks.TaskFactory.CompleteOnInvokePromise.Invoke(Task completingTask)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Threading.Tasks.TaskCompletionSource`1.TrySetResult(TResult result)
at System.Threading.Tasks.TaskCompletionSource`1.SetResult(TResult result)
at Umbraco.Web.Scheduling.RecurringTaskBase.<.ctor>b__3_0(Object _) in D:\a\1\s\src\Umbraco.Web\Scheduling\RecurringTaskBase.cs:line 37
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.TimerQueueTimer.CallCallback()
at System.Threading.TimerQueueTimer.Fire()
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
2022-07-06 18:30:16,280 [P21744/D12/T66] DEBUG Umbraco.Core.Persistence.UmbracoDatabase - Sql:
DELETE [umbracoPropertyData]
WHERE ([umbracoPropertyData].[versionId] IN (@0,@1,@2,@3,@4,@5,@6,@7,@8,@9,@10,@11,@12,@13,@14,@15,@16,@17,@18,@19,@20,@21,@22,@23,@24,@25,@26,@27,@28,@29,@30,@31,@32,@33,@34,@35,@36,@37,@38,@39,@40,@41,@42,@43,@44,@45,@46,@47,@48,@49,@50,@51,@52,@53,@54,@55,@56,@57,@58,@59,@60,@61,@62,@63,@64,@65,@66,@67,@68,@69,@70,@71,@72,@73,@74,@75,@76,@77,@78,@79,@80,@81,@82,@83,@84,@85,@86,@87,@88,@89,@90,@91,@92,@93,@94,@95,@96,@97,@98,@99,@100,@101,@102,@103,@104,@105,@106,@107,@108,@109,@110,@111,@112,@113,@114,@115,@116,@117,@118,@119,@120,@121,@122,@123,@124,@125,@126,@127,@128,@129,@130,@131,@132,@133,@134,@135,@136,@137,@138,@139,@140,@141,@142,@143,@144,@145,@146,@147,@148,@149,@150,@151,@152,@153,@154,@155,@156,@157,@158,@159,@160,@161,@162,@163,@164,@165,@166,@167,@168,@169,@170,@171,@172,@173,@174,@175,@176,@177,@178,@179,@180,@181,@182,@183,@184,@185,@186,@187,@188,@189,@190,@191,@192,@193,@194,@195,@196,@197,@198,@199,@200,@201,@202,@203,@204,@205,@206,@207,@208,@209,@210,@211,@212,@213,@214,@215,@216,@217,@218,@219,@220,@221,@222,@223,@224,@225,@226,@227,@228,@229,@230,@231,@232,@233,@234,@235,@236,@237,@238,@239,@240,@241,@242,@243,@244,@245,@246,@247,@248,@249,@250,@251,@252,@253,@254,@255,@256,@257,@258,@259,@260,@261,@262,@263,@264,@265,@266,@267,@268,@269,@270,@271,@272,@273,@274,@275,@276,@277,@278,@279,@280,@281,@282,@283,@284,@285,@286,@287,@288,@289,@290,@291,@292,@293,@294,@295,@296,@297,@298,@299,@300,@301,@302,@303,@304,@305,@306,@307,@308,@309,@310,@311,@312,@313,@314,@315,@316,@317,@318,@319,@320,@321,@322,@323,@324,@325,@326,@327,@328,@329,@330,@331,@332,@333,@334,@335,@336,@337,@338,@339,@340,@341,@342,@343,@344,@345,@346,@347,@348,@349,@350,@351,@352,@353,@354,@355,@356,@357,@358,@359,@360,@361,@362,@363,@364,@365,@366,@367,@368,@369,@370,@371,@372,@373,@374,@375,@376,@377,@378,@379,@380,@381,@382,@383,@384,@385,@386,@387,@388,@389,@390,@391,@392,@393,@394,@395,@396,@397,@398,@399,@400,@401,@402,@403,@404,@405,@406,@407,@408,@409,@410,@411,@412,@413,@414,@415,@416,@417,@418,@419,@420,@421,@422,@423,@424,@425,@426,@427,@428,@429,@430,@431,@432,@433,@434,@435,@436,@437,@438,@439,@440,@441,@442,@443,@444,@445,@446,@447,@448,@449,@450,@451,@452,@453,@454,@455,@456,@457,@458,@459,@460,@461,@462,@463,@464,@465,@466,@467,@468,@469,@470,@471,@472,@473,@474,@475,@476,@477,@478,@479,@480,@481,@482,@483,@484,@485,@486,@487,@488,@489,@490,@491,@492,@493,@494,@495,@496,@497,@498,@499,@500,@501,@502,@503,@504,@505,@506,@507,@508,@509,@510,@511,@512,@513,@514,@515,@516,@517,@518,@519,@520,@521,@522,@523,@524,@525,@526,@527,@528,@529,@530,@531,@532,@533,@534,@535,@536,@537,@538,@539,@540,@541,@542,@543,@544,@545,@546,@547,@548,@549,@550,@551,@552,@553,@554,@555,@556,@557,@558,@559,@560,@561,@562,@563,@564,@565,@566,@567,@568,@569,@570,@571,@572,@573,@574,@575,@576,@577,@578,@579,@580,@581,@582,@583,@584,@585,@586,@587,@588,@589,@590,@591,@592,@593,@594,@595,@596,@597,@598,@599,@600,@601,@602,@603,@604,@605,@606,@607,@608,@609,@610,@611,@612,@613,@614,@615,@616,@617,@618,@619,@620,@621,@622,@623,@624,@625,@626,@627,@628,@629,@630,@631,@632,@633,@634,@635,@636,@637,@638,@639,@640,@641,@642,@643,@644,@645,@646,@647,@648,@649,@650,@651,@652,@653,@654,@655,@656,@657,@658,@659,@660,@661,@662,@663,@664,@665,@666,@667,@668,@669,@670,@671,@672,@673,@674,@675,@676,@677,@678,@679,@680,@681,@682,@683,@684,@685,@686,@687,@688,@689,@690,@691,@692,@693,@694,@695,@696,@697,@698,@699,@700,@701,@702,@703,@704,@705,@706,@707,@708,@709,@710,@711,@712,@713,@714,@715,@716,@717,@718,@719,@720,@721,@722,@723,@724,@725,@726,@727,@728,@729,@730,@731,@732,@733,@734,@735,@736,@737,@738,@739,@740,@741,@742,@743,@744,@745,@746,@747,@748,@749,@750,@751,@752,@753,@754,@755,@756,@757,@758,@759,@760,@761,@762,@763,@764,@765,@766,@767,@768,@769,@770,@771,@772,@773,@774,@775,@776,@777,@778,@779,@780,@781,@782,@783,@784,@785,@786,@787,@788,@789,@790,@791,@792,@793,@794,@795,@796,@797,@798,@799,@800,@801,@802,@803,@804,@805,@806,@807,@808,@809,@810,@811,@812,@813,@814,@815,@816,@817,@818,@819,@820,@821,@822,@823,@824,@825,@826,@827,@828,@829,@830,@831,@832,@833,@834,@835,@836,@837,@838,@839,@840,@841,@842,@843,@844,@845,@846,@847,@848,@849,@850,@851,@852,@853,@854,@855,@856,@857,@858,@859,@860,@861,@862,@863,@864,@865,@866,@867,@868,@869,@870,@871,@872,@873,@874,@875,@876,@877,@878,@879,@880,@881,@882,@883,@884,@885,@886,@887,@888,@889,@890,@891,@892,@893,@894,@895,@896,@897,@898,@899,@900,@901,@902,@903,@904,@905,@906,@907,@908,@909,@910,@911,@912,@913,@914,@915,@916,@917,@918,@919,@920,@921,@922,@923,@924,@925,@926,@927,@928,@929,@930,@931,@932,@933,@934,@935,@936,@937,@938,@939,@940,@941,@942,@943,@944,@945,@946,@947,@948,@949,@950,@951,@952,@953,@954,@955,@956,@957,@958,@959,@960,@961,@962,@963,@964,@965,@966,@967,@968,@969,@970,@971,@972,@973,@974,@975,@976,@977,@978,@979,@980,@981,@982,@983,@984,@985,@986,@987,@988,@989,@990,@991,@992,@993,@994,@995,@996,@997,@998,@999,@1000,@1001,@1002,@1003,@1004,@1005,@1006,@1007,@1008,@1009,@1010,@1011,@1012,@1013,@1014,@1015,@1016,@1017,@1018,@1019,@1020,@1021,@1022,@1023,@1024,@1025,@1026,@1027,@1028,@1029,@1030,@1031,@1032,@1033,@1034,@1035,@1036,@1037,@1038,@1039,@1040,@1041,@1042,@1043,@1044,@1045,@1046,@1047,@1048,@1049,@1050,@1051,@1052,@1053,@1054,@1055,@1056,@1057,@1058,@1059,@1060,@1061,@1062,@1063,@1064,@1065,@1066,@1067,@1068,@1069,@1070,@1071,@1072,@1073,@1074,@1075,@1076,@1077,@1078,@1079,@1080,@1081,@1082,@1083,@1084,@1085,@1086,@1087,@1088,@1089,@1090,@1091,@1092,@1093,@1094,@1095,@1096,@1097,@1098,@1099,@1100,@1101,@1102,@1103,@1104,@1105,@1106,@1107,@1108,@1109,@1110,@1111,@1112,@1113,@1114,@1115,@1116,@1117,@1118,@1119,@1120,@1121,@1122,@1123,@1124,@1125,@1126,@1127,@1128,@1129,@1130,@1131,@1132,@1133,@1134,@1135,@1136,@1137,@1138,@1139,@1140,@1141,@1142,@1143,@1144,@1145,@1146,@1147,@1148,@1149,@1150,@1151,@1152,@1153,@1154,@1155,@1156,@1157,@1158,@1159,@1160,@1161,@1162,@1163,@1164,@1165,@1166,@1167,@1168,@1169,@1170,@1171,@1172,@1173,@1174,@1175,@1176,@1177,@1178,@1179,@1180,@1181,@1182,@1183,@1184,@1185,@1186,@1187,@1188,@1189,@1190,@1191,@1192,@1193,@1194,@1195,@1196,@1197,@1198,@1199,@1200,@1201,@1202,@1203,@1204,@1205,@1206,@1207,@1208,@1209,@1210,@1211,@1212,@1213,@1214,@1215,@1216,@1217))
-- @0:3689 @1:3595 @2:3625 @3:3655 @4:3666 @5:3667 @6:3880 @7:3903 @8:3910 @9:3685 @10:3691 @11:3723 @12:3754 @13:3814 @14:3879 @15:3905 @16:3668 @17:3669 @18:3670 @19:3671 @20:3672 @21:3459 @22:3470 @23:3675 @24:3875 @25:3881 @26:3899 @27:3489 @28:3518 @29:3548 @30:3578 @31:3608 @32:3452 @33:3524 @34:3361 @35:3449 @36:3453 @37:3451 @38:3455 @39:3612 @40:3614 @41:3554 @42:3584 @43:3441 @44:3496 @45:3360 @46:3440 @47:3494 @48:3522 @49:3552 @50:3582 @51:3439 @52:3493 @53:3521 @54:3551 @55:3581 @56:3611 @57:3568 @58:3598 @59:3432 @60:3442 @61:3457 @62:3357 @63:3260 @64:3358 @65:3437 @66:3479 @67:3508 @68:3538 @69:3579 @70:3609 @71:3270 @72:3326 @73:3400 @74:3490 @75:3231 @76:3259 @77:3271 @78:3300 @79:3519 @80:3549 @81:3556 @82:3586 @83:3526 @84:3269 @85:3337 @86:3411 @87:3484 @88:3226 @89:3242 @90:3252 @91:3266 @92:3616 @93:3602 @94:3807 @95:3162 @96:3220 @97:3317 @98:3391 @99:3224 @100:3123 @101:3132 @102:3512 @103:3542 @104:3572 @105:3340 @106:3414 @107:3528 @108:3084 @109:3222 @110:3223 @111:3080 @112:3558 @113:3588 @114:3236 @115:3253 @116:3267 @117:3486 @118:3514 @119:3544 @120:2906 @121:2909 @122:2937 @123:3812 @124:3822 @125:3574 @126:3604 @127:3321 @128:3395 @129:3567 @130:3597 @131:2905 @132:2908 @133:2931 @134:3311 @135:3297 @136:3806 @137:3385 @138:3478 @139:3507 @140:3537 @141:3427 @142:3500 @143:3532 @144:3562 @145:3592 @146:2869 @147:2815 @148:2819 @149:2949 @150:3622 @151:3803 @152:3351 @153:3601 @154:3792 @155:2934 @156:3316 @157:3390 @158:3483 @159:3520 @160:3550 @161:3580 @162:3511 @163:3541 @164:3571 @165:2941 @166:3610 @167:3796 @168:3329 @169:3403 @170:3491 @171:2930 @172:3308 @173:3382 @174:3476 @175:3506 @176:2813 @177:2895 @178:3536 @179:3566 @180:3596 @181:3789 @182:2896 @183:3424 @184:3499 @185:3531 @186:3561 @187:3591 @188:3621 @189:3585 @190:3615 @191:2811 @192:2948 @193:3348 @194:3802 @195:2943 @196:3335 @197:3798 @198:3409 @199:3525 @200:3555 @201:3495 @202:3523 @203:3553 @204:3583 @205:3613 @206:2810 @207:3589 @208:2808 @209:2942 @210:3332 @211:3797 @212:3406 @213:3800 @214:3341 @215:3415 @216:3497 @217:3529 @218:3559 @219:3352 @220:2805 @221:2950 @222:2806 @223:2946 @224:3619 @225:3623 @226:3804 @227:3533 @228:3563 @229:3428 @230:3501 @231:3498 @232:3530 @233:3560 @234:3590 @235:3620 @236:3593 @237:2944 @238:2803 @239:2947 @240:3346 @241:3801 @242:3422 @243:3338 @244:3412 @245:3527 @246:3557 @247:3587 @248:2802 @249:2800 @250:2936 @251:3318 @252:3392 @253:3617 @254:3799 @255:3793 @256:3485 @257:3513 @258:3543 @259:3573 @260:3603 @261:3805 @262:2818 @263:2951 @264:3354 @265:3430 @266:3502 @267:3487 @268:3516 @269:3534 @270:3564 @271:3594 @272:3624 @273:3606 @274:3794 @275:2817 @276:2938 @277:3322 @278:3396 @279:3314 @280:3388 @281:3482 @282:2816 @283:3546 @284:3576 @285:3791 @286:3510 @287:3540 @288:3570 @289:3600 @290:2933 @291:2814 @292:2932 @293:3312 @294:3386 @295:3480 @296:3509 @297:3488 @298:3517 @299:3539 @300:3569 @301:3599 @302:3790 @303:3607 @304:3795 @305:2801 @306:2939 @307:3324 @308:3398 @309:2862 @310:2901 @311:2553 @312:2891 @313:3547 @314:3577 @315:2898 @316:2974 @317:2861 @318:2899 @319:2977 @320:2991 @321:2866 @322:3302 @323:2859 @324:2897 @325:2973 @326:2860 @327:3059 @328:2563 @329:2858 @330:2900 @331:2978 @332:2562 @333:2838 @334:2566 @335:2839 @336:3062 @337:2565 @338:2856 @339:2990 @340:2550 @341:2863 @342:2968 @343:3048 @344:2564 @345:2799 @346:3063 @347:2826 @348:2987 @349:2830 @350:3065 @351:3050 @352:2832 @353:2980 @354:3055 @355:2988 @356:3064 @357:3052 @358:2893 @359:2985 @360:3058 @361:2821 @362:2971 @363:3054 @364:2981 @365:3056 @366:2892 @367:2825 @368:2975 @369:2554 @370:2846 @371:2972 @372:3051 @373:2823 @374:2979 @375:2843 @376:3305 @377:2890 @378:2913 @379:2970 @380:3049 @381:2966 @382:3251 @383:2556 @384:2874 @385:2552 @386:2795 @387:2982 @388:3057 @389:2557 @390:2835 @391:3290 @392:2889 @393:2986 @394:2849 @395:2827 @396:2976 @397:3053 @398:2820 @399:2929 @400:3306 @401:2829 @402:2989 @403:2558 @404:2836 @405:2798 @406:2847 @407:2793 @408:3261 @409:2560 @410:2837 @411:2551 @412:2928 @413:2848 @414:3111 @415:3163 @416:3288 @417:3167 @418:2904 @419:2911 @420:2912 @421:2828 @422:2983 @423:2548 @424:2844 @425:3217 @426:3255 @427:2924 @428:3081 @429:3141 @430:3142 @431:3164 @432:2840 @433:2687 @434:2886 @435:3233 @436:3304 @437:3740 @438:2855 @439:2926 @440:2969 @441:2842 @442:3047 @443:2967 @444:3295 @445:2809 @446:3219 @447:2925 @448:2927 @449:2965 @450:3046 @451:3106 @452:2549 @453:3156 @454:3107 @455:3116 @456:3122 @457:3131 @458:2872 @459:3262 @460:3140 @461:3143 @462:3153 @463:3154 @464:3155 @465:3565 @466:3157 @467:3158 @468:3159 @469:3218 @470:3232 @471:3673 @472:3289 @473:3366 @474:3475 @475:3505 @476:3535 @477:1615 @478:1626 @479:1666 @480:2154 @481:1913 @482:1551 @483:1553 @484:863 @485:1624 @486:903 @487:1635 @488:1636 @489:1637 @490:1562 @491:1552 @492:1634 @493:1540 @494:1613 @495:1620 @496:1614 @497:1627 @498:1628 @499:1633 @500:1638 @501:835 @502:1617 @503:1579 @504:1618 @505:1619 @506:1542 @507:1591 @508:1576 @509:1616 @510:1629 @511:1630 @512:1631 @513:1670 @514:2429 @515:2158 @516:2169 @517:1675 @518:2419 @519:2519 @520:804 @521:1677 @522:2437 @523:2543 @524:2128 @525:2129 @526:1622 @527:2569 @528:2525 @529:2034 @530:1692 @531:1722 @532:1919 @533:1593 @534:1912 @535:3112 @536:3115 @537:2703 @538:2711 @539:2880 @540:3082 @541:3083 @542:2719 @543:2720 @544:2875 @545:2877 @546:2878 @547:2876 @548:2219 @549:2311 @550:2812 @551:2489 @552:2712 @553:2717 @554:2216 @555:2214 @556:2220 @557:2212 @558:2218 @559:2213 @560:2152 @561:2159 @562:2833 @563:2211 @564:2217 @565:2210 @566:2033 @567:2058 @568:2036 @569:2047 @570:2121 @571:2149 @572:2568 @573:1966 @574:2032 @575:2513 @576:2671 @577:2672 @578:2054 @579:2071 @580:2577 @581:2702 @582:1964 @583:1965 @584:2822 @585:1899 @586:1900 @587:1898 @588:1962 @589:1972 @590:1930 @591:2063 @592:2865 @593:2134 @594:2136 @595:2831 @596:1920 @597:1713 @598:1848 @599:2581 @600:1721 @601:1793 @602:2579 @603:2580 @604:2584 @605:2585 @606:2586 @607:1712 @608:2873 @609:2588 @610:2589 @611:2590 @612:2591 @613:2593 @614:1665 @615:2062 @616:2857 @617:1711 @618:1845 @619:2845 @620:1981 @621:1678 @622:1901 @623:2439 @624:2544 @625:2485 @626:1671 @627:1689 @628:2305 @629:1667 @630:2046 @631:2572 @632:2488 @633:2600 @634:2692 @635:2306 @636:2570 @637:2841 @638:2601 @639:2602 @640:2604 @641:2605 @642:2606 @643:2607 @644:2608 @645:2609 @646:2610 @647:2611 @648:2612 @649:2613 @650:2614 @651:2615 @652:2616 @653:2617 @654:2618 @655:2619 @656:2620 @657:2621 @658:2622 @659:2623 @660:2624 @661:2625 @662:2626 @663:2627 @664:2628 @665:2629 @666:2630 @667:2631 @668:2632 @669:2633 @670:2634 @671:2641 @672:2642 @673:2643 @674:2644 @675:2645 @676:2646 @677:2647 @678:2648 @679:2649 @680:2650 @681:2651 @682:2652 @683:2653 @684:2654 @685:2655 @686:2656 @687:2657 @688:2658 @689:2659 @690:2660 @691:2661 @692:2662 @693:2663 @694:2664 @695:2665 @696:2666 @697:2667 @698:2668 @699:2669 @700:2670 @701:2681 @702:2682 @703:2683 @704:1924 @705:1931 @706:1980 @707:2578 @708:2252 @709:2315 @710:2599 @711:2487 @712:2595 @713:1921 @714:1922 @715:1923 @716:2867 @717:2870 @718:2871 @719:2596 @720:2597 @721:2598 @722:1623 @723:2520 @724:1513 @725:911 @726:918 @727:2603 @728:1688 @729:1641 @730:2571 @731:2885 @732:2902 @733:1642 @734:1643 @735:3110 @736:2903 @737:2907 @738:2910 @739:852 @740:1584 @741:1601 @742:1595 @743:1597 @744:1599 @745:1490 @746:1673 @747:2430 @748:2532 @749:1493 @750:1676 @751:1791 @752:2436 @753:2542 @754:1494 @755:1491 @756:1412 @757:1405 @758:1492 @759:1674 @760:2535 @761:1668 @762:1669 @763:1602 @764:1672 @765:1241 @766:1604 @767:1690 @768:1609 @769:1610 @770:1691 @771:1608 @772:1245 @773:1521 @774:1248 @775:1153 @776:1585 @777:1568 @778:1978 @779:1989 @780:2000 @781:1979 @782:1926 @783:1977 @784:1882 @785:2440 @786:2545 @787:1999 @788:1991 @789:1996 @790:1905 @791:1906 @792:1909 @793:1910 @794:2435 @795:2541 @796:1908 @797:1911 @798:2438 @799:1866 @800:1903 @801:1904 @802:1829 @803:2441 @804:2547 @805:1850 @806:1902 @807:1907 @808:1849 @809:2432 @810:2534 @811:1811 @812:2431 @813:2533 @814:1929 @815:2546 @816:1777 @817:2434 @818:2539 @819:1795 @820:1925 @821:1937 @822:2433 @823:2536 @824:1757 @825:1792 @826:1776 @827:1928 @828:1975 @829:2540 @830:1740 @831:1828 @832:1927 @833:1973 @834:1988 @835:1917 @836:1987 @837:1723 @838:1934 @839:2002 @840:2043 @841:2045 @842:2265 @843:1933 @844:2051 @845:2147 @846:2304 @847:2008 @848:2009 @849:2272 @850:2301 @851:2179 @852:2297 @853:2316 @854:2317 @855:2318 @856:2289 @857:2178 @858:2181 @859:2285 @860:2140 @861:2172 @862:2287 @863:2141 @864:2173 @865:2281 @866:2138 @867:2170 @868:2283 @869:2139 @870:2171 @871:2066 @872:2068 @873:2065 @874:2070 @875:2146 @876:2183 @877:1939 @878:1938 @879:1932 @880:1942 @881:1944 @882:2067 @883:1941 @884:2004 @885:2293 @886:2135 @887:2137 @888:1935 @889:1940 @890:1943 @891:2299 @892:1976 @893:2155 @894:2294 @895:2291 @896:2420 @897:2006 @898:2003 @899:2010 @900:2064 @901:2130 @902:2069 @903:2276 @904:2177 @905:2310 @906:2312 @907:2531 @908:2337 @909:2504 @910:2527 @911:2340 @912:2446 @913:2508 @914:2501 @915:2523 @916:2335 @917:2445 @918:2502 @919:2524 @920:2521 @921:2333 @922:2500 @923:2522 @924:2334 @925:2444 @926:2516 @927:2331 @928:2498 @929:2518 @930:2332 @931:2499 @932:2442 @933:2495 @934:2515 @935:2329 @936:2443 @937:2496 @938:2325 @939:2492 @940:2326 @941:2493 @942:2512 @943:2328 @944:2507 @945:2530 @946:2324 @947:2490 @948:2509 @949:2511 @950:2206 @951:2207 @952:2339 @953:2506 @954:2529 @955:2208 @956:2336 @957:2503 @958:2526 @959:2528 @960:2338 @961:2505 @962:2180 @963:2203 @964:2260 @965:2491 @966:2510 @967:2205 @968:2076 @969:2145 @970:2151 @971:2268 @972:2176 @973:2075 @974:2144 @975:2150 @976:2267 @977:2182 @978:2275 @979:2074 @980:2143 @981:2266 @982:2072 @983:2073 @984:2127 @985:2142 @986:2537 @987:2131 @988:2148 @989:2261 @990:2052 @991:2053 @992:2050 @993:2262 @994:2049 @995:2132 @996:2263 @997:2270 @998:2175 @999:2264 @1000:2271 @1001:2153 @1002:2269 @1003:2274 @1004:2044 @1005:2055 @1006:2309 @1007:2007 @1008:2040 @1009:2041 @1010:2042 @1011:2277 @1012:2313 @1013:2314 @1014:2273 @1015:2302 @1016:2298 @1017:2292 @1018:2280 @1019:2300 @1020:2295 @1021:2288 @1022:2290 @1023:2296 @1024:2282 @1025:2284 @1026:2278 @1027:2286 @1028:2279 @1029:2686 @1030:2691 @1031:2710 @1032:2684 @1033:2701 @1034:2699 @1035:2700 @1036:2698 @1037:2709 @1038:2694 @1039:2697 @1040:2693 @1041:2685 @1042:2637 @1043:2594 @1044:2690 @1045:2636 @1046:2708 @1047:2705 @1048:2706 @1049:2707 @1050:2695 @1051:2696 @1052:2688 @1053:2689 @1054:2675 @1055:2678 @1056:2640 @1057:2638 @1058:2674 @1059:2677 @1060:2679 @1061:2639 @1062:2486 @1063:2635 @1064:3925 @1065:3926 @1066:3892 @1067:3894 @1068:3896 @1069:3915 @1070:3917 @1071:3924 @1072:3770 @1073:3773 @1074:3883 @1075:3887 @1076:3888 @1077:3890 @1078:3724 @1079:3725 @1080:3729 @1081:3761 @1082:3764 @1083:3765 @1084:3687 @1085:3712 @1086:3713 @1087:3714 @1088:3715 @1089:3694 @1090:3454 @1091:3282 @1092:3283 @1093:3284 @1094:3456 @1095:3458 @1096:3264 @1097:3268 @1098:3279 @1099:3280 @1100:3281 @1101:3359 @1102:3215 @1103:3235 @1104:3238 @1105:3239 @1106:3265 @1107:3263 @1108:3843 @1109:3634 @1110:3786 @1111:3787 @1112:3788 @1113:3632 @1114:3860 @1115:3861 @1116:3862 @1117:3808 @1118:3809 @1119:3810 @1120:3866 @1121:3867 @1122:3868 @1123:3857 @1124:3858 @1125:3859 @1126:3870 @1127:3871 @1128:3872 @1129:3863 @1130:3864 @1131:3865 @1132:3626 @1133:3639 @1134:3631 @1135:3651 @1136:3627 @1137:3869 @1138:3649 @1139:3652 @1140:3648 @1141:3642 @1142:3644 @1143:3650 @1144:3629 @1145:3630 @1146:3635 @1147:3653 @1148:3633 @1149:3646 @1150:3250 @1151:3254 @1152:3474 @1153:3113 @1154:3166 @1155:3636 @1156:3755 @1157:3775 @1158:3776 @1159:3777 @1160:3378 @1161:3739 @1162:3686 @1163:3780 @1164:3778 @1165:3779 @1166:3628 @1167:3438 @1168:3918 @1169:3922 @1170:3906 @1171:3907 @1172:3908 @1173:3909 @1174:3731 @1175:3733 @1176:3734 @1177:3735 @1178:3736 @1179:3771 @1180:3693 @1181:3718 @1182:3885 @1183:3726 @1184:3727 @1185:3728 @1186:3944 @1187:3941 @1188:4028 @1189:3997 @1190:4024 @1191:4008 @1192:4017 @1193:3996 @1194:4015 @1195:4014 @1196:4013 @1197:4012 @1198:3990 @1199:3984 @1200:3983 @1201:3994 @1202:3993 @1203:3992 @1204:3974 @1205:3973 @1206:3972 @1207:3970 @1208:3968 @1209:3967 @1210:3692 @1211:3956 @1212:3955 @1213:3954 @1214:3958 @1215:3949 @1216:3923 @1217:3927
2022-07-06 18:30:16,291 [P21744/D12/T66] ERROR Umbraco.Web.Scheduling.BackgroundTaskRunner - [ContentVersionCleanup] Task has failed
System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
The statement has been terminated. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) in D:\a\1\s\src\Umbraco.Core\Persistence\FaultHandling\RetryPolicy.cs:line 172
at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
at Umbraco.Core.Persistence.Repositories.Implement.DocumentVersionRepository.DeleteVersions(IEnumerable`1 versionIds) in D:\a\1\s\src\Umbraco.Core\Persistence\Repositories\Implement\DocumentVersionRepository.cs:line 123
at Umbraco.Core.Services.Implement.ContentService.CleanupDocumentVersions(DateTime asAtDate) in D:\a\1\s\src\Umbraco.Core\Services\Implement\ContentService.cs:line 3355
at Umbraco.Core.Services.Implement.ContentService.PerformContentVersionCleanup(DateTime asAtDate) in D:\a\1\s\src\Umbraco.Core\Services\Implement\ContentService.cs:line 3232
at Umbraco.Web.Scheduling.ContentVersionCleanup.PerformRun() in D:\a\1\s\src\Umbraco.Web\Scheduling\ContentVersionCleanup.cs:line 68
at Umbraco.Web.Scheduling.RecurringTaskBase.Run() in D:\a\1\s\src\Umbraco.Web\Scheduling\RecurringTaskBase.cs:line 67
at Umbraco.Web.Scheduling.BackgroundTaskRunner`1.<RunAsync>d__40.MoveNext() in D:\a\1\s\src\Umbraco.Web\Scheduling\BackgroundTaskRunner.cs:line 569
ClientConnectionId:37583191-6192-411a-b537-5f5d1dcb8958
Error Number:-2,State:0,Class:11
Our website had a lot of content version history build up over the years, so we knew that had something to do with it.
We then dug a little deeper in what caused the task to fail. It had to do with a query that iterates the entire database. This caused a major spike in performance and caused a timeout after a couple of minutes. This is from the Umbraco source that caused the issue:
Our solution was to reindex the database (I'm no DB admin so I don't know exactly what we did) and upscale the app service (Azure) temporarily so the query could finish. Since then we had no issues.
Another way to solve this problem is to (temporarily) set a higher timeout for the command to execute. Add Connect Timeout=3000; to your database connection string.
Notice that Connect Timeout is not the same as Command Timeout. It's not possible to set a Command timeout using the connection string so Umbraco will look at Connect Timeout and use that value to programmatically set the Command Timeout.
ContentVersionCleanup task fails
Currently, we have an issue with one of our sites that runs on the latest Umbraco version (8.18.0) that introduced content version cleanup.
The scheduled task fails to execute due to a time-out on the SQL database. We suspect that it's caused by a large number of records that the script is trying to delete and it's timing out because of it. In Umbraco we see this in the logging:
Has anyone run into this issue? The project is around for quite some time, so there is a lot of page history present in the database. Maybe that's causing issues?
I'm also having trouble getting the content cleanup to work on v8.18.3. Have you had any luck since you posted this? I turned on debug logging to see if I could find more info, but I'm not having much luck determining where the timeout is occurring. I think it may be timing out trying to obtain a write lock, but I'm not very confident that I understand how the locking works. I added the logs below in case someone sees something obvious that I missed.
Our website had a lot of content version history build up over the years, so we knew that had something to do with it.
We then dug a little deeper in what caused the task to fail. It had to do with a query that iterates the entire database. This caused a major spike in performance and caused a timeout after a couple of minutes. This is from the Umbraco source that caused the issue:
Our solution was to reindex the database (I'm no DB admin so I don't know exactly what we did) and upscale the app service (Azure) temporarily so the query could finish. Since then we had no issues.
Hope this helps!
Yes. Thank you so much for responding. I'll give it a shot.
Another way to solve this problem is to (temporarily) set a higher timeout for the command to execute. Add
Connect Timeout=3000;to your database connection string.Notice that
Connect Timeoutis not the same asCommand Timeout. It's not possible to set a Command timeout using the connection string so Umbraco will look atConnect Timeoutand use that value to programmatically set theCommand Timeout.is working on a reply...