We have added Courier 3.1.9 to an Umbraco 7.15.6 site (site originally started on 7.2.5). Courier works as expected when deploying individual nodes or some parts of the site with child items.

However, we are trying to deploy a node with child items and it always fails with a generic error "Something went wrong" with the Exception Message simply being "There was an error processing the request".

Following the network calls it gets through a few documents and finally appears to get stuck on one with the state being "Packaging: Document xxxxxxxxxxxx Property Data", Progress 91, it returns the same data for many consecutive calls. I thought it was timing out, but the time until it fails varies. I have also attempted to deploy just the node it gets stuck on, that works (albeit doesn't have any changes).

The node we are trying to deploy with children has about 360 child nodes and each of them has a multiple media picker with ~30 images picked (among other properties). The environments are both using (separate) Azure Blob Storage for media and are hosted on VPS. The environments were in sync when we installed Courier.

Digging deeper into CourierTraceLog on the SOURCE has the following lines:

2020-11-02 17:53:05,032 [95] INFO  Umbraco.Courier.Core.TaskManager - [Thread 95] Added PackagingTask behind queue of : 0
2020-11-02 17:53:05,032 [95] INFO  Umbraco.Courier.Core.BackgroundTaskRunner - [Thread 95]  Task added Umbraco.Courier.Core.Tasks.PackagingTask
2020-11-02 17:53:05,032 [73] INFO  Umbraco.Courier.Core.TaskManager - [Thread 73] Starting PackagingTask with id: d4d68447-e25c-41b5-b8ba-5ce3fff56c41
2020-11-02 17:53:05,032 [73] INFO  Umbraco.Courier.Core.Tasks.PackagingTask - [Thread 73] PackagingTask running by user admin
2020-11-02 17:53:11,000 [73] INFO  Umbraco.Courier.Core.Packaging.RevisionPackaging - [Thread 73] Documents: ad31b33e-6976-4de4-8d46-d30f0975fa91 hash-mismatch (local/remote) 58eb73ae4551e0e4c9a90c96aa1cf14aa68d72c5 / c84fefed48ae2482a3080201dd867aa4cbe06652
2020-11-02 17:54:34,141 [73] DEBUG Umbraco.Courier.Persistence.V6.NHibernate.NHibernateProvider - [Thread 73] Closing session, UseGlobalSession: False, key: 925a7050-6a4d-4774-89ac-50d367fb9ff0
2020-11-02 17:54:34,188 [73] ERROR Umbraco.Courier.Core.TaskManager - [Thread 73] Error PackagingTask with id: d4d68447-e25c-41b5-b8ba-5ce3fff56c41
System.Net.WebException: The request was aborted: The request was canceled.
   at System.Web.Services.Protocols.WebClientAsyncResult.WaitForResponse()
   at System.Web.Services.Protocols.WebClientProtocol.EndSend(IAsyncResult asyncResult, Object& internalAsyncState, Stream& responseStream)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)
   at Umbraco.Courier.RepositoryProviders.WebServiceProvider.RepositoryWebservice.EndGetHashes(IAsyncResult asyncResult)
   at Umbraco.Courier.RepositoryProviders.CourierWebserviceRepositoryProvider.GetHashes(ItemIdentifier[] itemIds)
   at Umbraco.Courier.Core.Packaging.RevisionPackaging.GetTargetHashes(IPackagingTarget2 target, IEnumerable`1 ids)
   at Umbraco.Courier.Core.Packaging.RevisionPackaging.PackageBatch()
   at Umbraco.Courier.Core.Packaging.RevisionPackaging.Package()
   at Umbraco.Courier.Core.Tasks.PackagingTask.Run()
   at Umbraco.Courier.Core.BackgroundTaskRunner`1.ConsumeTaskInternal(T task)
2020-11-02 17:54:34,188 [73] ERROR Umbraco.Courier.Core.BackgroundTaskRunner - [Thread 73] An error occurred consuming task
System.Net.WebException: The request was aborted: The request was canceled.
   at System.Web.Services.Protocols.WebClientAsyncResult.WaitForResponse()
   at System.Web.Services.Protocols.WebClientProtocol.EndSend(IAsyncResult asyncResult, Object& internalAsyncState, Stream& responseStream)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)
   at Umbraco.Courier.RepositoryProviders.WebServiceProvider.RepositoryWebservice.EndGetHashes(IAsyncResult asyncResult)
   at Umbraco.Courier.RepositoryProviders.CourierWebserviceRepositoryProvider.GetHashes(ItemIdentifier[] itemIds)
   at Umbraco.Courier.Core.Packaging.RevisionPackaging.GetTargetHashes(IPackagingTarget2 target, IEnumerable`1 ids)
   at Umbraco.Courier.Core.Packaging.RevisionPackaging.PackageBatch()
   at Umbraco.Courier.Core.Packaging.RevisionPackaging.Package()
   at Umbraco.Courier.Core.Tasks.PackagingTask.Run()
   at Umbraco.Courier.Core.BackgroundTaskRunner`1.ConsumeTaskInternal(T task)

CourierTraceLog on the DESTINATION has nothing useful:

2020-11-02 17:53:05,078 [145] INFO  Umbraco.Courier.RepositoryProviders.Webservices.Repository - [Thread 145] OpenSession :: Source Info: Source User - 0, Source IP - xxx.xxx.xxx.xxx
2020-11-02 17:53:05,078 [145] INFO  Umbraco.Courier.Core.GlobalExecutionContextManager - [Thread 145] A session Id has been set: 925a7050-6a4d-4774-89ac-50d367fb9ff0
2020-11-02 17:54:34,141 [145] INFO  Umbraco.Courier.RepositoryProviders.Webservices.Repository - [Thread 145] CloseSession :: Source Info: Source User - 0, Source IP - xxx.xxx.xxx.xxx
2020-11-02 17:54:34,141 [145] DEBUG Umbraco.Courier.Persistence.V6.NHibernate.NHibernateProvider - [Thread 145] Closing session, UseGlobalSession: True, key: 925a7050-6a4d-4774-89ac-50d367fb9ff0
2020-11-02 17:54:34,141 [145] INFO  Umbraco.Courier.Persistence.V6.NHibernate.NHibernateGlobalSessionManager - [Thread 145] CloseSession 925a7050-6a4d-4774-89ac-50d367fb9ff0, implicit roll back? True
2020-11-02 17:54:34,141 [145] INFO  Umbraco.Courier.Core.GlobalExecutionContextManager - [Thread 145] Ending session: 925a7050-6a4d-4774-89ac-50d367fb9ff0
2020-11-02 17:54:34,141 [145] DEBUG Umbraco.Courier.Persistence.V6.NHibernate.NHibernateProvider - [Thread 145] Closing session, UseGlobalSession: True, key: 925a7050-6a4d-4774-89ac-50d367fb9ff0

IIS log on the DESTNATION shows a 500 error, but Event Viewer has no details, guessing because the exception is caught somehow:

2020-11-02 04:54:33 xxx.xxx.xxx.xxx POST /umbraco/plugins/courier/webservices/repository.asmx fromExternalClient=true 443 - xxx.xxx.xxx.xxx Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+4.0.30319.42000) - 500 0 64 50 277322 80265

At this point I'm at a loss. I know Courier is generally able to deploy a number of nodes at once, as we regularly do on other sites. In the current state it's useless for our client on this site though. Any pointers appreciated.