Umbraco - Umbraco version 7.7.6 assembly: 1.0.6528.26344 Courier Express - 3.1.6

We have a "dev" and "staging" environment set up as: "admin" on one instance in an elastic beanstalk load balanced (ELB) environment "front end" on one (but elastic up to 4) in ELB

We've used the Umbraco S3 Provider bucket plugin (https://github.com/ElijahGlover/Umbraco-S3-Provider) to set up the "media" folder on a separate S3 bucket (dev and staging in different S3 buckets).

This set up seems to be working well for editing and viewing media in respective environments. However Courier does not play ball..

When we try to Courier a media item between dev and staging we get an error in the UI and in the logs (twice, once from "Umbraco.Courier.Core.TaskManager" and again at the same time from "Umbraco.Courier.Core.BackgroundTaskRunner".

The deployment of the content page fails.

    [Thread 24] Error PackagingTask with id: e84b5190-fd04-4586-a897-eca017369a49
System.Web.Services.Protocols.SoapException: System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.IO.FileNotFoundException: The specified key does not exist. ---> Amazon.S3.AmazonS3Exception: The specified key does not exist. ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (404) Not Found. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.GetResponse()
   at Amazon.Runtime.Internal.HttpRequest.GetResponse()
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpRequest.GetResponse()
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RedirectHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.AmazonServiceClient.Invoke[TRequest,TResponse](TRequest request, IMarshaller`2 marshaller, ResponseUnmarshaller unmarshaller)
   at Amazon.S3.AmazonS3Client.GetObject(GetObjectRequest request)
   at Umbraco.Storage.S3.BucketFileSystem.Execute[T](Func`2 request)
   --- End of inner exception stack trace ---
   at Umbraco.Storage.S3.BucketFileSystem.Execute[T](Func`2 request)
   at Umbraco.Storage.S3.BucketFileSystem.OpenFile(String path)
   at System.Lazy`1.CreateValue()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Lazy`1.get_Value()
   at Umbraco.Courier.Core.DeferredResourceContents.Dispose()
   --- End of inner exception stack trace ---
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)
   at Umbraco.Courier.RepositoryProviders.WebServiceProvider.RepositoryWebservice.EndGetHashes(IAsyncResult asyncResult)
   at Umbraco.Courier.RepositoryProviders.CourierWebserviceRepositoryProvider.GetHashes(ItemIdentifier[] itemIds)
   at Umbraco.Courier.Core.Packaging.RevisionPackaging.GetTargetHashes(IPackagingTarget2 target, IEnumerable`1 ids)
   at Umbraco.Courier.Core.Packaging.RevisionPackaging.PackageBatch()
   at Umbraco.Courier.Core.Packaging.RevisionPackaging.Package()
   at Umbraco.Courier.Core.Tasks.PackagingTask.Run()
   at Umbraco.Courier.Core.BackgroundTaskRunner`1.ConsumeTaskInternal(T task)

When we attempt Courier a content page that contains a new image, then we get no errors in the logs on the source, the Courier screen reports a successful deployment and the page is transferred/amended successfully.

However on the destination, we get the image missing (the content container for the image is present, but no actual image, nothing is produced in the S3 bucket and furthermore there is an error in Umbraco Tracelog as follows:

An unhandled exception occurred
System.Net.WebException: The remote server returned an error: (403) Forbidden.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at ImageProcessor.Web.Helpers.RemoteFile.<GetWebResponseAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at ImageProcessor.Web.Services.CloudImageService.<GetImage>d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at ImageProcessor.Web.HttpModules.ImageProcessingModule.<ProcessImageAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar)
   at System.Web.HttpApplication.AsyncEventExecutionStep.OnAsyncEventCompletion(IAsyncResult ar)

I've successfully used Courier many times with either a local "media" folder, or a virtual folder, and I suspect there's still a permissions issue somewhere, but we've opened up all the permissions we think we can to enable communication between the different parts of the process.

I've seen another forum post (https://our.umbraco.org/forum/umbraco-courier/85114-deploying-media-with-courier-and-umbracofilesystemprovidersazure) where someone resolved in Azure it by adding some custom code, but I'm hoping not to have to do this (essentially fixing Courier's shortcomings)

Can anyone help with any more suggested lines of investigation, examples where this scenario has worked in the past etc?