Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Alan Hill 14 posts 57 karma points
    Dec 04, 2017 @ 11:29
    Alan Hill
    0

    Courier failing to deploy images in AWS S3 bucket

    Umbraco - Umbraco version 7.7.6 assembly: 1.0.6528.26344 Courier Express - 3.1.6

    We have a "dev" and "staging" environment set up as: "admin" on one instance in an elastic beanstalk load balanced (ELB) environment "front end" on one (but elastic up to 4) in ELB

    We've used the Umbraco S3 Provider bucket plugin (https://github.com/ElijahGlover/Umbraco-S3-Provider) to set up the "media" folder on a separate S3 bucket (dev and staging in different S3 buckets).

    This set up seems to be working well for editing and viewing media in respective environments. However Courier does not play ball..

    When we try to Courier a media item between dev and staging we get an error in the UI and in the logs (twice, once from "Umbraco.Courier.Core.TaskManager" and again at the same time from "Umbraco.Courier.Core.BackgroundTaskRunner".

    The deployment of the content page fails.

        [Thread 24] Error PackagingTask with id: e84b5190-fd04-4586-a897-eca017369a49
    System.Web.Services.Protocols.SoapException: System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.IO.FileNotFoundException: The specified key does not exist. ---> Amazon.S3.AmazonS3Exception: The specified key does not exist. ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (404) Not Found. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
       at System.Net.HttpWebRequest.GetResponse()
       at Amazon.Runtime.Internal.HttpRequest.GetResponse()
       --- End of inner exception stack trace ---
       at Amazon.Runtime.Internal.HttpRequest.GetResponse()
       at Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext executionContext)
       at Amazon.Runtime.Internal.RedirectHandler.InvokeSync(IExecutionContext executionContext)
       at Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext)
       at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync(IExecutionContext executionContext)
       at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
       --- End of inner exception stack trace ---
       at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception)
       at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
       at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
       at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
       at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext)
       at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
       at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
       at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeSync(IExecutionContext executionContext)
       at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext executionContext)
       at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext)
       at Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext executionContext)
       at Amazon.Runtime.AmazonServiceClient.Invoke[TRequest,TResponse](TRequest request, IMarshaller`2 marshaller, ResponseUnmarshaller unmarshaller)
       at Amazon.S3.AmazonS3Client.GetObject(GetObjectRequest request)
       at Umbraco.Storage.S3.BucketFileSystem.Execute[T](Func`2 request)
       --- End of inner exception stack trace ---
       at Umbraco.Storage.S3.BucketFileSystem.Execute[T](Func`2 request)
       at Umbraco.Storage.S3.BucketFileSystem.OpenFile(String path)
       at System.Lazy`1.CreateValue()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Lazy`1.get_Value()
       at Umbraco.Courier.Core.DeferredResourceContents.Dispose()
       --- End of inner exception stack trace ---
       at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
       at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)
       at Umbraco.Courier.RepositoryProviders.WebServiceProvider.RepositoryWebservice.EndGetHashes(IAsyncResult asyncResult)
       at Umbraco.Courier.RepositoryProviders.CourierWebserviceRepositoryProvider.GetHashes(ItemIdentifier[] itemIds)
       at Umbraco.Courier.Core.Packaging.RevisionPackaging.GetTargetHashes(IPackagingTarget2 target, IEnumerable`1 ids)
       at Umbraco.Courier.Core.Packaging.RevisionPackaging.PackageBatch()
       at Umbraco.Courier.Core.Packaging.RevisionPackaging.Package()
       at Umbraco.Courier.Core.Tasks.PackagingTask.Run()
       at Umbraco.Courier.Core.BackgroundTaskRunner`1.ConsumeTaskInternal(T task)
    

    When we attempt Courier a content page that contains a new image, then we get no errors in the logs on the source, the Courier screen reports a successful deployment and the page is transferred/amended successfully.

    However on the destination, we get the image missing (the content container for the image is present, but no actual image, nothing is produced in the S3 bucket and furthermore there is an error in Umbraco Tracelog as follows:

    An unhandled exception occurred
    System.Net.WebException: The remote server returned an error: (403) Forbidden.
       at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
       at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at ImageProcessor.Web.Helpers.RemoteFile.<GetWebResponseAsync>d__17.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at ImageProcessor.Web.Services.CloudImageService.<GetImage>d__16.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at ImageProcessor.Web.HttpModules.ImageProcessingModule.<ProcessImageAsync>d__34.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar)
       at System.Web.HttpApplication.AsyncEventExecutionStep.OnAsyncEventCompletion(IAsyncResult ar)
    

    I've successfully used Courier many times with either a local "media" folder, or a virtual folder, and I suspect there's still a permissions issue somewhere, but we've opened up all the permissions we think we can to enable communication between the different parts of the process.

    I've seen another forum post (https://our.umbraco.org/forum/umbraco-courier/85114-deploying-media-with-courier-and-umbracofilesystemprovidersazure) where someone resolved in Azure it by adding some custom code, but I'm hoping not to have to do this (essentially fixing Courier's shortcomings)

    Can anyone help with any more suggested lines of investigation, examples where this scenario has worked in the past etc?

  • Krishnan V 5 posts 75 karma points
    Nov 12, 2019 @ 16:52
    Krishnan V
    0

    Hi,

    I'm experiencing the same issue. Alan: Can you share your findings please ?

    Thanks, Kris

  • Jamie Attwood 141 posts 354 karma points
    Dec 03, 2019 @ 18:27
    Jamie Attwood
    0

    Me too! Have you files a bug report on this one? This is critical for us.

  • Krishnan V 5 posts 75 karma points
    Jan 03, 2020 @ 15:52
    Krishnan V
    0

    No Jamie Attwood

  • Jamie Attwood 141 posts 354 karma points
    Jan 04, 2020 @ 18:48
    Jamie Attwood
    0

    Contacted HQ and this is not supported by Courier but it's supposed to be introduced in the release of a new product in Q1 2020. There is mention of it here (Umbraco Deploy on Premises): https://umbraco.com/products/roadmap/ I am assuming that the new version of Courier will be for V8 only so those that have not crossed over to the new V8 will be SOL.

    There is a post here: https://our.umbraco.com/forum/umbraco-courier/85114-deploying-media-with-courier-and-umbracofilesystemprovidersazure on how to create triggers on your local and override media create/save events on your destination environments that seems to work.

Please Sign in or register to post replies

Write your reply to:

Draft