Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Hugo 12 posts 147 karma points
    Apr 11, 2017 @ 10:04
    Hugo
    0

    Deploying media with Courier and UmbracoFileSystemProviders.Azure

    We use Courier to deploy content changes from our staging environment to our production environment. Ideally we wouldn't have to do any content changes in production directly.

    We also use a cdn to host our images. We've installed the UmbracoFileSystemProviders.Azure package, which hooks up the media folder to blob storage, which serves as the source for the cdn.

    Now this is where it gets tricky: Courier will detect any dependant media fine, and deploy it to production. As expected it gets the content from the storage account, and sends it over. But the media is not picked up on the receiving end, courier puts it in the media folder, and I can see the files there, but they're not being picked up by the filesystemprovider.

    When I manually upload media to the production website, it works as expected. It appears Courier is bypassing some code by putting the images directly on the filesystem, instead of going through the channels you would when manually uploading.

    It appears I'll have to implement some custom code in between. Also because I'll have to rename the folders that contain the media, because of different id's (I'm pretty sure the folder name is not the same as media id, just an increment, but they don't line up exactly) but I can probably work around that issue.

    I've looked through some documentation and blog posts, but I'm not sure where to begin. There's a lot of confusing lingo and not a lot of documentation, so I'm getting a bit stuck.

    Anybody here who has some ideas or can point me in the right direction? Would be much appreciated.

    Hugo

  • Hugo 12 posts 147 karma points
    Apr 25, 2017 @ 09:09
    Hugo
    110

    Well, I solved it. It was an extremely tiresome process of trial and error trying to figure out how it all works without any proper documentation.

    In short: I hook up to the ExtractedItemResources event, which fires whenever courier finds some media it needs to send (among other things). This happens on the sending side. In this event I create a custom event, a couriercontext event, which will be received by code in the receiving courier site.

    The receiving website listens to my custom event and executes some code to upload the media that courier dropped in the media folder to the azure storage account.

    That's the gist of it, now some code that shows my implementation of it.

    This is run on the sending side:

     public class MediaEventTrigger : ApplicationEventHandler
    {
        protected override void ApplicationStarted(UmbracoApplicationBase umbracoApplication, ApplicationContext applicationContext)
        {
            ExtractionManagerEvents.ExtractedItemResources += ExtractedItemResources;
        }
    
        private void ExtractedItemResources(object sender, ItemEventArgs e)
        {
            var provider = sender as RevisionExtraction;
            //make sure the media event only fires for media items, not other resources
            if (e.Item.ItemId.ProviderId == ItemProviderIds.mediapropertyDataItemProviderGuid)
            {
                var media = e.Item.Dependencies.Single(i => i.ItemId.ProviderId == ItemProviderIds.mediaItemProviderGuid);
                provider.Context.ExecuteEvent("QueueCourierMediaEvent", e.ItemId, new SerializableDictionary<string, string>());
            }
        }
    }
    

    There are a lot of events in ExtractionManagerEvents that you can use, but I was only able to trigger around 2 or 3 of them. The general postprocessing ones and this ExtractedItemResources one. I read somewhere this is because I'm deploying content directly, instead of using revisions.

    Now you can either queue the courier event for later handling or execute it right away. Because I am updating media items in the database to reflect their new location, I have to wait for the deployment to be done. It will not save otherwise but it will also not throw any exceptions. For queueing you can specify a queue, e.g. "DeploymentComplete" queue, but I noticed this does not work in my setup. The sender seems unable to queue anything for the receiver to handle. Executing directly works fine however, which is what I'm doing in above example. The receiver has to delay the event until the deployment is done. Fortunately the receiver can in fact use the queues as intended, so after receiving the first event I just queue another:

     public class MediaEventListener : ItemEventProvider
    {
        public override string Alias
        {
            get
            {
                return "QueueCourierMediaEvent";
            }
        }
    
        public override void Execute(ItemIdentifier itemId, SerializableDictionary<string, string> parameters)
        {
            ExecutionContext.QueueEvent("UpdateMediaPath", itemId, parameters, Umbraco.Courier.Core.Enums.EventManagerSystemQueues.DeploymentComplete);
        }
    }
    
    public class MediaEventQueueListener : ItemEventProvider
    {
        public override string Alias
        {
            get
            {
                return "UpdateMediaPath";
            }
        }
    
        public override void Execute(ItemIdentifier itemId, SerializableDictionary<string, string> parameters)
        { /*code here */ }
    

    MediaEventQueueListener.Execute method will fire once for every media that was transfered with courier. It has to read the file on the filesystem, save it to azure storage, update the media path in umbraco and delete the file on the filesystem.

    The folder structure of media items is messy. It looks like each media file is a folder of its ID, e.g. "/media/1234/image.jpg", but it's a different id. The image could have an umbraco id of 2352 and be put in a folder called 1824. In my case I think only the first few media items line up and the rest doesn't.

    So to determine what folder you have to put the image you have to read out the current highest folder in your storage account. That's what I did at first. Then I changed my mind and used the item guid as folder name. This helps updating cached content, which is a bit out of scope for this question but is also a huge problem when transferring media with courier.

    Here's the code for my execute function. It's a bit messy because sometimes I serialize the json and other times I just regex my way out of it:

    public override void Execute(ItemIdentifier itemId, SerializableDictionary<string, string> parameters)
        {
    
            var fs = FileSystemProviderManager.Current.GetFileSystemProvider<MediaFileSystem>();
            var ms = ApplicationContext.Current.Services.MediaService;
    
            var media = ms.GetById(Guid.Parse(itemId.Id));
            //the umbracofile property contains the src for the image, which at the beginning is the location of the file on the filesystem.
            var umbracoFileJson = JsonConvert.DeserializeObject<UmbracoFile>(media.GetValue<string>("umbracoFile"));
            var oldPath = $"~{umbracoFileJson.src}";
    
            var folder = itemId.Id;
            var filename = Path.GetFileName(oldPath);
            var path = $"/media/{folder}/{filename}"; //what will be the new path
            var absolutePath = HttpContext.Current.Server.MapPath(oldPath); //path to file on filesystem
            using (var fileStream = System.IO.File.OpenRead(absolutePath))
            {
                fs.AddFile(path, fileStream); //saving the file to storage
            }
            Directory.Delete(Path.GetDirectoryName(absolutePath), true);
    
            //now to update umbraco database
            var umbracoFile = media.GetValue("umbracoFile");
            var newUmbracoFile = ReplaceFolder(media.GetValue<string>("umbracoFile"), folder);
            media.SetValue("umbracoFile", newUmbracoFile);
            media.UpdateDate = DateTime.UtcNow;
            media.CreateDate = DateTime.UtcNow;
            ms.Save(media);
    
        }
    
        private string ReplaceFolder(string umbracoFile, string newFolder)
        {
            return Regex.Replace(umbracoFile, @"(?<=/media/)\d{4,5}", newFolder);
        }
    

    When this is done the media is transferred properly to the other side, but as I mentioned before the content is not always updated. I use a lot of grid layouts in my website which need to be updated with the new path, because that doesn't happen automatically.

    I inherit PropertyDataResolverProvider to create a class that can execute custom code when packaging editors with the "Umbraco.Grid" alias. Then for each media item I find I replace its folder with its guid. In case of RTE's with media you also have to add the media item as a dependency to the contentPropertyData, otherwise it will not get shipped.

  • Gordon Saxby 1444 posts 1855 karma points
    Oct 09, 2020 @ 14:18
    Gordon Saxby
    0

    It's a shame it took me so long to find this post - as it solved my problem too! (well, actually, Courier's problem, not mine!!).

    Courier does not create the cmsMedia record for Images, which means they appear in the Media Library OK, but do not appear properly in media pickers nor do they show a thumbnail on pages where the image has been selected.

    Creating a cmsMedia record is easy enough, but you need the versionId from cmsPropertyData ... which isn't available until Courier has finished (committed) the deployment.

    This technique allowed me to fire an event on the destination server and "fix" all images by creating the necessary cmsMedia records :-)

  • Barry Fogarty 493 posts 1129 karma points
    May 04, 2017 @ 16:12
    Barry Fogarty
    0

    Thank you for your detailed account for this problem Hugo. I had this exact problem and your guide helped enormously. I appreciate your efforts! #H5YR

  • Jay 413 posts 639 karma points
    May 22, 2017 @ 20:40
    Jay
    0

    Hey Hugo,

    I'm trying to do something similar. But I need to trigger reindexing on the receiving end.

    When you are calling a queue on your sending end with

    provider.Context.ExecuteEvent("QueueCourierMediaEvent", e.ItemId, new SerializableDictionary<string, string>());
    

    Do they hit the receiving end's MediaEventListener class of below right away?

    public class MediaEventListener : ItemEventProvider
    {
        public override string Alias
        {
            get
            {
                return "QueueCourierMediaEvent";
            }
        }
    
        public override void Execute(ItemIdentifier itemId, SerializableDictionary<string, string> parameters)
        {
            ExecutionContext.QueueEvent("UpdateMediaPath", itemId, parameters, Umbraco.Courier.Core.Enums.EventManagerSystemQueues.DeploymentComplete);
        }
    }
    
  • Hugo 12 posts 147 karma points
    May 23, 2017 @ 05:51
    Hugo
    1

    Hi JLon,

    That is correct. When the sending end executes that event, the receiving end code fires right away.

    Because my goal was to execute code after the deployment, and queueing events didn't seem possible from the sending end, I had to loop it through the receiver like that.

    I tested it by running my website locally, then starting remote debugging on the remote machine which ran the same dll's. Locally it would not hit a breakpoint in the MediaEventListener class, but remotely it would.

  • Jay 413 posts 639 karma points
    May 23, 2017 @ 10:02
    Jay
    0

    Thanks Hugo, i'll try it out.

    It's so difficult to find out about things especially with Courier where there's not much proper documentation about it :(

    Spent so much time digging around here and there

  • Jay 413 posts 639 karma points
    May 23, 2017 @ 12:31
    Jay
    0

    oh Hugo, just wanted to say you saved my day :D

    It works like a charm. I've wasted like 2 days to try to figure out how to trigger / detect the events from the receiving end. Can't find anything relevant at all

  • Hugo 12 posts 147 karma points
    May 23, 2017 @ 14:20
    Hugo
    0

    That's great JLon! I know your struggle, I had spend a week testing, figuring it all out, and trying to make it work.

    Actually I recognize your username now, it was on a bunch of related threads I came across trying to find a solution. There were others who popped up again and again on similar threads on here and github, hunting for fixes and documentation. It was the reason I made this post.

    Glad I could be of help :)

  • Jay 413 posts 639 karma points
    May 23, 2017 @ 16:46
    Jay
    0

    Oh yeah I've been posting it everywhere. i've even raised on the Courier Issues site too but got nothing back :(

    I've even tried to upgrade courier to the latest version but the issue is still there.

  • Jordan 24 posts 182 karma points
    Feb 07, 2018 @ 10:19
    Jordan
    2

    Thanks for this awnser! this really helped me setting something up a solution in our project. From what I can tell this problem still exists in the latest version of Courier (3.1.6), Umbraco(7.7.9) and UmbracoFileSystemProviders.Azure(1.0.2).

    Below I made some tweaks to folder naming (issues on large sites with lots of media) Also tweaked the Model used for deserialzing the json (needed for when image cropper is in use)

        public override void Execute(ItemIdentifier itemId, SerializableDictionary<string, string> parameters)
        {
            var fs = FileSystemProviderManager.Current.GetFileSystemProvider<MediaFileSystem>();
            var ms = ApplicationContext.Current.Services.MediaService;
    
            var media = ms.GetById(Guid.Parse(itemId.Id));
            //the umbracofile property contains the src for the image, which at the beginning is the location of the file on the filesystem.
            var umbracoFileJson = JsonConvert.DeserializeObject<ImageCropDataSet>(media.GetValue<string>("umbracoFile")); //ImageCropDataSet used to deserialize the json.
            var oldPath = $"~{umbracoFileJson.Src}";
    
            var folder = itemId.Id;
            var filename = Path.GetFileName(oldPath);
            var path = $"/media/{folder}/{filename}"; //what will be the new path
            var absolutePath = HttpContext.Current.Server.MapPath(oldPath); //path to file on filesystem
            using (var fileStream = System.IO.File.OpenRead(absolutePath))
            {
                fs.AddFile(path, fileStream); //saving the file to storage
            }
            Directory.Delete(Path.GetDirectoryName(absolutePath), true);
    
            //now to update umbraco database
            var umbracoFile = media.GetValue("umbracoFile");
            var newUmbracoFile = ReplaceFolder(media.GetValue<string>("umbracoFile"), folder);
            media.SetValue("umbracoFile", newUmbracoFile);
            media.UpdateDate = DateTime.UtcNow;
            media.CreateDate = DateTime.UtcNow;
            ms.Save(media);
        }
    
        internal string ReplaceFolder(string umbracoFile, string newFolder)
        {
            return Regex.Replace(umbracoFile, @"(?<=/media/)\d{4,10}", newFolder); //5 has been changed to 10, to allow up to 10 degits to be selected
        }
    

    ImageCropDataSet is located in the Umbraco.Web.Models namespace.

    Hope this helps someone else that hits this problem.

  • Matt Barlow | jacker.io 164 posts 740 karma points c-trib
    Dec 20, 2018 @ 15:29
    Matt Barlow | jacker.io
    0

    Hugo's solution combined with the execute method from Jordan worked for me. (Umbraco 7.12, Courier 3.1.5, Umbraco.Storage.S3 1.0.31)

    Thanks guys for the hard work to figure this out!

    Can also confirm that this works with the Umbraco.Storage.S3 file system provider.

  • Jamie Attwood 201 posts 493 karma points c-trib
    Dec 03, 2019 @ 19:18
    Jamie Attwood
    0

    Matt, we are desperately trying to make courier work on S3! Would you mind posting your solution here? We are using the S3 media providers and also the S3 image cache solution for the crops.

    Hugo, huge help - thanks-you as well.

    Big question is where is HQ on this? With a big price tag on this product and most deployments being made in the cloud and decentralized CDN media locations - (including Umbraco Cloud). At the very least this should be on the todo list...

  • Matt Barlow | jacker.io 164 posts 740 karma points c-trib
    Dec 04, 2019 @ 09:21
    Matt Barlow | jacker.io
    1

    Hi Jamie,

    Take Hugo's code and replace the execute method with Jordan's.

    See here:

    using System;
    using System.IO;
    using System.Text.RegularExpressions;
    using System.Web;
    using Newtonsoft.Json;
    using Umbraco.Core;
    using Umbraco.Core.IO;
    using Umbraco.Courier.Core;
    using Umbraco.Courier.Core.ProviderModel;
    using Umbraco.Web.Models;
    
    namespace Cms.Core.Events
    {
        public class MediaEventListener : ItemEventProvider
        {
            public override string Alias
            {
                get { return "QueueCourierMediaEvent"; }
            }
    
            public override void Execute(ItemIdentifier itemId, SerializableDictionary<string, string> parameters)
            {
                ExecutionContext.QueueEvent("UpdateMediaPath", itemId, parameters,
                    Umbraco.Courier.Core.Enums.EventManagerSystemQueues.DeploymentComplete);
            }
    
            public class MediaEventQueueListener : ItemEventProvider
            {
                public override string Alias
                {
                    get { return "UpdateMediaPath"; }
                }
    
                public override void Execute(ItemIdentifier itemId, SerializableDictionary<string, string> parameters)
                {
                    var fs = FileSystemProviderManager.Current.GetFileSystemProvider<MediaFileSystem>();
                    var ms = ApplicationContext.Current.Services.MediaService;
    
                    var media = ms.GetById(Guid.Parse(itemId.Id));
                    //the umbracofile property contains the src for the image, which at the beginning is the location of the file on the filesystem.
                    var umbracoFileJson =
                        JsonConvert.DeserializeObject<ImageCropDataSet>(
                            media.GetValue<string>("umbracoFile")); //ImageCropDataSet used to deserialize the json.
                    var oldPath = $"~{umbracoFileJson.Src}";
    
                    var folder = itemId.Id;
                    var filename = Path.GetFileName(oldPath);
                    var path = $"/media/{folder}/{filename}"; //what will be the new path
                    var absolutePath = HttpContext.Current.Server.MapPath(oldPath); //path to file on filesystem
                    using (var fileStream = System.IO.File.OpenRead(absolutePath))
                    {
                        fs.AddFile(path, fileStream); //saving the file to storage
                    }
    
                    Directory.Delete(Path.GetDirectoryName(absolutePath), true);
    
                    //now to update umbraco database
                    var umbracoFile = media.GetValue("umbracoFile");
                    var newUmbracoFile = ReplaceFolder(media.GetValue<string>("umbracoFile"), folder);
                    media.SetValue("umbracoFile", newUmbracoFile);
                    media.UpdateDate = DateTime.UtcNow;
                    media.CreateDate = DateTime.UtcNow;
                    ms.Save(media);
                }
    
                internal string ReplaceFolder(string umbracoFile, string newFolder)
                {
                    return Regex.Replace(umbracoFile, @"(?<=/media/)\d{4,10}",
                        newFolder); //5 has been changed to 10, to allow up to 10 degits to be selected
                }
            }
        }
    }
    

    and register your event on applicationstarted:

     public class MediaEventTrigger : ApplicationEventHandler {
        protected override void ApplicationStarted(UmbracoApplicationBase umbracoApplication, ApplicationContext applicationContext)
        {
            ExtractionManagerEvents.ExtractedItemResources += ExtractedItemResources;
        }
    
        private void ExtractedItemResources(object sender, ItemEventArgs e)
        {
            var provider = sender as RevisionExtraction;
            //make sure the media event only fires for media items, not other resources
            if (e.Item.ItemId.ProviderId == ItemProviderIds.mediapropertyDataItemProviderGuid)
            {
                var media = e.Item.Dependencies.Single(i => i.ItemId.ProviderId == ItemProviderIds.mediaItemProviderGuid);
                provider.Context.ExecuteEvent("QueueCourierMediaEvent", e.ItemId, new SerializableDictionary<string, string>());
            }
        } }
    

    you should also have this package installed via nuget: https://github.com/ElijahGlover/Umbraco-S3-Provider

    Umbraco.Storage.S3

  • Jamie Attwood 201 posts 493 karma points c-trib
    Dec 04, 2019 @ 18:33
    Jamie Attwood
    0

    Awesome. Thanks very much, I will give this a whirl!

    Jamie

  • Ray 13 posts 103 karma points notactivated
    Jul 25, 2022 @ 00:46
    Ray
    0

    Hello Hugo,

    Thank you very much. When I use the code in the project, the following part of the codeļ¼š

       //now to update umbraco database
        var umbracoFile = media.GetValue("umbracoFile");
        var newUmbracoFile = ReplaceFolder(media.GetValue<string>("umbracoFile"), folder);
        media.SetValue("umbracoFile", newUmbracoFile);
        media.UpdateDate = DateTime.UtcNow;
        media.CreateDate = DateTime.UtcNow;
        ms.Save(media);
    

    After Save method, the URL of media in CMS did not take effect. Do you know the possible reason?

    Regards,

    Ray

  • YESU RAJA CHINTA 22 posts 89 karma points
    Oct 19, 2022 @ 12:59
    YESU RAJA CHINTA
    0

    Hello Every one,

    I am making the same effort. We have media files stored in the Amazon Web Services S3 (MediaEventQueueListener: ItemEventProvider), Someone kindly assist me since this strategy is not triggering for me. Images are still not moving in the desired direction. I adhered to the instructions supplied by Matt Barlow | jacker.io.

    Can someone help here. Thanks.

Please Sign in or register to post replies

Write your reply to:

Draft