Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Hugo 12 posts 142 karma points
    Apr 11, 2017 @ 10:04
    Hugo
    0

    Deploying media with Courier and UmbracoFileSystemProviders.Azure

    We use Courier to deploy content changes from our staging environment to our production environment. Ideally we wouldn't have to do any content changes in production directly.

    We also use a cdn to host our images. We've installed the UmbracoFileSystemProviders.Azure package, which hooks up the media folder to blob storage, which serves as the source for the cdn.

    Now this is where it gets tricky: Courier will detect any dependant media fine, and deploy it to production. As expected it gets the content from the storage account, and sends it over. But the media is not picked up on the receiving end, courier puts it in the media folder, and I can see the files there, but they're not being picked up by the filesystemprovider.

    When I manually upload media to the production website, it works as expected. It appears Courier is bypassing some code by putting the images directly on the filesystem, instead of going through the channels you would when manually uploading.

    It appears I'll have to implement some custom code in between. Also because I'll have to rename the folders that contain the media, because of different id's (I'm pretty sure the folder name is not the same as media id, just an increment, but they don't line up exactly) but I can probably work around that issue.

    I've looked through some documentation and blog posts, but I'm not sure where to begin. There's a lot of confusing lingo and not a lot of documentation, so I'm getting a bit stuck.

    Anybody here who has some ideas or can point me in the right direction? Would be much appreciated.

    Hugo

  • Hugo 12 posts 142 karma points
    Apr 25, 2017 @ 09:09
    Hugo
    105

    Well, I solved it. It was an extremely tiresome process of trial and error trying to figure out how it all works without any proper documentation.

    In short: I hook up to the ExtractedItemResources event, which fires whenever courier finds some media it needs to send (among other things). This happens on the sending side. In this event I create a custom event, a couriercontext event, which will be received by code in the receiving courier site.

    The receiving website listens to my custom event and executes some code to upload the media that courier dropped in the media folder to the azure storage account.

    That's the gist of it, now some code that shows my implementation of it.

    This is run on the sending side:

     public class MediaEventTrigger : ApplicationEventHandler
    {
        protected override void ApplicationStarted(UmbracoApplicationBase umbracoApplication, ApplicationContext applicationContext)
        {
            ExtractionManagerEvents.ExtractedItemResources += ExtractedItemResources;
        }
    
        private void ExtractedItemResources(object sender, ItemEventArgs e)
        {
            var provider = sender as RevisionExtraction;
            //make sure the media event only fires for media items, not other resources
            if (e.Item.ItemId.ProviderId == ItemProviderIds.mediapropertyDataItemProviderGuid)
            {
                var media = e.Item.Dependencies.Single(i => i.ItemId.ProviderId == ItemProviderIds.mediaItemProviderGuid);
                provider.Context.ExecuteEvent("QueueCourierMediaEvent", e.ItemId, new SerializableDictionary<string, string>());
            }
        }
    }
    

    There are a lot of events in ExtractionManagerEvents that you can use, but I was only able to trigger around 2 or 3 of them. The general postprocessing ones and this ExtractedItemResources one. I read somewhere this is because I'm deploying content directly, instead of using revisions.

    Now you can either queue the courier event for later handling or execute it right away. Because I am updating media items in the database to reflect their new location, I have to wait for the deployment to be done. It will not save otherwise but it will also not throw any exceptions. For queueing you can specify a queue, e.g. "DeploymentComplete" queue, but I noticed this does not work in my setup. The sender seems unable to queue anything for the receiver to handle. Executing directly works fine however, which is what I'm doing in above example. The receiver has to delay the event until the deployment is done. Fortunately the receiver can in fact use the queues as intended, so after receiving the first event I just queue another:

     public class MediaEventListener : ItemEventProvider
    {
        public override string Alias
        {
            get
            {
                return "QueueCourierMediaEvent";
            }
        }
    
        public override void Execute(ItemIdentifier itemId, SerializableDictionary<string, string> parameters)
        {
            ExecutionContext.QueueEvent("UpdateMediaPath", itemId, parameters, Umbraco.Courier.Core.Enums.EventManagerSystemQueues.DeploymentComplete);
        }
    }
    
    public class MediaEventQueueListener : ItemEventProvider
    {
        public override string Alias
        {
            get
            {
                return "UpdateMediaPath";
            }
        }
    
        public override void Execute(ItemIdentifier itemId, SerializableDictionary<string, string> parameters)
        { /*code here */ }
    

    MediaEventQueueListener.Execute method will fire once for every media that was transfered with courier. It has to read the file on the filesystem, save it to azure storage, update the media path in umbraco and delete the file on the filesystem.

    The folder structure of media items is messy. It looks like each media file is a folder of its ID, e.g. "/media/1234/image.jpg", but it's a different id. The image could have an umbraco id of 2352 and be put in a folder called 1824. In my case I think only the first few media items line up and the rest doesn't.

    So to determine what folder you have to put the image you have to read out the current highest folder in your storage account. That's what I did at first. Then I changed my mind and used the item guid as folder name. This helps updating cached content, which is a bit out of scope for this question but is also a huge problem when transferring media with courier.

    Here's the code for my execute function. It's a bit messy because sometimes I serialize the json and other times I just regex my way out of it:

    public override void Execute(ItemIdentifier itemId, SerializableDictionary<string, string> parameters)
        {
    
            var fs = FileSystemProviderManager.Current.GetFileSystemProvider<MediaFileSystem>();
            var ms = ApplicationContext.Current.Services.MediaService;
    
            var media = ms.GetById(Guid.Parse(itemId.Id));
            //the umbracofile property contains the src for the image, which at the beginning is the location of the file on the filesystem.
            var umbracoFileJson = JsonConvert.DeserializeObject<UmbracoFile>(media.GetValue<string>("umbracoFile"));
            var oldPath = $"~{umbracoFileJson.src}";
    
            var folder = itemId.Id;
            var filename = Path.GetFileName(oldPath);
            var path = $"/media/{folder}/{filename}"; //what will be the new path
            var absolutePath = HttpContext.Current.Server.MapPath(oldPath); //path to file on filesystem
            using (var fileStream = System.IO.File.OpenRead(absolutePath))
            {
                fs.AddFile(path, fileStream); //saving the file to storage
            }
            Directory.Delete(Path.GetDirectoryName(absolutePath), true);
    
            //now to update umbraco database
            var umbracoFile = media.GetValue("umbracoFile");
            var newUmbracoFile = ReplaceFolder(media.GetValue<string>("umbracoFile"), folder);
            media.SetValue("umbracoFile", newUmbracoFile);
            media.UpdateDate = DateTime.UtcNow;
            media.CreateDate = DateTime.UtcNow;
            ms.Save(media);
    
        }
    
        private string ReplaceFolder(string umbracoFile, string newFolder)
        {
            return Regex.Replace(umbracoFile, @"(?<=/media/)\d{4,5}", newFolder);
        }
    

    When this is done the media is transferred properly to the other side, but as I mentioned before the content is not always updated. I use a lot of grid layouts in my website which need to be updated with the new path, because that doesn't happen automatically.

    I inherit PropertyDataResolverProvider to create a class that can execute custom code when packaging editors with the "Umbraco.Grid" alias. Then for each media item I find I replace its folder with its guid. In case of RTE's with media you also have to add the media item as a dependency to the contentPropertyData, otherwise it will not get shipped.

  • Barry Fogarty 486 posts 1109 karma points
    May 04, 2017 @ 16:12
    Barry Fogarty
    0

    Thank you for your detailed account for this problem Hugo. I had this exact problem and your guide helped enormously. I appreciate your efforts! #H5YR

  • JLon 289 posts 399 karma points
    May 22, 2017 @ 20:40
    JLon
    0

    Hey Hugo,

    I'm trying to do something similar. But I need to trigger reindexing on the receiving end.

    When you are calling a queue on your sending end with

    provider.Context.ExecuteEvent("QueueCourierMediaEvent", e.ItemId, new SerializableDictionary<string, string>());
    

    Do they hit the receiving end's MediaEventListener class of below right away?

    public class MediaEventListener : ItemEventProvider
    {
        public override string Alias
        {
            get
            {
                return "QueueCourierMediaEvent";
            }
        }
    
        public override void Execute(ItemIdentifier itemId, SerializableDictionary<string, string> parameters)
        {
            ExecutionContext.QueueEvent("UpdateMediaPath", itemId, parameters, Umbraco.Courier.Core.Enums.EventManagerSystemQueues.DeploymentComplete);
        }
    }
    
  • Hugo 12 posts 142 karma points
    May 23, 2017 @ 05:51
    Hugo
    1

    Hi JLon,

    That is correct. When the sending end executes that event, the receiving end code fires right away.

    Because my goal was to execute code after the deployment, and queueing events didn't seem possible from the sending end, I had to loop it through the receiver like that.

    I tested it by running my website locally, then starting remote debugging on the remote machine which ran the same dll's. Locally it would not hit a breakpoint in the MediaEventListener class, but remotely it would.

  • JLon 289 posts 399 karma points
    May 23, 2017 @ 10:02
    JLon
    0

    Thanks Hugo, i'll try it out.

    It's so difficult to find out about things especially with Courier where there's not much proper documentation about it :(

    Spent so much time digging around here and there

  • JLon 289 posts 399 karma points
    May 23, 2017 @ 12:31
    JLon
    0

    oh Hugo, just wanted to say you saved my day :D

    It works like a charm. I've wasted like 2 days to try to figure out how to trigger / detect the events from the receiving end. Can't find anything relevant at all

  • Hugo 12 posts 142 karma points
    May 23, 2017 @ 14:20
    Hugo
    0

    That's great JLon! I know your struggle, I had spend a week testing, figuring it all out, and trying to make it work.

    Actually I recognize your username now, it was on a bunch of related threads I came across trying to find a solution. There were others who popped up again and again on similar threads on here and github, hunting for fixes and documentation. It was the reason I made this post.

    Glad I could be of help :)

  • JLon 289 posts 399 karma points
    May 23, 2017 @ 16:46
    JLon
    0

    Oh yeah I've been posting it everywhere. i've even raised on the Courier Issues site too but got nothing back :(

    I've even tried to upgrade courier to the latest version but the issue is still there.

  • Jordan 24 posts 176 karma points
    Feb 07, 2018 @ 10:19
    Jordan
    0

    Thanks for this awnser! this really helped me setting something up a solution in our project. From what I can tell this problem still exists in the latest version of Courier (3.1.6), Umbraco(7.7.9) and UmbracoFileSystemProviders.Azure(1.0.2).

    Below I made some tweaks to folder naming (issues on large sites with lots of media) Also tweaked the Model used for deserialzing the json (needed for when image cropper is in use)

        public override void Execute(ItemIdentifier itemId, SerializableDictionary<string, string> parameters)
        {
            var fs = FileSystemProviderManager.Current.GetFileSystemProvider<MediaFileSystem>();
            var ms = ApplicationContext.Current.Services.MediaService;
    
            var media = ms.GetById(Guid.Parse(itemId.Id));
            //the umbracofile property contains the src for the image, which at the beginning is the location of the file on the filesystem.
            var umbracoFileJson = JsonConvert.DeserializeObject<ImageCropDataSet>(media.GetValue<string>("umbracoFile")); //ImageCropDataSet used to deserialize the json.
            var oldPath = $"~{umbracoFileJson.Src}";
    
            var folder = itemId.Id;
            var filename = Path.GetFileName(oldPath);
            var path = $"/media/{folder}/{filename}"; //what will be the new path
            var absolutePath = HttpContext.Current.Server.MapPath(oldPath); //path to file on filesystem
            using (var fileStream = System.IO.File.OpenRead(absolutePath))
            {
                fs.AddFile(path, fileStream); //saving the file to storage
            }
            Directory.Delete(Path.GetDirectoryName(absolutePath), true);
    
            //now to update umbraco database
            var umbracoFile = media.GetValue("umbracoFile");
            var newUmbracoFile = ReplaceFolder(media.GetValue<string>("umbracoFile"), folder);
            media.SetValue("umbracoFile", newUmbracoFile);
            media.UpdateDate = DateTime.UtcNow;
            media.CreateDate = DateTime.UtcNow;
            ms.Save(media);
        }
    
        internal string ReplaceFolder(string umbracoFile, string newFolder)
        {
            return Regex.Replace(umbracoFile, @"(?<=/media/)\d{4,10}", newFolder); //5 has been changed to 10, to allow up to 10 degits to be selected
        }
    

    ImageCropDataSet is located in the Umbraco.Web.Models namespace.

    Hope this helps someone else that hits this problem.

Please Sign in or register to post replies

Write your reply to:

Draft