Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Damian Maclennan 5 posts 35 karma points
    Dec 23, 2013 @ 23:57
    Damian Maclennan
    0

    Distributed publishing, Examine and Media

    Hi all

    I have a site running in a load balancer with distributed publishing.

    I use a lot of Examine queries due to the rather large number of documents and media in the site. I've found that once in the distributed mode, the Examine indexes for media aren't populating on the other servers. In having a bit of a dig through the Umbraco source it seems the distributed publishing happens when you publish a piece of content, but not when media is saved.

    I've found the question asked here a few times but never really answered, and again in this tweet convo.

    So, is there a "known" good way to get examine to populate media indexes in a distributed environment ? I'm guessing that there's not a built in way to make it happen, I can write some of my own hooks and webservices if people can point me in the right direction.

     

    Thanks!

  • Shannon Deminick 1526 posts 5272 karma points MVP 2x
    Dec 24, 2013 @ 01:29
    Shannon Deminick
    100

    We need to implement this in the core obviously :) This wasn't actually possible until 6.1.1 since there was no events for anything other than content that would execute on each server during a distributed cache call. I've created a task here: http://issues.umbraco.org/issue/U4-3937

    In the meantime, you can make this work on your own if required. There's a new (currently un-documented) event:

    CacheRefresherBase<TInstanceType>.CacheUpdated
    

    For media you'd use:

    CacheRefresherBase<MediaCacheRefresher>.CacheUpdated
    

    The media cache refresher sends either a single id or a json packet to each server to notify them how to update cache (depending on what media is updating), when this occurs this event will be raised on each server that receives the packet. Currently the serialization of this package is internal but you can copy the logic. You'll see it in the class:

    Umbraco.Web.Cache.MediaCacheRefresher
    

    You'd then have to add an ApplicationEventHandler to your codebase to listen to this event and deal with it accordingly to ensure that the local Examine index is updated. The best place to look for examples on how to do this is to look at the class:

    Umbraco.Web.Search.ExamineEvents
    

    Hope that helps!

  • Damian Maclennan 5 posts 35 karma points
    Dec 24, 2013 @ 03:48
    Damian Maclennan
    0

    This helps a lot, thanks Shannon!

    So if I read it right, the event is already getting distributed across the servers, I just need to catch it and trigger the examine update ?

    I did see what looked like the media ids firing when I grabbed a Wireshark dump between the servers, so that would make sense.

     

  • Shannon Deminick 1526 posts 5272 karma points MVP 2x
    Dec 24, 2013 @ 03:49
    Shannon Deminick
    0

    yup so long as you are using a version later than 6.1.1

  • Damian Maclennan 5 posts 35 karma points
    Dec 24, 2013 @ 03:56
    Damian Maclennan
    0

    I'm on 6.1.5 so that should work nicely!

     

    Thanks for your help, will let you know how I go.

  • Mads Krohn 211 posts 504 karma points c-trib
    Dec 28, 2013 @ 23:01
    Mads Krohn
    0

    I'm a bit late to the party, but don't hesitate to let us know if you need any more help on this.

  • Damian Maclennan 5 posts 35 karma points
    Dec 29, 2013 @ 10:41
    Damian Maclennan
    0

    Thanks Mads, yes, Shannon's pointers did the trick.

     

  • Mads Krohn 211 posts 504 karma points c-trib
    Dec 30, 2013 @ 09:36
    Mads Krohn
    0

    That's awesome! :)
    Just remember to mark as answer then :)

    Cheers

  • Shannon Deminick 1526 posts 5272 karma points MVP 2x
    Jan 07, 2014 @ 01:03
    Shannon Deminick
    0

    Yup, would be good to have this marked as 'answered'

  • Randy McCluer 59 posts 87 karma points
    Jan 07, 2014 @ 17:21
    Randy McCluer
    0

    Damian, running into the exact same issue. Any chance you can post the code you ended up with? Thanks!

  • Randy McCluer 59 posts 87 karma points
    Jan 09, 2014 @ 18:49
    Randy McCluer
    0

    For anyone else who needs it this is what I came up with. Seems to be working.

     void DMagEventHandler_CacheUpdated(MediaCacheRefresher sender, CacheRefresherEventArgs e)
        {
            var o = DeserializeFromJsonPayload((string)e.MessageObject);
            if (o.Any())
            {
                o.ForEach(a => IndexMedia(ApplicationContext.Current.Services.MediaService.GetById(a.Id)));
            }
        }
        private static JsonPayload[] DeserializeFromJsonPayload(string json)
        {
            var serializer = new JavaScriptSerializer();
            var jsonObject = serializer.Deserialize<JsonPayload[]>(json);
            return jsonObject;
        }
        private class JsonPayload
        {
            public Guid UniqueId { get; set; }
            public int Id { get; set; }
        }
        private static void IndexMedia(IMedia sender)
        {
            ExamineManager.Instance.ReIndexNode(
                sender.ToXml(), "media",
                ExamineManager.Instance.IndexProviderCollection.OfType<BaseUmbracoIndexer>().Where(x => x.EnableDefaultEventHandler));
        }
    
  • Damian Maclennan 5 posts 35 karma points
    Jan 10, 2014 @ 01:31
    Damian Maclennan
    0

    Sorry guys, was having a break from internets!

    Reply marked, glad you got it working Randy. 

  • Elias 25 posts 108 karma points
    Jan 29, 2014 @ 21:36
    Elias
    0

    Hi, i can't implement a class from CacheRefresherBase to hook ; OnCacheUpdated because i get this error:

    ERROR Umbraco.Core.Sync.DefaultServerMessenger - [Thread 102] Error refreshing a node in the distributed list System.Web.Services.Protocols.SoapException: System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.InvalidOperationException: The cache refresher: b29286dd-2d40-4ddb -b325-681226589fec is not of type Umbraco.Core.Cache.IJsonCacheRefresher at umbraco.presentation.webservices.CacheRefresher.RefreshByJson(Guid uniqueIdentifier, String jsonPayload, String Login, String Password) --- End of inner exception stack trace --- at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult) at Umbraco.Core.Sync.DefaultServerMessenger.PerformDistributedCall(IEnumerable`1 servers, ICacheRefresher refresher, MessageType dispatchType, IEnumerable`1 ids, Type idArrayType, String jsonPayload)

    Instead inherited from MediaCacheRefresher and hooked on Refresh(string jsonPayload) i.e:

    public class DistributedIndexUpdate : MediaCacheRefresher
    {   public override void Refresh(string jsonPayload)
        {
            try
            {
                var medias = DeserializeFromJsonPayload(jsonPayload);
                if (medias.Any())
                {
                    medias.ForEach(delegate(JsonPayload media)
                    {
                        IndexMedia(ApplicationContext.Current.Services.MediaService.GetById(media.Id));
                    });
                }
            }
            catch (Exception ex)
            {
                Log.InfoFormat("....");
            }
            base.Refresh(jsonPayload);
        }
        //the rest of the code...
    }
    

    would work on the same way ? Thanks, Elias

  • Shannon Deminick 1526 posts 5272 karma points MVP 2x
    Mar 06, 2014 @ 09:02
    Shannon Deminick
    1

    You'll be happy to know this should all be resolved with 7.1/6.2 see: http://issues.umbraco.org/issue/U4-3937

Please Sign in or register to post replies

Write your reply to:

Draft