Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Ben Jermy 9 posts 78 karma points
    Feb 14, 2024 @ 11:23
    Ben Jermy
    0

    Are you able to send custom data from database to a different environment?

    Hi,

    I'm trying to send custom imported data from my database to another environment. I've seen you can hook into the uSyncExportCompletedNotification but how would you send the data to then grab it from this notification?

    Is there a trigger event i can hook into to then push the data over?

    Any help would be appreciated!

    Thanks

  • Kevin Jump 2310 posts 14694 karma points MVP 7x c-trib
    Feb 15, 2024 @ 14:01
    Kevin Jump
    1

    Hi Ben,

    the best way to get data from one server to another is to have a SyncHandler, and Serializer that take the data from the database and put it onto disk.

    This is how we do it for Umbraco.Forms and Umbraco.Commerce.

    uSync.Forms - code https://github.com/KevinJump/uSync.Forms/tree/v13/main/uSync.Forms

    there is a bit in here, but if you look for example at the prevalue handler and prevalue serializer, you should get an idea as to what is happening.

    (you don't have to have events to handle saves, delete etc, but they are needed if you want to keep things in sync on disk).

    Once you have this you will have added your data to uSync , (it can even appear in the normal uSync dashboard).


    and then uSync.Complete can use that to send the data between server.

    uSync.Complete uses a thing called an Item Handler to work out what needs to be transferred.

    https://github.com/KevinJump/uSync.Forms/blob/v13/main/uSync.Forms/Sync/FormSyncManager.cs

    the handle means if someone right clicks on the tree, uSync will know how to then send those items across with other things.


    If you want your data to go across between servers when it has been included / referenced on a content page, you can add a 'value mapper` that tells uSync there is a dependency for that value.

    So the example forms one works, when someone picks a form on a content page the value mapper is taked with working out what form actually needs to be sent over with the content page for the form to then work on the other side.

    https://github.com/KevinJump/uSync.Forms/blob/v13/main/uSync.Forms/Mappers/FormPickerValueMapper.cs


    Its not 'super' simple, but once you have the first two things you can usually transfere things between sites reliably and consistantly.

  • Ben Jermy 9 posts 78 karma points
    30 days ago
    Ben Jermy
    0

    Hi Kevin,

    I've ran into some issues with my custom data serializer and handler, The export/import works completely fine.

    The issue here is when specifying my handlers entity type as "Document" when right-clicking a node and pushing it to another server it comes back with an error reading "Nothing to export". This is because for some reason when a push is initialized the "FindItem(Guid key)" function in my custom serializer is ran, I assume this is due to my entity type being "Document" but then how would you hook into usync to then detect what node is being pushed and tell the handler to use the correct serializer. For Example, On push of a "Brand" node, serialize and deserialize the brand data etc.

    I've tried using a "SyncValueMapperBase" but on a push, the code locally doesn't hit any break points which i assume usync is not picking it up entirely, unless i'm wrong here.

    Any help on this would be much appreciated.

    Thanks

  • Kevin Jump 2310 posts 14694 karma points MVP 7x c-trib
    30 days ago
    Kevin Jump
    0

    Hi,

    Are your custom entities properties on a document type or things in their own right?

    internally Umbraco content nodes are of entity type 'Document' so unless you are replacing the core way of syncing documents you want to use a different entity type for your items?

    if your data (or a link to the data) is stored in document properties, then you need a value mapper (for example like forms has - https://github.com/KevinJump/uSync.Forms/blob/v13/main/uSync.Forms/Mappers/FormPickerValueMapper.cs.

    This will tell uSync that a certain property has a dependency on your item(s) - so the export etc will then fire for them.


    If your data is attached to documents but not properties, then does it behave more like Culture and Hostnames. (which has its own tables but keys of the document). or languages.

    for them we have a dependency checker for IContent that tells uSync that these other items are dependencies on

    e.g. this is the Language Checker - this finds all the languages are calls them as a dependency on content, so they get synced (for certain setups)

    internal class ContentLanguageChecker : ISyncDependencyChecker<IContent>
    {
        private readonly ILocalizationService _localizationService;
    
        public ContentLanguageChecker(ILocalizationService localizationService)
        {
            _localizationService = localizationService;
        }
    
        public UmbracoObjectTypes ObjectType => UmbracoObjectTypes.Language;
    
        public IEnumerable<uSyncDependency> GetDependencies(IContent item, DependencyFlags flags)
        {
            if (!flags.Contains(DependencyFlags.IncludeDependencies)) yield break;
    
            foreach (var culture in item.AvailableCultures)
            {
                var language = _localizationService.GetLanguageByIsoCode(culture);
    
                if (language == null) continue;
    
                yield return new uSyncDependency
                {
                    Name = $"Language {language.CultureName}",
                    Order = DependencyOrders.Languages,
                    Udi = language.GetUdi(),
                    Flags = DependencyFlags.None,
                    Level = 0
                };
            }
        }
    }
    

    if your data is in its own custom section/tree, then you will need the item manager as this is what controls the push/pull buttons on the dialogs. but the entity type(s) can't be document as that is already taken by Umbraco.

    Again forms does this, (so when you right click on a form, you can push pull it) with this code. https://github.com/KevinJump/uSync.Forms/blob/v13/main/uSync.Forms/Sync/FormSyncManager.cs

  • Ben Jermy 9 posts 78 karma points
    29 days ago
    Ben Jermy
    0

    Hi,

    Thanks for your help on this its helped alot. I've managed to get to the point where on brand nodes the brand data exports and sends over. But the bit i'm struggling on is if i am to change anything in the exported brand data xml then do a push the response comes back as "No changes".

    For some reason the brand data doesn't detect changes when pushing to another environment.

    I've break pointed the functions that should run on a push and all seem to be hitting fine. The changes are just not detecting.

    I've looked in the logs on the target environment and the brand gets serialized correctly but doesn't deserialize (As i am assuming it doesn't detect any changes so doesn't need too.)

    Maybe i'm missing a class here?

    Cheers

  • Ben Jermy 9 posts 78 karma points
    28 days ago
    Ben Jermy
    0

    Hi Kevin,

    I've noticed that if a piece of content is changed on the node, the brand data will then deserialize.

    I'm assuming its because i added the dependency for IContent (To send my brand data) it will only change the brand data when a change in icontent is made.

    Currently, the custom handler etc works with the nodes i allow it too (Which is what i was trying to achieve) by adding a dependency for IContent and checking alias type as "brand". but i need it to detect changes even if no content has changed. Acting in the same way the contenthandler works but obviously only checking my brand data xml files.

    Is there any way of doing this?

    Thanks

  • Kevin Jump 2310 posts 14694 karma points MVP 7x c-trib
    28 days ago
    Kevin Jump
    0

    Hi Ben,

    I am not sure (so i will double check) but the process should allow for your items to fire changes, even if the content item they are from hasn't.

    • uSync first goes through the things you want to push and gets all the dependencies,
    • then it exports all items based on the list of dependencies.
    • then the exported items copied to the target server
    • then the report on the target server will tell of any changes.

    So if your items are dependencies they will get exported regardless of changes, and they should get copied over and then the report should run.

    when reported your item should return a uSyncAction with a changetype of create to update then the change will be notified (if the xml is different and you have inherited any of the core handler stuff this should happen).

    during the process these files will appear in the temp folder (temp/usync/receive) but they get deleted as part of the report if they are not changes so it can be hard to see them in that folder.

    One way to see this might be to use uSync.Exporter to export the content with the dependencies because this should generate a uSyncPack (zip) file with a usync folder and all of your items in their respective child folder?

  • Ben Jermy 9 posts 78 karma points
    27 days ago
    Ben Jermy
    0

    Hi Kevin,

    Thanks for your help.

    I've tested this via a sync-pack and when importing the said sync pack the message reads "0 changes across 289 items" but then when i go to "Import" that's when i get all my brand data imported.

    So to me the report pack isn't picking up that theirs changes in the XML for my Handler/Serializer.

    Below you can see my Handler code

        [SyncHandler("brandHandler", "Brand Data", "BrandData", uSyncBrandPriorties.Brands, Icon = "icon-library color-light-green", EntityType = uSyncBrand.EntityType)]
    public class BrandHandler : SyncHandlerBase<BrandEntityType, IService>, ISyncHandler
    {
        private readonly BrandFolder? _brandFolder;
        Int32 Iteration = 0;
    
        public BrandHandler(IUmbracoContextAccessor content, ILogger<SyncHandlerBase<BrandEntityType, IService>> logger, IEntityService entityService, AppCaches appCaches, IShortStringHelper shortStringHelper, SyncFileService syncFileService, uSyncEventService mutexService, uSyncConfigService uSyncConfig, ISyncItemFactory syncItemFactory) : base(logger, entityService, appCaches, shortStringHelper, syncFileService, mutexService, uSyncConfig, syncItemFactory)
        {
            content.TryGetUmbracoContext(out var UmbracoContext);
    
    
            var ceiRoot = UmbracoContext?.Content?.GetAtRoot().FirstOrDefault()?.DescendantOfType("customerEffortIndex");
            _brandFolder = (BrandFolder?)ceiRoot?.DescendantOfType("brandFolder");
        }
    
        public override string Group => "Content";
    
        public override IEnumerable<uSyncAction> ExportAll(string folder, HandlerSettings config, SyncUpdateCallback callback)
        {
            var actions = new List<uSyncAction>();
    
            foreach (Brand item in _brandFolder?.Children().OrderBy(x => x.Name))
            {
                BrandEntityType Brand = new BrandEntityType();
                Brand.brand = item;
                callback?.Invoke(GetItemName(Brand), 2, 4);
                actions.AddRange(Export(Brand, folder, config));
            }
    
            return actions;
        }
    
        protected override string GetItemName(BrandEntityType item)
        {
            return item.brand.Name;
        }
    }
    

    Below you can see my serializer code

    [SyncSerializer("fde05407-9833-4713-9c6c-d13bf946707f", "Brand Serializer", "IContent", IsTwoPass = true, Priority = uSyncBrandPriorties.Brands)]
    public class BrandSerializer : SyncSerializerRoot<BrandEntityType>, ISyncSerializer<BrandEntityType>
    {
        private readonly ICacheCeiService _cache;
        private readonly IContentService _content;
        private readonly IUmbracoContextFactory _contextAccessor;
        private readonly ICeiService _ceiService;
        private readonly ILogger<SyncSerializerRoot<BrandEntityType>> _logger;
        private List<CeiModel> _current;
    
        public BrandSerializer(IContentService content, IUmbracoContextFactory context, ICeiService ceiservice, ICacheCeiService cache, IEntityService entityService, ILogger<SyncSerializerRoot<BrandEntityType>> logger) : base(logger)
        {
            _cache = cache;
            _contextAccessor = context;
            _ceiService = ceiservice;
            _logger = logger;
            _content = content;
        }
    
        public override void DeleteItem(BrandEntityType item)
        {
            var brand = _content.GetById(item.Id);
            _content.Delete(brand);
        }
    
        public override BrandEntityType FindItem(int id) {
            using (var cref = _contextAccessor.EnsureUmbracoContext())
            {
                var node = cref.UmbracoContext.Content?.GetById(id);
                if (node != null && node.ContentType.Alias == "brand")
                {
                    BrandEntityType brandentity = new BrandEntityType();
                    brandentity.brand = (Brand)node;
                    return brandentity;
                }
    
                return null;
            }
        }
    
        public override BrandEntityType FindItem(Guid key) 
        {
            using (var cref = _contextAccessor.EnsureUmbracoContext())
            {
                var node = cref.UmbracoContext.Content?.GetById(key);
    
                if (node != null && node.ContentType.Alias == "brand")
                {
                    BrandEntityType brandentity = new BrandEntityType();
                    brandentity.brand = (Brand)node;
                    return brandentity;
                }
    
                return null;
            }
        }
    
        public override BrandEntityType FindItem(string alias) 
        {
            if (String.IsNullOrEmpty(alias))
            {
                return null; 
            }
    
            using (var cref = _contextAccessor.EnsureUmbracoContext())
            {
    
                BrandFolder? brands = (BrandFolder?)cref.UmbracoContext.Content?.GetAtRoot().FirstOrDefault()?.DescendantOfType("brandFolder");
                Brand? brand        = (Brand?)brands?.Children()?.Where(x => ((Brand)x).BrandDataKey == alias).FirstOrDefault();
                BrandEntityType brandentity = new BrandEntityType();
                brandentity.brand = (Brand)brand;
                return brandentity;
            }
        }
    
    
        public override string ItemAlias(BrandEntityType item)
        {
            if (item == null)
            {
                return string.Empty;
            }
            using (var cref = _contextAccessor.EnsureUmbracoContext())
            {
                return item.brand.BrandDataKey;
            }
        }
    
    
        public override Guid ItemKey(BrandEntityType item)
        {
            return item.brand.Key;
        }
    
    
        public override void SaveItem(BrandEntityType item)
        {
            if (item != null)
            {
                foreach (CeiModel objModel in _current)
                {
                    _ceiService.UsyncSet(objModel, _logger);
                }
            }
        }
    
        protected override SyncAttempt<BrandEntityType> DeserializeCore(XElement node, SyncSerializerOptions options)
        {
            _current = new List<CeiModel>();
            var country  = node.Element("country").ValueOrDefault(string.Empty);
            var sector   = node.Element("sector").ValueOrDefault(string.Empty);
            var brand    = node.Element("name").ValueOrDefault(string.Empty);
            Brand brandNode = (Brand)_contextAccessor.EnsureUmbracoContext().UmbracoContext.Content.GetById(node.GetKey());
    
            BrandEntityType brandEntity = new BrandEntityType();
            brandEntity.brand = brandNode;
    
            for (int index = 0; index <= 3; index++) 
            {
                if (!String.IsNullOrEmpty(node.Element($"Q{index}id")?.Value)) 
                {
                    CeiModel model = new CeiModel(); 
                    model.Id = int.Parse(node.Element($"Q{index}id")?.Value ?? "0");
                    model.Position = int.Parse(node.Element($"Q{index}position")?.Value ?? "0");
                    model.OldPosition = int.Parse(node.Element($"Q{index}oldposition")?.Value ?? "0");
                    model.Country = country;
                    model.Category = sector;
                    model.CustomerServiceCommunicationStyle = Double.Parse(node.Element($"Q{index}communication")?.Value ?? "0");
                    model.DataRange = node.Element($"Q{index}datarange")?.Value ?? "0";
                    model.EffortScore = Double.Parse(node.Element($"Q{index}effortscore")?.Value ?? "0");
                    model.CustomerServiceKnowledgeAndExpertise = Double.Parse(node.Element($"Q{index}knowledge")?.Value ?? "0");
                    model.EaseOfSpeedSolvingTheProblem = Double.Parse(node.Element($"Q{index}speed")?.Value ?? "0");
                    model.UnderstandingAndValuingCustomer = Double.Parse(node.Element($"Q{index}valuing")?.Value ?? "0");
                    model.NPS = Double.Parse(node.Element($"Q{index}nps")?.Value ?? "0");
                    model.Notes = node.Element($"Q{index}notes")?.Value;
                    model.Updated = DateTime.Parse(node.Element($"Q{index}updated")?.Value ?? DateTime.MinValue.ToString(), new CultureInfo("en-GB"));
                    model.Valid = Boolean.Parse(node.Element($"Q{index}valid")?.Value ?? "0");
                    model.Brand = brand;
                    _current.Add(model);
                    _logger.LogInformation($"USYNC: Brand {brand} has been saved to the database");
                }
            }
    
            _logger.LogInformation($"USYNC: Brand {brand} has been deserialized successfully");
            return SyncAttempt<BrandEntityType>.Succeed(brandNode?.Name, brandEntity, ChangeType.Import, Array.Empty<uSyncChange>());
        }
    
        protected override SyncAttempt<XElement> SerializeCore(BrandEntityType item, SyncSerializerOptions options)
        {
            var node = InitializeBaseNode(item, item.brand.BrandDataKey);
            IEnumerable<CeiModel> ceiData = _cache.GetAllBrandData(((Country?)item.brand.Country)?.DataKey ?? string.Empty, ((Sector?)item.brand.Sector)?.DataKey ?? string.Empty, item.brand.DisplayName ?? string.Empty);
            var s = ceiData.OrderByDescending(x => x.Updated).Take(4);
    
            node.Add(new XElement("name", item.brand.DisplayName));
            node.Add(new XElement($"country", ((Country?)item.brand.Country)?.DataKey));
            node.Add(new XElement($"sector", ((Sector?)item.brand.Sector)?.DataKey));
    
            // Here we only need to export 4 dataranges in the CeiModels.
            Int32 index = 0;
            String effortscore = string.Empty;
            foreach (var CeiModel in ceiData)
            {
                node.Add(new XElement($"Q{index}id", CeiModel.Id));
                node.Add(new XElement($"Q{index}position", CeiModel.Position));
                node.Add(new XElement($"Q{index}oldposition", CeiModel.OldPosition));
                node.Add(new XElement($"Q{index}datarange", CeiModel.DataRange));
                node.Add(new XElement($"Q{index}effortscore", CeiModel.EffortScore));
                node.Add(new XElement($"Q{index}knowledge", CeiModel.CustomerServiceKnowledgeAndExpertise));
                node.Add(new XElement($"Q{index}valuing", CeiModel.UnderstandingAndValuingCustomer));
                node.Add(new XElement($"Q{index}speed", CeiModel.EaseOfSpeedSolvingTheProblem));
                node.Add(new XElement($"Q{index}nps", CeiModel.NPS));
                node.Add(new XElement($"Q{index}updated", CeiModel.Updated));
                node.Add(new XElement($"Q{index}valid", CeiModel.Valid));
                node.Add(new XElement($"Q{index}notes", CeiModel.Notes));
                node.Add(new XElement($"Q{index}communication", CeiModel.CustomerServiceCommunicationStyle));
                effortscore = CeiModel.EffortScore.ToString();
                index++;
            }
    
    
            _logger.LogInformation($"USYNC: Brand {item.brand.Name} has been serialized successfully - with effort score {effortscore}");
            return SyncAttempt<XElement>.Succeed(item.brand.Name, node, ChangeType.Export, Array.Empty<uSyncChange>());
        }
    }
    

    And my Dependency code:

    using Umbraco.Cms.Web.Common.PublishedModels;
    using CE.Website.Models;
    using Umbraco.Cms.Core.Web;
    using Umbraco.Cms.Core.Models;
    using uSync.Core.Dependency;
    using Umbraco.Cms.Core.Models.ContentEditing;
    using uSync.Forms;
    using Umbraco.Cms.Core;
    
    namespace uSync.Brands.Dependencies
    {
        public class BrandDependencyChecker : ISyncDependencyChecker<IContent>
        {
            public IUmbracoContextFactory _factory;
            public BrandFolder? _brandFolder;
            public UmbracoObjectTypes ObjectType => UmbracoObjectTypes.Document;
            public BrandDependencyChecker(IUmbracoContextFactory factory)
            {
                _factory = factory;
            }
    
            public IEnumerable<uSyncDependency> GetDependencies(IContent item, DependencyFlags flags)
            {
                if (item == null || item.ContentType.Alias != "brand") 
                { 
                     return Enumerable.Empty<uSyncDependency>();
                }
    
                Brand? brand = (Brand?)_factory.EnsureUmbracoContext().UmbracoContext.Content?.GetById(item.Key);
    
    
                if (brand != null)
                { 
                    List<uSyncDependency> dependencies = new List<uSyncDependency>
                    {
                        new uSyncDependency()
                        {
                            Name = $"Brand {brand?.Name}",
                            Order = 1,
                            Udi = Udi.Create(uSyncBrand.EntityType, item.Key),
                            Flags = DependencyFlags.None,
                            Level = 0,
                        }
                    };
                    return dependencies;
                }
    
                return Enumerable.Empty<uSyncDependency>();
            }
        }
    }
    

    Not sure what i'm doing wrong here but maybe i'm missing something in one of these classes?

    Thanks

  • Kevin Jump 2310 posts 14694 karma points MVP 7x c-trib
    27 days ago
    Kevin Jump
    0

    Hi Ben,

    I can't see anything obvious.

    two things to check.

    1. have you registered your entity type with Umbraco ? you can do this in a composer. e.g this is how we register the extra ones we use for users/access. (the first parameter is a constant, but its a string)

          UdiParser.RegisterUdiType(PeopleEdition.UserEntityType, UdiType.StringUdi);
          UdiParser.RegisterUdiType(PeopleEdition.UserGroupEntityType, UdiType.StringUdi);
          UdiParser.RegisterUdiType(uSyncExpansions.EntityTypes.PublicAccess, UdiType.GuidUdi);
      

      .

    2. on the dependency checker set the flags on a a dependency to the same as the flags passed in,. I don't think its this, but its the only difference i could spot s

  • Ben Jermy 9 posts 78 karma points
    23 days ago
    Ben Jermy
    0

    Hi Kevin,

    I've registered my UDI type already and have tested the report with the same flags that are parsed in. But seem to still not get any report on what changes are made for the brand data.

    Really not sure whats going on tbh,

    Any more help would be appreciated on this!

    Thanks

  • Ben Jermy 9 posts 78 karma points
    15 days ago
    Ben Jermy
    0

    Hi Kevin,

    Doing some decompilation of the code the SaveActions function is called to write the updates to the _actions.config and line 116 gets any record with the same GUID and replaces it at line 119 (file is an extract from SyncPackService.cs from the namespace uSync.Expansions.Core.Services) enter image description here

    I've worked out that because i was using the key of the content node, the reports were not showing. This has been changed and all works as intended.

    Thought i would share this information with you for future reference.

    Thanks

Please Sign in or register to post replies

Write your reply to:

Draft