I would rather save and not publish, then "save an publish" the parent node and include the children.

Kind regards Damiaan

Hi,

I can't answer your disable cache rebuild question (not programmatically but I'm sure you're aware it can be disabled in the config) - but have you tried only publishing / updating nodes that actually need updating? Are all 15k actually changing on each import? Publishing data is expensive in Umbraco - you're also going to be creating yourself a massive version history which is going to grow every time the job runs (hint see ContentServices.DeleteVersions but this is likely to be expensive too!).

If only a few nodes have changed then getting them from the cache first and checking the values before doing the expensive ContentService get, update and SaveAndPublish would massively help.

The only other thing I could think of doing was to try only saving and then publishing the parent (including all children). In my quick test this took just as long - test code below if you're interested.

One thing I did notice was disabling events seemed to have a marginal speed increase.

var watch = System.Diagnostics.Stopwatch.StartNew();
var cs = ApplicationContext.Current.Services.ContentService;

var productsNode = cs.GetById(1116);

foreach(var curNode in cs.GetChildren(1116))
{
    curNode.SetValue("pageTitle", DateTime.Now.ToShortTimeString());
    cs.SaveAndPublishWithStatus(curNode, raiseEvents: true);
}

watch.Stop();
var elapsedMs = watch.ElapsedMilliseconds;

Versus publish with children...

@{

    var watch = System.Diagnostics.Stopwatch.StartNew();
    var cs = ApplicationContext.Current.Services.ContentService;

    var productsNode = cs.GetById(1116);

    foreach (var curNode in cs.GetChildren(1116))
    {
        curNode.SetValue("pageTitle", DateTime.Now.ToShortTimeString());
        cs.Save(curNode);
    }

    // do one big save and publish of the parent including children at the end
    cs.PublishWithChildrenWithStatus(productsNode);

    watch.Stop();
    var elapsedMs = watch.ElapsedMilliseconds;

}

Anyone else know of a better way of doing this?

Steve

You can just read it as IPublishedContent, like you would in any front-end query (assuming it's published, of course). So, for instance, if all your products were under a page called Products you could go to that page and get all the descendants/ children - https://our.umbraco.org/documentation/Reference/Querying/IPublishedContent/Collections

Hi,

The raise events flag was included in my code but I think this is a red herring for you as it won't fix your problem. Instead focus on what Dan's saying.

As I hinted at and Dan's done a much better job of explaining you need to get each node via the IPublishedContent cache (e.g. Umbraco.TypedContent(1234) - check the contents haven't changed and only use the ContentService if the content node needs updating. The slightly confusing thing is you'll be checking the data via IPublishingContent and then having to "get" the content node again (probably by ID) via the ContentService service. This means a bit more code which will "feel" slower but it will execute in seconds as the database is only hit for nodes that require updating rather than 15k - and of course you won't have the expensive cache updates for every node.

Hope I've explained myself better!

Steve

Thanks to all answers, this is the solution that I implemented:

1) Implement a checksum mechanism using MD5 hashing, or check each property value against changes. If there are only a few property values, checksum generation might not be necessary. In my case, there are five fields but two of them are used for primary keys, so I only need to compare three fields, which is not a big deal.

2) Traverse nodes on the cache, not via ContentService, i.e. use UmbracoHelper.TypedContent(id) instead of ContentService.GetChildren(id).

(I don't use "frontend queries", I prefer to do data related tasks on controller side. Reason is that I am used to MVC architecture. If my approach is wrong in this case, please warn me.)

3) Only change the node if the checksum of the node and the version in the external data do not match. (or if the property values and new data fields do not match)

4) Get and save only the changed nodes via ContentService. Don't publish.

5) Delete all nodes that are not existent in the new external dataset.

6) Create nodes for the data that are existent in the external dataset but not existent in current database. Save these nodes, do not publish.

7) If there was at least one changed/added node, publish parent with children.

This ensures that:

• XML cache will only be reconstructed after all process is finished.
• Database will only hit for deletion, creation and updates and only when it is necessary.

The only drawback I see in my tests is that "publish with children" takes a lot of time. Sometimes I get a "task cancelled" error (that I can only observe when I check the logs). But that might be related with my environment, too.

I am currently testing this approach with the >15k dataset. I will update this post when I have my results.

Thanks to everyone who contributed.

Hi,

That sounds right to me - I would try with just a simple publish as the cache only dirties what it needs to I believe and you might find it's more stable rather than the publish all children method.

Nice to hear you've hit on a solution!

Kind regards

Steve

is working on a reply...