Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • George Kent-Nye 16 posts 119 karma points
    1 week ago
    George Kent-Nye
    0

    Best way to save processed Umbraco data?

    At the moment the site I am working on is very inefficient with its data.

    Every time a user requests a page, the site gets data from Umbraco, and then heavily processes it within controllers, before sending it to the user as part of the view.

    A really simple example of this is if a custom "create date" is null within the Umbraco document type, when it gets to the controller, it will replace that null date with the parent's document date instead.

    I've also got much more expensive calculations, like running through all the child document types to check certain variables, and then work out how many fit a certain criteria.

    The key issue is, I'm doing the same calculations every time a user requests the page.

    Almost all of my logic only need to be done once until something is changed in the backend, or the date changes.

    It's really important I keep the original data inputted by a human, and remains unchanged - otherwise I'd just have it change the original fields.

    So now I want to move my logic to happen when a document type is submitted or the date changes (which I think I know how to do), but my question is, what is the best way to store the new processed data after the logic is done in a controller?

    Any thoughts would be appreciated!

  • Brendan Rice 473 posts 814 karma points
    1 week ago
    Brendan Rice
    100

    How about just storing the data in a custom data table?

    https://our.umbraco.com/documentation/extending/database/

  • Søren Gregersen 393 posts 1554 karma points MVP 2x c-trib
    1 week ago
    Søren Gregersen
    1

    Hi,

    If you need to do some counting on children / descendants, you may be able to get better performance from examine.

    Other than that, I would suggest just caching the data, and then invalidating the cache when publishing.

    Read more of what Umbraco can help with in terms of caching https://our.umbraco.com/Documentation/Reference/Cache/

    HTH :)

  • George Kent-Nye 16 posts 119 karma points
    1 week ago
    George Kent-Nye
    0

    Thanks for the answers!

    I think custom tables might be a really good solution, and after a bit of reading, I think I should be able to store all my processed data there without too much trouble.

    Quick question though, is accessing a custom table any slower than how Umbraco normally handles it?

    Also, that's a good suggestion Søren - I hadn't thought about using examine for counting children. I also didn't realise you could store specific data in the cache (never had to interact with it directly).

    My main concern with the cache is how it is stored - my site currently uses a lot of memory on its server - does the cache store it in memory, or is it written into SQL/some other storage?

  • Brendan Rice 473 posts 814 karma points
    1 week ago
    Brendan Rice
    0

    For the custom table you'd still have a controller getting the data from the custom table, and this could have caching on it as well (but it's probably overkill).

    Here's the v7 docs around caching:

    https://our.umbraco.com/Documentation/Reference/Cache/index-v7

    Paul Seal has a good article about donut caching that you might be interested in:

    https://codeshare.co.uk/blog/how-to-use-donut-caching-in-umbraco-and-mvc/

  • George Kent-Nye 16 posts 119 karma points
    1 week ago
    George Kent-Nye
    1

    Thanks a tonne - Donut caching looks like a really good solution for the things I don't put in a custom table.

    Really appreciate the help guys!

  • George Kent-Nye 16 posts 119 karma points
    16 hours ago
    George Kent-Nye
    1

    I've actually successfully been able to write all my data to custom sql tables (still working on the reading side though)!

    I was a SQL noob before doing this, and it was a steep learning curve, so I just wanted to leave some notes here for anyone trying this in the future, because the documentation on doing this in v8 is sparse to none.

    1. https://our.umbraco.com/documentation/extending/database/ this tutorial is about migrations, which seems to be the recommended way to create databases. They only trigger once. If you want to trigger them again, like you need to when testing your code, you need to go into you SQL database, and delete your migration from the umbracoKeyValue table, so it runs again on startup (optionally delete your created tables too).

    2. The first tutorial helps with creating a table, but doesn't show how you populate them! For more info on this, I would heavily recommend NPoco's wiki: https://github.com/schotime/NPoco/wiki

    3. Not all C# variables translate well to SQL. For example, SQL's DateTime objects can't go below ~1753, but C#'s DateTime's MinValue is 0001. I had to get creative and save my C# DateTime's Tick's as longs instead to get around this.

    4. C# Objects do not translate at all! In order to save an object, you need to create a whole separate table for them, with an ID that would relate them back to their parent object so you can find them all later.

    5. Array's also can't be saved to SQL - I used JSON serialization to squeeze my things in to a single string field, but considering string fields have a max size of 4000 char's, even this wont work in all situations

    6. This wont ably to everyone, but I was initially hooking into Umbraco's publishing event to save data to the database, but I couldn't create my IPublishedContent as expected, because it hadn't created the cache yet (since it hadn't finished publishing it). It turns out there are Publishing, Published, Saving and Saved events - using the wrong one means you can't use IPublishedContent and instead can only use IContent instead.

    I'll update this if I can think of any more.

Please Sign in or register to post replies

Write your reply to:

Draft