At the moment the site I am working on is very inefficient with its data.
Every time a user requests a page, the site gets data from Umbraco, and then heavily processes it within controllers, before sending it to the user as part of the view.
A really simple example of this is if a custom "create date" is null within the Umbraco document type, when it gets to the controller, it will replace that null date with the parent's document date instead.
I've also got much more expensive calculations, like running through all the child document types to check certain variables, and then work out how many fit a certain criteria.
The key issue is, I'm doing the same calculations every time a user requests the page.
Almost all of my logic only need to be done once until something is changed in the backend, or the date changes.
It's really important I keep the original data inputted by a human, and remains unchanged - otherwise I'd just have it change the original fields.
So now I want to move my logic to happen when a document type is submitted or the date changes (which I think I know how to do), but my question is, what is the best way to store the new processed data after the logic is done in a controller?
I think custom tables might be a really good solution, and after a bit of reading, I think I should be able to store all my processed data there without too much trouble.
Quick question though, is accessing a custom table any slower than how Umbraco normally handles it?
Also, that's a good suggestion Søren - I hadn't thought about using examine for counting children. I also didn't realise you could store specific data in the cache (never had to interact with it directly).
My main concern with the cache is how it is stored - my site currently uses a lot of memory on its server - does the cache store it in memory, or is it written into SQL/some other storage?
For the custom table you'd still have a controller getting the data from the custom table, and this could have caching on it as well (but it's probably overkill).
I've actually successfully been able to write all my data to custom sql tables (still working on the reading side though)!
I was a SQL noob before doing this, and it was a steep learning curve, so I just wanted to leave some notes here for anyone trying this in the future, because the documentation on doing this in v8 is sparse to none.
https://our.umbraco.com/documentation/extending/database/ this tutorial is about migrations, which seems to be the recommended way to create databases. They only trigger once. If you want to trigger them again, like you need to when testing your code, you need to go into you SQL database, and delete your migration from the umbracoKeyValue table, so it runs again on startup (optionally delete your created tables too).
The first tutorial helps with creating a table, but doesn't show how you populate them! For more info on this, I would heavily recommend NPoco's wiki: https://github.com/schotime/NPoco/wiki
Not all C# variables translate well to SQL. For example, SQL's DateTime objects can't go below ~1753, but C#'s DateTime's MinValue is 0001. I had to get creative and save my C# DateTime's Tick's as longs instead to get around this.
C# Objects do not translate at all! In order to save an object, you need to create a whole separate table for them, with an ID that would relate them back to their parent object so you can find them all later.
Array's also can't be saved to SQL - I used JSON serialization to squeeze my things in to a single string field, but considering string fields have a max size of 4000 char's, even this wont work in all situations
This wont ably to everyone, but I was initially hooking into Umbraco's publishing event to save data to the database, but I couldn't create my IPublishedContent as expected, because it hadn't created the cache yet (since it hadn't finished publishing it). It turns out there are Publishing, Published, Saving and Saved events - using the wrong one means you can't use IPublishedContent and instead can only use IContent instead.
Best way to save processed Umbraco data?
At the moment the site I am working on is very inefficient with its data.
Every time a user requests a page, the site gets data from Umbraco, and then heavily processes it within controllers, before sending it to the user as part of the view.
A really simple example of this is if a custom "create date" is null within the Umbraco document type, when it gets to the controller, it will replace that null date with the parent's document date instead.
I've also got much more expensive calculations, like running through all the child document types to check certain variables, and then work out how many fit a certain criteria.
The key issue is, I'm doing the same calculations every time a user requests the page.
Almost all of my logic only need to be done once until something is changed in the backend, or the date changes.
It's really important I keep the original data inputted by a human, and remains unchanged - otherwise I'd just have it change the original fields.
So now I want to move my logic to happen when a document type is submitted or the date changes (which I think I know how to do), but my question is, what is the best way to store the new processed data after the logic is done in a controller?
Any thoughts would be appreciated!
How about just storing the data in a custom data table?
https://our.umbraco.com/documentation/extending/database/
Hi,
If you need to do some counting on children / descendants, you may be able to get better performance from examine.
Other than that, I would suggest just caching the data, and then invalidating the cache when publishing.
Read more of what Umbraco can help with in terms of caching https://our.umbraco.com/Documentation/Reference/Cache/
HTH :)
Thanks for the answers!
I think custom tables might be a really good solution, and after a bit of reading, I think I should be able to store all my processed data there without too much trouble.
Quick question though, is accessing a custom table any slower than how Umbraco normally handles it?
Also, that's a good suggestion Søren - I hadn't thought about using examine for counting children. I also didn't realise you could store specific data in the cache (never had to interact with it directly).
My main concern with the cache is how it is stored - my site currently uses a lot of memory on its server - does the cache store it in memory, or is it written into SQL/some other storage?
For the custom table you'd still have a controller getting the data from the custom table, and this could have caching on it as well (but it's probably overkill).
Here's the v7 docs around caching:
https://our.umbraco.com/Documentation/Reference/Cache/index-v7
Paul Seal has a good article about donut caching that you might be interested in:
https://codeshare.co.uk/blog/how-to-use-donut-caching-in-umbraco-and-mvc/
Thanks a tonne - Donut caching looks like a really good solution for the things I don't put in a custom table.
Really appreciate the help guys!
I've actually successfully been able to write all my data to custom sql tables (still working on the reading side though)!
I was a SQL noob before doing this, and it was a steep learning curve, so I just wanted to leave some notes here for anyone trying this in the future, because the documentation on doing this in v8 is sparse to none.
https://our.umbraco.com/documentation/extending/database/ this tutorial is about migrations, which seems to be the recommended way to create databases. They only trigger once. If you want to trigger them again, like you need to when testing your code, you need to go into you SQL database, and delete your migration from the umbracoKeyValue table, so it runs again on startup (optionally delete your created tables too).
The first tutorial helps with creating a table, but doesn't show how you populate them! For more info on this, I would heavily recommend NPoco's wiki: https://github.com/schotime/NPoco/wiki
Not all C# variables translate well to SQL. For example, SQL's DateTime objects can't go below ~1753, but C#'s DateTime's MinValue is 0001. I had to get creative and save my C# DateTime's Tick's as longs instead to get around this.
C# Objects do not translate at all! In order to save an object, you need to create a whole separate table for them, with an ID that would relate them back to their parent object so you can find them all later.
Array's also can't be saved to SQL - I used JSON serialization to squeeze my things in to a single string field, but considering string fields have a max size of 4000 char's, even this wont work in all situations
This wont ably to everyone, but I was initially hooking into Umbraco's publishing event to save data to the database, but I couldn't create my IPublishedContent as expected, because it hadn't created the cache yet (since it hadn't finished publishing it). It turns out there are Publishing, Published, Saving and Saved events - using the wrong one means you can't use IPublishedContent and instead can only use IContent instead.
I'll update this if I can think of any more.
is working on a reply...