We have a Trello board full of little ideas for "some day maybe" type packages that we know we and others would like to reuse. None of them are really ground breaking but they all do a little something to automate or ease a task and make the general Umbraco experience a little nicer long term.
We've already built quite a few packages over the years and each time we notice the demand on whats required to make a package "good" gets longer and longer. This all sort of gets in the way of building what in essence is a simple feature.
Let me give you an example. From our work we did for the Back Office Speed Hacks talk at Code Garden (it was a good talk btw, well worth a watch!) one of my tips was empty the recycle bin regularly. Well why not have something that will do that for you automatically so you don't have to worry about it? Great! This is how that might look:
We hook into the app start event and do a quick search for all content in the recycle bin sorted by date and then delete the stuff thats X days old.
Easy! Could knock that up in about 30 minutes. But to package it up and make it reusable by others and ourselves we really need to go a bit further. What if I want to change the duration before we delete (the "X days" in the above spec)? Well we could just put that in a config. Ok which one? A custom one just for this? Seems a bit excessive but ok. But now the client says they can't find something they deleted last month, the X is too short can we change it. Sure! but its now a release we have to do and a deploy. Isn't this meant to be a CMS? Shouldn't I be able to change this in the UI? I mean its just a drop down not like I'm changing the connection string or anything!
So now we need a UI to change the X, this is a good thing because if we just stashed this away in a config its got no visibility, you won't see it so mgiht not even know its there. By having something you can find or see in the UI you allow for folks to explore and discover features and use them if needed.
So we need to inject an edit screen for a simple numeric field stored in a separate config. Lets say I'm going to try to inject it into the top of the Recycled area itself (you didn't know you can click on it directly did you, go try it, it will show you a listing of all the content. There is a nice space up the top where we could put a drop down and a label like so:
Lets just assume we can do some Angular Injector magic and get this in there (lets also assume that this page will never change in Core so that this dark magic will never break...that is a whole other story).
Of course now we have a drop down we need to set those values somewhere as someone is going to want to change them on a site by site basis. So back to that config file for more settings. I don't know about you but I think I'd be happy with having those set in a config and they don't need a UI to edit the available options themselves, there are limits to how far we need to go here. But for our drop down to know about the options we now need a custom API endpoint that can read the config file and return the options.
Now there are a number of different ways to do this. I personally would go quick and dirty and just xpath it as I'm super happy with xpath, others like to make everything into objects (Go take a look at the HealthCheck config to see what I'm on about) which I personally find next to impossible to extend (as discussed here) either way it doesn't really matter but its a bit more effort on top of the pile already expended.
Now we are about there! However...translations, we didn't do translations. So we have to create some keys, add those in to the right file, change our Angular to use the keys and then hope that someone else can translate the other languages for us.
What about Courier resolvers or whatever Deploy uses...I've not idea? I think only Lee K does :)
Either way, that is a simple idea drowning in package admin that I think could just be greatly streamlined away so we could focus on the meat of the issue rather than the surrounding chores. Currently your ability to do Chores well is what makes the different from a nice idea package to a killer package and if you don't like Chores is easy to just take the easy route of doing that 30 minutes of start up event code and pushing it into your build and being done with it, never to share it with the wider world.
So how could we streamline it?
A generic settings store of some sort. A dictionary keyed to your package name or something like that:
var duration = Umbraco.Config.PackageConfig<int>["RecycleBinEmptier"]("duration");
That in a nut shell makes a lot of pain go away. We just need something like that built in. I'd suggest it stores settings in config files for ease (and you can source control it) or in a database if scaling is your bag? If you need more config then you can store JSON in your dictionary and do some deserialisation magic or just roll your own.
A generic way to get and set these config settings via Ajax calls would be great too. Send an Ajax payload up asking for the whole config dictionary or a part of it for a package and get it back as JSON. No custom endpoint required, its just works.
Then we have the AngularJS magic. We could do with there being set areas on all back office pages where you could safely inject some stuff and know its not going to break. A placeholder that you can ask to put something, that would be pretty cool. Not against doing the AngularJS work but messing/hacking the back office and hoping nothing breaks between versions is not best. Currently thats not baked in but with the idea of Content Apps coming in V8 maybe something around that idea could work?
Translations, we need them but they are a right pain as everyone does them so infrequently. I could live with spinning one up and putting it in the AngularJS (could do with some more docs on this though). But getting them translated should really be doable by the community. When you log into Our it could auto suggest files that need translating if it knew which languages you speak or some such. Or even flash a word up to translate almost as a recaptcha and slowly but surely we could get everything translated. I'd tripple key it to ensure no one ends up just answering "sitecore rocks" for everything of course.
Then there is the bundling and the nuget ness. These could do with some work or tools but thats another issue.
I hope you can see how the amount of work needed to make a nice "little" feature public mounts up to simply tip the scales over to it never getting done or not done well enough. We can and should do better but we need ti have a think and provide some tools to ease this pain.
There might already be a lot of this in place...I simply don't know about it, but I sort of know quite a bit about Umbraco so thats a worry. Do we need to do more to make this stuff more visible or joined up?
I'm tempted to do the above package as a case study to show people how to go about doing a package from start to finish so if you can think of anything more let me know. If you beat me to it by all means crack on with it but I hope you can see that some of the nuts and bolts of Umbraco currently are adding too many Chores to the todo list and not enough "doing".
As usual you raise some great points and I'll be honest, I'm writing this without having finished reading your post. But you know, when an Idea strikes it's sensible to share it :-)
Something that stuck out was your comment about settings and where they should be stored.
It would be really cool if Umbraco contained something like a "Simple Packages Settings" table in the database (with appropriate interfaces/pocos etc). This would allow something, like your example above, to have a location to store simple / non-deployment reliant settings in, for example, a key value pair, or a JSon blob that Umbraco loads into memory at start up. Rather than them being in config files where having to update them can trigger things like site restarts.
I talk about something specific for this rather that it polluting existing tables even though I know it could essentially be similar to the Property configs etc, mainly for the point of isolation.
It's just an idea that could help with package creation that require a few basic settings and can't justify a full config file.
Right back to reading the rest of the post :-)
I carried on reading and saw you suggested something similar to this
facepalm
I see your problem. My opinion this is something that should be built in the core. We could create a package to maintain these settings. But that would require people to install this first before installing your package. eg..you need to install InnerContent to use Stacked Content. If you forget to install InnerContent installing Stacked Content breaks your website. This is when you use a "umbraco" package. With nuget this is not a problem because you can mark the package as a dependency.
So that's why I think it should be in core.
What I see is that you could have different storage (eg database or config file). Because your example you don't want to run this event on every server in a loadbalanced environment. There you would only want to run it on the primary server.
And how will developers register a setting. I was thinking about a fluent API like Matt uses in his Fluidity package. So you could register a setting, set the view and other config options like your dropdown values for example.
You raise a really interesting point there Dave, package dependencies. I wanted to knock up a quick UI for the GraphQL package using Fluidity but then I'd have that as a dependancy. Or if I pulled my API permissions stuff out into another package that would be a dependancy. Or what if you already have an older version of the API permissions (which does exist so just imagine) install for another package and then I want to install another package that uses (or expects) a newer version...ouch. Yes Nuget will fix this and there was talk of trying to move to that in Core at some point. I might have to swerve away from that one for now and chose my battles but its an interesting problem and yet more Chores for package devs to think about.
This sort of stuff belongs in Core but how to get it in there. Have we had enough discussion that this is a good idea? If so then I can spin up an issue for a feature request and start specing out what I think we need. If not shout up now with ideas! :)
For example you have a version of stacked Content that uses v1 of InnerContent.
Then you install a new package that uses InnerContent v2. You will need to updated stacked content because of the dependency on InnerContent being updated. And that could be even not possible. And if so there could be breaking changes.
Suggested changes to ease the creation of packages
We have a Trello board full of little ideas for "some day maybe" type packages that we know we and others would like to reuse. None of them are really ground breaking but they all do a little something to automate or ease a task and make the general Umbraco experience a little nicer long term.
We've already built quite a few packages over the years and each time we notice the demand on whats required to make a package "good" gets longer and longer. This all sort of gets in the way of building what in essence is a simple feature.
Let me give you an example. From our work we did for the Back Office Speed Hacks talk at Code Garden (it was a good talk btw, well worth a watch!) one of my tips was empty the recycle bin regularly. Well why not have something that will do that for you automatically so you don't have to worry about it? Great! This is how that might look:
We hook into the app start event and do a quick search for all content in the recycle bin sorted by date and then delete the stuff thats X days old.
Easy! Could knock that up in about 30 minutes. But to package it up and make it reusable by others and ourselves we really need to go a bit further. What if I want to change the duration before we delete (the "X days" in the above spec)? Well we could just put that in a config. Ok which one? A custom one just for this? Seems a bit excessive but ok. But now the client says they can't find something they deleted last month, the X is too short can we change it. Sure! but its now a release we have to do and a deploy. Isn't this meant to be a CMS? Shouldn't I be able to change this in the UI? I mean its just a drop down not like I'm changing the connection string or anything!
So now we need a UI to change the X, this is a good thing because if we just stashed this away in a config its got no visibility, you won't see it so mgiht not even know its there. By having something you can find or see in the UI you allow for folks to explore and discover features and use them if needed.
So we need to inject an edit screen for a simple numeric field stored in a separate config. Lets say I'm going to try to inject it into the top of the Recycled area itself (you didn't know you can click on it directly did you, go try it, it will show you a listing of all the content. There is a nice space up the top where we could put a drop down and a label like so:
"Automatically delete content [Never|Weekly|Monthly|Quarterly]".
Lets just assume we can do some Angular Injector magic and get this in there (lets also assume that this page will never change in Core so that this dark magic will never break...that is a whole other story).
Of course now we have a drop down we need to set those values somewhere as someone is going to want to change them on a site by site basis. So back to that config file for more settings. I don't know about you but I think I'd be happy with having those set in a config and they don't need a UI to edit the available options themselves, there are limits to how far we need to go here. But for our drop down to know about the options we now need a custom API endpoint that can read the config file and return the options.
Now there are a number of different ways to do this. I personally would go quick and dirty and just xpath it as I'm super happy with xpath, others like to make everything into objects (Go take a look at the HealthCheck config to see what I'm on about) which I personally find next to impossible to extend (as discussed here) either way it doesn't really matter but its a bit more effort on top of the pile already expended.
Now we are about there! However...translations, we didn't do translations. So we have to create some keys, add those in to the right file, change our Angular to use the keys and then hope that someone else can translate the other languages for us.
What about Courier resolvers or whatever Deploy uses...I've not idea? I think only Lee K does :)
Either way, that is a simple idea drowning in package admin that I think could just be greatly streamlined away so we could focus on the meat of the issue rather than the surrounding chores. Currently your ability to do Chores well is what makes the different from a nice idea package to a killer package and if you don't like Chores is easy to just take the easy route of doing that 30 minutes of start up event code and pushing it into your build and being done with it, never to share it with the wider world.
So how could we streamline it?
A generic settings store of some sort. A dictionary keyed to your package name or something like that:
That in a nut shell makes a lot of pain go away. We just need something like that built in. I'd suggest it stores settings in config files for ease (and you can source control it) or in a database if scaling is your bag? If you need more config then you can store JSON in your dictionary and do some deserialisation magic or just roll your own.
A generic way to get and set these config settings via Ajax calls would be great too. Send an Ajax payload up asking for the whole config dictionary or a part of it for a package and get it back as JSON. No custom endpoint required, its just works.
Then we have the AngularJS magic. We could do with there being set areas on all back office pages where you could safely inject some stuff and know its not going to break. A placeholder that you can ask to put something, that would be pretty cool. Not against doing the AngularJS work but messing/hacking the back office and hoping nothing breaks between versions is not best. Currently thats not baked in but with the idea of Content Apps coming in V8 maybe something around that idea could work?
Translations, we need them but they are a right pain as everyone does them so infrequently. I could live with spinning one up and putting it in the AngularJS (could do with some more docs on this though). But getting them translated should really be doable by the community. When you log into Our it could auto suggest files that need translating if it knew which languages you speak or some such. Or even flash a word up to translate almost as a recaptcha and slowly but surely we could get everything translated. I'd tripple key it to ensure no one ends up just answering "sitecore rocks" for everything of course.
Then there is the bundling and the nuget ness. These could do with some work or tools but thats another issue.
I hope you can see how the amount of work needed to make a nice "little" feature public mounts up to simply tip the scales over to it never getting done or not done well enough. We can and should do better but we need ti have a think and provide some tools to ease this pain.
There might already be a lot of this in place...I simply don't know about it, but I sort of know quite a bit about Umbraco so thats a worry. Do we need to do more to make this stuff more visible or joined up?
I'm tempted to do the above package as a case study to show people how to go about doing a package from start to finish so if you can think of anything more let me know. If you beat me to it by all means crack on with it but I hope you can see that some of the nuts and bolts of Umbraco currently are adding too many Chores to the todo list and not enough "doing".
Hi Pete,
As usual you raise some great points and I'll be honest, I'm writing this without having finished reading your post. But you know, when an Idea strikes it's sensible to share it :-)
Something that stuck out was your comment about settings and where they should be stored.
It would be really cool if Umbraco contained something like a "Simple Packages Settings" table in the database (with appropriate interfaces/pocos etc). This would allow something, like your example above, to have a location to store simple / non-deployment reliant settings in, for example, a key value pair, or a JSon blob that Umbraco loads into memory at start up. Rather than them being in config files where having to update them can trigger things like site restarts.
I talk about something specific for this rather that it polluting existing tables even though I know it could essentially be similar to the Property configs etc, mainly for the point of isolation.
It's just an idea that could help with package creation that require a few basic settings and can't justify a full config file.
Right back to reading the rest of the post :-)
Hi Pete,
I see your problem. My opinion this is something that should be built in the core. We could create a package to maintain these settings. But that would require people to install this first before installing your package. eg..you need to install InnerContent to use Stacked Content. If you forget to install InnerContent installing Stacked Content breaks your website. This is when you use a "umbraco" package. With nuget this is not a problem because you can mark the package as a dependency.
So that's why I think it should be in core.
What I see is that you could have different storage (eg database or config file). Because your example you don't want to run this event on every server in a loadbalanced environment. There you would only want to run it on the primary server.
And how will developers register a setting. I was thinking about a fluent API like Matt uses in his Fluidity package. So you could register a setting, set the view and other config options like your dropdown values for example.
Something like this
I think building this would not be that hard if it is integrated in core. But would love to hear some input from HQ on this.
Dave
You raise a really interesting point there Dave, package dependencies. I wanted to knock up a quick UI for the GraphQL package using Fluidity but then I'd have that as a dependancy. Or if I pulled my API permissions stuff out into another package that would be a dependancy. Or what if you already have an older version of the API permissions (which does exist so just imagine) install for another package and then I want to install another package that uses (or expects) a newer version...ouch. Yes Nuget will fix this and there was talk of trying to move to that in Core at some point. I might have to swerve away from that one for now and chose my battles but its an interesting problem and yet more Chores for package devs to think about.
This sort of stuff belongs in Core but how to get it in there. Have we had enough discussion that this is a good idea? If so then I can spin up an issue for a feature request and start specing out what I think we need. If not shout up now with ideas! :)
Hi Pete,
Even with nuget you can have version conflicts.
For example you have a version of stacked Content that uses v1 of InnerContent.
Then you install a new package that uses InnerContent v2. You will need to updated stacked content because of the dependency on InnerContent being updated. And that could be even not possible. And if so there could be breaking changes.
My opinion should be in core.
Dave
I've spun up two issues for this:
"Create a generic store for simple package config settings" https://github.com/umbraco/Umbraco-CMS/issues/3911
and
"Add safe placeholder areas that we can inject custom AngularJS UI into" https://github.com/umbraco/Umbraco-CMS/issues/3910
is working on a reply...