Is feature development via packages a help or hinderance to progress?
I had an issue closed last week that I had an interest in. The parting comment on it had the sentiment of "this would make a great package, work out the issues there and then we might look to put it into core via a PR".
The underlying message was "do your experimentation in a package not the Core, then if and only if its good enough should it be reworked to get into Core...maybe"
I don't know why but this left a bad taste in my mouth. I pondered it a while to ensure it wasn't just a knee jerk reaction or my "chimp" being angry. Then I went to the white board and wrote down my thoughts on why it irked me so. I'd like to run them by you all to open up the conversation, I was going to put this up as a blog but we don't have comments on there (for a reason) and I think it will need some discussion so here it will live.
I've noticed a few features get the "make this into a package and then we might be interested" responses. Typically I can't find any as the search on the issue tracker it horrid. This line might not be used very often and I hope not but when it is used it can cause some friction.
I dislike the idea that features or even experimentation should be done as a package and prove their worth before HQ will take a look for a number of reason:
Package development ain't free, its quite time consuming to pull a package together these days and go through all the steps to get it up and out into the wild.
It often leads to lots of extra work and work arounds, hitting with hammers and over writing to get the back office to do what you want it to rather than editing the source directly. This means precious "interest points" for developers already stretched on time and takes their focus away from fixing the problem to doing chores around the problem.
It requires people to actually install your package and trying it which then opens up a spare time dev to now support and helping out on an idea when things go wrong, regenerate the package, re upload it. All doable but more effort than just a commit and push.
Updates might not get installed which in turn might break future upgrades to Umbraco and so on
Few successful packages get pulled into Core, they end up living on as separate projects. Great for HQ, less support required from an already over-stretched team. Possibly good for the developer as they get some Kudos but how many other users miss out on what might be a great generic feature that they just don't know about or find?
An generic is the key word there. If you idea is very very very niche or super edge case then yes a package is the best way to go. But when its something that could be of benefit to the many then it should be in Core. But this leads on to bigger questions like who judges that and who are the gate keepers? I prefer a good rule of thumb, if it feels like its a Core thing then it should be in Core. I'd say that this image rotation package for instance from Marc Goodson is a belter and should be in core. Its smart, simple and useful to lots of people. But lots of you might never have heard of it, as its a package and doesn't get shouted about much. It shouldn't be a package anymore, its should really be in Core. But that is just my gut feeling and who the hell am I? I don't make the choices.
I could spin up a PR to get it into Core of course and let those in HQ make that choice instead but once its out there as a package and fixing the original developers immediate problem the drive to make it into a PR diminishes. Another reason why we shouldn't by default be pushing people to work on packages. Good ideas are being left behind by doing this.
The only benefits to doing features as packages seems to be:
An escape hatch for niche feature requests ("can we add a MySpace account picker to Core please"?)
The ease with which others can install it and try it out.
The shifting of responsibility to the community rather than HQ for managing it until its "ready".
Suggesting a feature should be a package is an easy way to kick the can down the road for HQ but I don't think its one that should be dusted down too often personally if we want to keep making this thing awesome.
The feedback that sparked this particular issue is from real world experience and requests rather than just a crazy idea dream up in the lift at work. As users and developers using Umbraco day to day when we raise an issue based on real user need it should be heeded (IF its generic enough to be helpful to many). I could see the need for the "create a package" escape hatch (should this be know as a CrAP response?) for some of those crazy super edge case feature requests but I don't want it to become the norm.
The whole point of raising an issue is to get eyes on it, brains on it and thought on it. Thinking is the hardest part often with an idea (after actually having the idea of course) and that is best done on the issue tracker rather than burning time coming up with a package in isolation.
Lots of the issues we that are getting logged and fixed take time, I simply don't think adding the weight of package development into the mix is worth while (just look at Owin's "simple" looking colour picker for an example of how a simple looking idea can turn into an epic amount of work and a great outcome https://github.com/umbraco/Umbraco-CMS/pull/2555), this is what branches are for is it not? Faster and easier and you get full control to edit what you need to get the changes needed to get your feature in and working. No need to "strip out" all your hacks and work arounds once your ideas have proved their worth, your PR should in theory be accepted quicker.
"Sometimes" a feature makes sense as a package, I'm just against good features being pushed out as packages when they really would make better features.
There are a couple of other thoughts I had while reading it and to me (someone who hasn't yet made a package) kinda important.
Sometimes a dev can know what they want to produce, how to produce it for a single instance, but not how to package it and get it out into the wild. It can be much easier to do your prototype/idea in a branch of the core Source because, as you say, less hacking. Not everyone is knowledgeable on hacking around Angular to extend a controller, hook into a service etc. I know I'm not.
If you want to add a feature to something core, are there any guides on how you do that in a package?
There aren't good places to get a potentially long term discussion going for feature/package ideas imo. The Issue tracker, as you say, has poor search and we know they are planning on moving away from it to GitHub at some point. Discussions get lost on Our after a short period. We don't really want conversations being "bumped" just to get it up the list for people to see.
So those are my initial query based thoughts. But I also had a thought on something that could help devs make decisions on their ideas.
It would be awesome, and I know it would take time, if the HQ could (possibly working with some of the big package dev's out there), produce a decision matrix style thing to guide people. The outcome being should this be a package, a prototype, or a core PR. I appreciate each feature is unique and needs to be evaluated on it's own merit, but it could help improve what goes where?
I like the matrix idea Nik. That might be something worth looking at, it need not be anything too detailed but a little something so we all know roughly what the guidance is would be great I think.
I'm all for easing the burden for getting people involved while still ensuring that that involvement is constructive and helpful. If you've never done a PR before then Hackathons and Virtual Meetups can do wonders for getting you over the hump.
As usual although this will all be seen as a rant/nag its more meant to be an effort to raise the issue so we can discus a solution and said with love and a smile (as ever).
I've done a core PR, but I've never made a package :-) And that is something else that maybe more guidance is needed on. I wonder how many people would find it easier to do a PR they would to take their idea and package it.
I think it would be good if there was a formal way to submit packages for Core consideration as well. There are some packages out there that, as you rightly say, should probably be in the core. Things do get lost on the Issue Tracker, so maybe a different way of tracking those considerations would be useful?
Just reporting on my experience on making Umbraco packages here: it was terrible. Absolutely terrible.
I think is a very important functionality and some work should be put on making it effortless because 1) people give up on sharing their code and 2) devs copy/paste lots of code to avoid making packages.
Great initiative, Pete. I concur with Nik's matrix idea. Sounds really helpful.
Here's my few cents:
[rant] First of all, a fairly non constructive argument, but the current package format and "noob friendliness" sucks. The whole "make it easy for noobs" by providing a zip you can unzip to IIS, the cloud project structure and the install package via backoffice shit is just encouraging peeps to make messes. IMO it would be a lot better if everyone learned to make "proper" c# projects and used nuget for everything. I know Shannon for one is eager on moving over to nuget for packages, but I'm fairly sure HQ as a whole does not prioritize it what-so-ever. It's a developer feature, and I don't know if that's interesting to HQ at all any more. After all, they have "won" the devs already, so no need to make them happier. [/rant]
That being as it is, I'm all for encouraging niche features to be made into packages instead of PRs. However, several times the package will, as you say, be very hard to do. Several of my feature ideas have basically not been possible to do as packages, so I've had them as "maturing" PRs for years.
To get around all the hacking, there is one option I think might be worth investiaging further. If the idea is good but hard to do as a package, the solution would be to add the necessary extension points to the core.
It would make the core adhere more to the open closed principle, and it would possibly open up for more good features. Even new ideas since the option to extend is there.
The dev making the feature might not be skilled enough to add that extension point as a PR though, but maybe the new PR review team could help with that, or relay the extension need to community members they know can do it in a jiffy.
For instance, I added several events to the content editor and the grid at the retreat this summer, which makes it possible to package my "grid navigator", mine and skttl's copy/paste feature for the grid, extra validation and business logic for content open/save etc.
The implications for the core were minimal, but for me as a package dev it was everything I needed to easily add those packages without hacking or preventing further updates. (Which is what happens when you hack the core)
Then there's visibility. I think there's been some work done to increase visibility of new packages on our, but I'm willing to bet it's damn near impossible to get a package out there for new, shy or otherwise unknown devs. It really takes quite some marketing, even if it's a top notch feature. Which implies the "feature" will never be brought into the core because nobody uses it and it doesn't have any downloads. Not because it sucks, but because nobody knows about it.
I really have no good suggestion for that, but it's a big challenge if CRAP is more and more the default answer. Love that abbreviation btw. :)
Gotta run for shrimps, beer and sun now, but love the thread. Excited to see where it leads.
Personally I've developed a few things over the years that have saved some time and might of been useful to others, but I've shied away from developing packages as I know I won't have the time to dedicate to replying to the questions that may come up and then review and possibly refactor the code every time a new version of Umbraco comes out. Also, as mentioned, creating a package hasn't always been a straight line of there to here.
That being said, I can understand why HQ would prefer to go this route than have everyone dump their half baked and dare I say it, code rantings and ravings of their latest good idea into a branch or something for them to deal with. It'll be impossible for them to review them all and at least this way they get some code that's been nurtured and refined and is popular, which makes it an easy win to please us all when it gets integrated into the core.
tl;dr I agree with you and see the HQ point of view, but not sure what the answer is!
Having done both packages and core PR's I've come to this conclusion for myself:
Packages are indeed safer for the HQ as they can have very little care of the maintenance. However Umbraco Deploy/Courier threatens that model. e.g. If a package becomes popular they may have to support an adapter for Deploy/Courier.
Packages are prone to being broken since the dependency graph points to the Core and not the other way around. So it's a huge "ask" to have package developers keep maintaining plugins over time due to breaking changes. Again this benefits the HQ.
Core features are very susceptible to not being accepted for many reasons: code style/hygiene, complexity, need/ROI, etc. My biggest issue with doing core PR's is the dissonance between what I consider a great feature and what the HQ considers a great feature. If I have a short-term need, I can create a private package or a public one. Either option is fairly quick (for me at least). Trying to get it into the core is a long term thing and I've likely got no time to wait. Packages in this case benefit the builder.
Packages are more friendly to older versions of the core whereas core PR's require that you adopt v.latest of the core. I know many many companies who don't want the latest due to stability issues and/or compatibility issues. PR's require future version whereas a package can be backwards compatible.
Getting something into the core isn't necessarily a good thing. Nested Content has gone core now it seems that (like the Grid) will be more or less not curated like it would have been if it were a separate package. All inquiries to NC need now go to the HQ. This is a wash for me since now it's technically supported.
Umbraco is a for-profit business. OSS doesn't change that fact. I appreciate that it is OSS and free (so don't misinterpret my appreciation). Their reason for accepting features has to be guided by what's best for the company. It's very easy to keep a layer of new features in an incubator (plugins) to vet them. HQ benefits by being choosey.
Core PR's that are small are likely to get merged, adding anything remotely complex is a risk for the HQ. They have to maintain it going forward. Any PR accepted is synonymous with accepting responsibility. Responsibility incurs risk and it must be accepted or mitigated.
So in conclusion, as a business; sending requests out to be a package is safe and easy for the HQ. Core PR's are risky incur a maintenance cost (potentially). We all think our idea is useful to the masses when it's probably actually not.
I think it's in the HQ's best interest to apply the "CRAP" pattern to be perfectly honest, then cherry pick the ideas they deem worthy.
As time goes on, I think PR's will come less and less to the core if more CRAP happens. Maybe that's ok. In other OSS projects I have been involved with (OBS for instance), plugins are usually never meant to be into the core ever. If Umbraco were like OBS, Umbraco might be more stripped down and the plugins would be more of the status quo.
I'm a proponent of packages for just about everything outside of the core functionality. This is completely biased though based on my skills. Umbraco tends to want to favor the "everything out of the box you need" solution. That means in order to get into the core, you have to be sort of watered down and generic.
Getting something into the core isn't necessarily a good thing. Nested Content has gone core now it seems that (like the Grid) will be more or less not curated like it would have been if it were a separate package. All inquiries to NC need now go to the HQ. This is a wash for me since now it's technically supported.
Maybe, but when it comes to develop a new site if you use a functionality that's in core then you are at least guaranteed that it will keep working. If you use a package it could get broken at any upgrade (as it happened to DTGE a few minor versions ago) and every time you upgrade you'll have to check manually.
Their reason for accepting features has to be guided by what's best for the company.
True, but naturally a OSS for-profit company owe something to the community that develop their product and, at the same time, if they make harder for developers to contribute then they'll get less contribution.
Yep, packages incur upgrade risk. 100% agree. But this is give/take. Umbraco can't put all packages in the core either. They simply can't support them all. Also anything in the core also means a fix is dependent upon a future release of Umbraco (longer) than a package release (shorter).
Of course the package dev could simply stop supporting it.
Pete, I completely understand your trepidation when receiving the CrAP* message, but upon reading Kevin's succinct reply, I certainly appreciate both points of view.
As a developer who's dabbled in packages, I completely and wholeheartedly agree with Stefano on it's a right PiTa to develop them.
I also agree with Lars, and Shannon indirectly, that we need to move over to Nuget for distribution for a wealth of reasons, like handling dependencies, for example, that will make developing packages a whole lot easier for everyone.
Thanks for putting this up for us all, looking forward to more replies!
Great discussion all and many great points overall. Without re-iterating the above I just want to add a couple of my own thoughts.
My own opinion is that a large part of the challenges mentioned above is related to the package experience and ecosystem and maybe not so much related to whether a feature should exist in Core vs Package. Much of what is mentioned above relates to:
Packages don't seem like first class citizens.
It is difficult to know they exist.
It's not easy to find the right package for your needs.
Sometimes it's not easy to create packages.
Sometimes it's not possible to create a package because the extension points required in the CMS don't exist.
Maintaining a package can be difficult.
I feel like if those issues are tackled that we would overcome many of the issues voiced here. I would love to see the package ecosystem thrive but in order for that to happen I think these things should be considered:
Package format changed to Nuget and Package Migrations in place: http://issues.umbraco.org/issue/U4-8605, which lends itself to easier package development and maintenance
Back office packaging system updated to make it easier to create and maintain packages and some best practice docs/scripts created to support this too.
When extension points in the CMS are missing, they are created. Generally this is fairly simple an unobtrusive to do.
Update Our and the Back office packaging area to make it more robust and somehow improve visibility of popular packages to users, somehow improve searching and finding a package.
Updating Our and the Back office to prompt the user to vote on the package whenever it's installed ... without votes its difficult to know what is popular
Somehow promote the package ecosystem, make this a first class citizen, inspire the community that packages are a good thing. Make Packages Great Again :P
I'm not saying that this is the solution to everything mentioned above, just noting that I think this is one avenue that would alleviate some of the current problems mentioned with packages.
Hi Shannon, I'm all for making packaging easier, smoother and searchable.
My biggest grumble was with the advice to develop a feature as a package rather than in a branch of the source. I get why it might work for some features but I didn't want it to become the norm. That was my main grumble about package and one I think should been taken on board.
If as a bonus of that we end up with a better package system then great! But what you list sounds like a big chunk of work and effort. A flow chart about when to make something into a package or not might be a quicker first step? :)
Good discussion. I think this was the 2nd time we've ever encouraged anyone to create a package for. It could have been clearer though:
There's a few unknowns which will result in discussions to be had around the proposed feature
The proposed feature .. sounds like a big chunk of work :)
If you're truly in need of this feature at the moment, you shouldn't wait for core to implement it but develop it custom for now
If you're already making this custom, it would be great to open it up, indeed as an incubator
There's currently no evidence for large demand of the proposed feature, it was requested first a few months ago and has 1 vote (I know it is now easy to go on Twitter and lobby for more upvotes ;-) )
In any case, we will sparingly suggest people who are passionate about a feature that we're not sure about to create a package.
Additionally, we're working on publishing our process for dealing with new incoming issues and PRs so that it doesn't feel as random.
One of the more important parts in that process is that we really would like to get away from stockpiling issues. We'll attempt to classify everything coming in, instead of just letting issues linger forever.
More news to come when the Danish holidays are over and everybody is back in full force again!
Great discussion. As someone who's a bit nervous about approaching package building, I suggested a "Package Building Course" at CG18. I wonder if many would be interested and if it's a flyer? I'm thinking it might help standardise the approach, help with the logistics, and relieve anxiety.
For me personally the key point from Seb's reply is "no evidence for large demand of the proposed feature"...
I love Umbraco and the fact that at it's core it is a focussed well built app. You then build what you want, either by installing packages, or via your own cleverness.
Is feature development via packages a help or hinderance to progress?
Yes, it is a hindrance. Having been involved with porting a few packages (or features from packages) over to the Umbraco core, I can say that the effort involved is quite significant. This isn't a fault of Umbraco by any means, I'm only highlighting that porting requires additional time & effort, (on both sides).
It involves many discussions that unpick the decisions you made for your own package and how they fit architecturally inside Umbraco core.
(To note, I wouldn't want anyone to think the effort hasn't been worthwhile - the benefits outweigh the effort!)
With my own packages, I've never started to develop a package with the intend of it becoming a feature of Umbraco core.
That thinking feels backwards to me. If I felt a feature was right (and justified) for Umbraco core, I would try to put forward my case (via the issue tracker, or sometimes lesser formal channels - note: I appreciate that I'm in a somewhat privileged position here).
If I did get a friendly "CrAP" response, I probably wouldn't develop as a package. In my mind, I'd see it as a failure of my "sales pitch" (so to speak). Like Seb says, there'd be either too many unknowns or it'd be a big chunk of work. To which I'd ask myself can I clarify this feature? / could the feature be broken down incrementally?
It's been good to read everyone's thoughts on this discussion. I do see this more as a "how to sell your feature idea" discussion, rather than highlight the woes of current package development.
I was part of the open space session at CG18 that talked about all things package-related, and my takeaway from that was there are plenty of punters interested in writing packages, but unsure how to start. No doubt they have the technical nous to do so, it's just a bit of a black box - not because it's overly complicated, just not perhaps as clearly a defined process as it could be.
I enjoy building packages. I can have an idea and turn it into a thing, so long as it fits within Umbraco's constraints (haven't had any issues as yet).
I've worked out a workflow that works for me, around tooling and structure and so forth, but highly likely what I'm doing is completely different from the next package dev. Neither is probably more correct, but it wouldn't hurt to establish a best practice model.
In the open space chat, we touched on the idea of a boilerplate for packages. It's probably unrealistic to standardise completely, which is unfortunate, but some clearer definition around what to do, what not to do, would be really useful. I've thought about creating a git repo with a package framework that fits my approach. A package package.
As for the core vs package question, I think going package-first is an opportunity to prove value. If a proposed feature, delivered as a package, sees great uptake, then sure, maybe it belongs in the core.
But looking at the ecosystems around other CMS platforms, plugins/extensions/packages are an industry unto themselves. People make a living off one or two well-executed ideas. Having a downstream ecosystem is evidence of a mature product.
Build a marketplace for packages and the desire to move into the core would diminish if package development is a financially viable option. That might be difficult given the perception that open source means free, but difficult problems are the rewarding ones to solve.
I think the ideas are important, it's sad when they are lost or waylaid.
I think it's incredibly useful to know up front whether HQ think an idea would be good to develop for the core (or if they are in the process of building something similar) or not, or if it is more niche and would suit a package, you approach the two end results in different ways, so knowing upfront can really save time, and be motivating.
With introversion rife within our industry though, it's sometimes difficult for a person to suggest an idea, without implicitly putting pressure on themselves to build it, either as a PR or a package.
If you are working on something for the core, then you receive feedback and can reach out for help on your PR, if you are building a package then there isn't necessarily anywhere specific to turn to for help, which I guess is why it might feel like a negative thing... when it's not meant to be.
Think we lack a friendly place to discuss, and incubate ideas, or maybe not a focussed place - 'eg there are posts on the forum, slack, the new gitter channel I think will help with this, but be great to have a specific place for ideas, (not necessarily issues, ideas can be half baked... they need to be aired and discussed, and often they are, particularly at meetups and festivals).
I think there is also an obsession in the industry for wanting to be 'the person' that built 'the thing', that doesn't always encourage discussion and collaboration.
It's hard for anyone to make the decision on package vs core, without fully understanding the idea, it's hard to illustrate the idea without building something, have found videos of rough functionality really useful to get across quite what you mean, building a prototype and blogging, but then time passes, and the ideas fade, (and what if someone builds the thing I have prototyped, crikey, I won't be 'the person'!) - Had thought before about a semi regular, webinar with core dev team where ideas are pitched/evaluated, and maybe put into room 101, increase visibility of ideas, and how HQ weigh them up (having to maintain the functionality vs benefits of functionality being there in the core), I don't know how this would work, maybe the gitter channel is enough.
A lot of this is all really a symptom of the scaling of Umbraco, and the shortfall in discussing and responding to issues and pull requests, which everyone is painfully aware of, and experiments are in place to start to address this, which will take time to unfold...
But additionally another thing to consider, switching back to the example of the ability to rotate images in the backoffice (in Pete's opening gambit) an issue was created in September 2016 and is marked as up for grabs, I developed the uSpinMeRound package largely to make the joke about the title, unaware of the issue, then discovered it http://issues.umbraco.org/issue/U4-9028 and now that issue has a comment (May 2017):
Now I know I'd need to rework this to put this into the core, and I'd be happy to do this, but if you see my introversion around how I've phrased my comment... I'd do this if you thought it was right... having no reply on this issue makes me conclude 'yeah, we don't like what you've done'.
When in reality I sort of know 'yeah, sorry, we haven't seen this, is there an issue on the tracker about it?'.
So eventually Seb will wade through and discover this, and we'll pick up on it again later, and we just have to accept the shortfall was there, and it will take time to recover -
But I use the example to also raise the point from a community perspective the issue only has 4 votes... and the package only has 4 hearts.. so it's hard to consider this as 'a belter that should be in the core' - Why haven't you voted for it Pete?, why has't anyone voted for it? (It's shit Marc, please let it go) - or is this the bit that is also a little bit broken?,
mechanisms are there for people to provide feedback on issues and packages - but we've somehow lost the culture around doing so, which ties into Shannon's observations on sorting out and then re-promoting the package ecosystem, which I won't repeat.
Sorry if you read this far, I got over excited searching for an old reply with my name in and this was the first post that came up.
Up voting stuff. I've got an issue with that. We were warned off it during the death of V5..."You all up vote packages when you've not even downloaded them" and we've also been trained that up voting an issue isn't a sign of something needing to be done:
Is feature development via packages a help or hinderance to progress?
I had an issue closed last week that I had an interest in. The parting comment on it had the sentiment of "this would make a great package, work out the issues there and then we might look to put it into core via a PR".
The underlying message was "do your experimentation in a package not the Core, then if and only if its good enough should it be reworked to get into Core...maybe"
I don't know why but this left a bad taste in my mouth. I pondered it a while to ensure it wasn't just a knee jerk reaction or my "chimp" being angry. Then I went to the white board and wrote down my thoughts on why it irked me so. I'd like to run them by you all to open up the conversation, I was going to put this up as a blog but we don't have comments on there (for a reason) and I think it will need some discussion so here it will live.
I've noticed a few features get the "make this into a package and then we might be interested" responses. Typically I can't find any as the search on the issue tracker it horrid. This line might not be used very often and I hope not but when it is used it can cause some friction.
I dislike the idea that features or even experimentation should be done as a package and prove their worth before HQ will take a look for a number of reason:
An generic is the key word there. If you idea is very very very niche or super edge case then yes a package is the best way to go. But when its something that could be of benefit to the many then it should be in Core. But this leads on to bigger questions like who judges that and who are the gate keepers? I prefer a good rule of thumb, if it feels like its a Core thing then it should be in Core. I'd say that this image rotation package for instance from Marc Goodson is a belter and should be in core. Its smart, simple and useful to lots of people. But lots of you might never have heard of it, as its a package and doesn't get shouted about much. It shouldn't be a package anymore, its should really be in Core. But that is just my gut feeling and who the hell am I? I don't make the choices.
I could spin up a PR to get it into Core of course and let those in HQ make that choice instead but once its out there as a package and fixing the original developers immediate problem the drive to make it into a PR diminishes. Another reason why we shouldn't by default be pushing people to work on packages. Good ideas are being left behind by doing this.
The only benefits to doing features as packages seems to be:
Suggesting a feature should be a package is an easy way to kick the can down the road for HQ but I don't think its one that should be dusted down too often personally if we want to keep making this thing awesome.
The feedback that sparked this particular issue is from real world experience and requests rather than just a crazy idea dream up in the lift at work. As users and developers using Umbraco day to day when we raise an issue based on real user need it should be heeded (IF its generic enough to be helpful to many). I could see the need for the "create a package" escape hatch (should this be know as a CrAP response?) for some of those crazy super edge case feature requests but I don't want it to become the norm.
The whole point of raising an issue is to get eyes on it, brains on it and thought on it. Thinking is the hardest part often with an idea (after actually having the idea of course) and that is best done on the issue tracker rather than burning time coming up with a package in isolation.
Lots of the issues we that are getting logged and fixed take time, I simply don't think adding the weight of package development into the mix is worth while (just look at Owin's "simple" looking colour picker for an example of how a simple looking idea can turn into an epic amount of work and a great outcome https://github.com/umbraco/Umbraco-CMS/pull/2555), this is what branches are for is it not? Faster and easier and you get full control to edit what you need to get the changes needed to get your feature in and working. No need to "strip out" all your hacks and work arounds once your ideas have proved their worth, your PR should in theory be accepted quicker.
"Sometimes" a feature makes sense as a package, I'm just against good features being pushed out as packages when they really would make better features.
Nice discussion Pete :-)
There are a couple of other thoughts I had while reading it and to me (someone who hasn't yet made a package) kinda important.
Sometimes a dev can know what they want to produce, how to produce it for a single instance, but not how to package it and get it out into the wild. It can be much easier to do your prototype/idea in a branch of the core Source because, as you say, less hacking. Not everyone is knowledgeable on hacking around Angular to extend a controller, hook into a service etc. I know I'm not.
If you want to add a feature to something core, are there any guides on how you do that in a package?
There aren't good places to get a potentially long term discussion going for feature/package ideas imo. The Issue tracker, as you say, has poor search and we know they are planning on moving away from it to GitHub at some point. Discussions get lost on Our after a short period. We don't really want conversations being "bumped" just to get it up the list for people to see.
So those are my initial query based thoughts. But I also had a thought on something that could help devs make decisions on their ideas.
It would be awesome, and I know it would take time, if the HQ could (possibly working with some of the big package dev's out there), produce a decision matrix style thing to guide people. The outcome being should this be a package, a prototype, or a core PR. I appreciate each feature is unique and needs to be evaluated on it's own merit, but it could help improve what goes where?
I like the matrix idea Nik. That might be something worth looking at, it need not be anything too detailed but a little something so we all know roughly what the guidance is would be great I think.
I'm all for easing the burden for getting people involved while still ensuring that that involvement is constructive and helpful. If you've never done a PR before then Hackathons and Virtual Meetups can do wonders for getting you over the hump.
As usual although this will all be seen as a rant/nag its more meant to be an effort to raise the issue so we can discus a solution and said with love and a smile (as ever).
I've done a core PR, but I've never made a package :-) And that is something else that maybe more guidance is needed on. I wonder how many people would find it easier to do a PR they would to take their idea and package it.
I think it would be good if there was a formal way to submit packages for Core consideration as well. There are some packages out there that, as you rightly say, should probably be in the core. Things do get lost on the Issue Tracker, so maybe a different way of tracking those considerations would be useful?
Just reporting on my experience on making Umbraco packages here: it was terrible. Absolutely terrible.
I think is a very important functionality and some work should be put on making it effortless because 1) people give up on sharing their code and 2) devs copy/paste lots of code to avoid making packages.
Great initiative, Pete. I concur with Nik's matrix idea. Sounds really helpful.
Here's my few cents:
[rant] First of all, a fairly non constructive argument, but the current package format and "noob friendliness" sucks. The whole "make it easy for noobs" by providing a zip you can unzip to IIS, the cloud project structure and the install package via backoffice shit is just encouraging peeps to make messes. IMO it would be a lot better if everyone learned to make "proper" c# projects and used nuget for everything. I know Shannon for one is eager on moving over to nuget for packages, but I'm fairly sure HQ as a whole does not prioritize it what-so-ever. It's a developer feature, and I don't know if that's interesting to HQ at all any more. After all, they have "won" the devs already, so no need to make them happier. [/rant]
That being as it is, I'm all for encouraging niche features to be made into packages instead of PRs. However, several times the package will, as you say, be very hard to do. Several of my feature ideas have basically not been possible to do as packages, so I've had them as "maturing" PRs for years.
To get around all the hacking, there is one option I think might be worth investiaging further. If the idea is good but hard to do as a package, the solution would be to add the necessary extension points to the core. It would make the core adhere more to the open closed principle, and it would possibly open up for more good features. Even new ideas since the option to extend is there.
The dev making the feature might not be skilled enough to add that extension point as a PR though, but maybe the new PR review team could help with that, or relay the extension need to community members they know can do it in a jiffy.
For instance, I added several events to the content editor and the grid at the retreat this summer, which makes it possible to package my "grid navigator", mine and skttl's copy/paste feature for the grid, extra validation and business logic for content open/save etc.
The implications for the core were minimal, but for me as a package dev it was everything I needed to easily add those packages without hacking or preventing further updates. (Which is what happens when you hack the core)
Then there's visibility. I think there's been some work done to increase visibility of new packages on our, but I'm willing to bet it's damn near impossible to get a package out there for new, shy or otherwise unknown devs. It really takes quite some marketing, even if it's a top notch feature. Which implies the "feature" will never be brought into the core because nobody uses it and it doesn't have any downloads. Not because it sucks, but because nobody knows about it.
I really have no good suggestion for that, but it's a big challenge if CRAP is more and more the default answer. Love that abbreviation btw. :)
Gotta run for shrimps, beer and sun now, but love the thread. Excited to see where it leads.
I agree with you Pete.
Personally I've developed a few things over the years that have saved some time and might of been useful to others, but I've shied away from developing packages as I know I won't have the time to dedicate to replying to the questions that may come up and then review and possibly refactor the code every time a new version of Umbraco comes out. Also, as mentioned, creating a package hasn't always been a straight line of there to here.
That being said, I can understand why HQ would prefer to go this route than have everyone dump their half baked and dare I say it, code rantings and ravings of their latest good idea into a branch or something for them to deal with. It'll be impossible for them to review them all and at least this way they get some code that's been nurtured and refined and is popular, which makes it an easy win to please us all when it gets integrated into the core.
tl;dr I agree with you and see the HQ point of view, but not sure what the answer is!
Comment author was deleted
Having done both packages and core PR's I've come to this conclusion for myself:
So in conclusion, as a business; sending requests out to be a package is safe and easy for the HQ. Core PR's are risky incur a maintenance cost (potentially). We all think our idea is useful to the masses when it's probably actually not.
I think it's in the HQ's best interest to apply the "CRAP" pattern to be perfectly honest, then cherry pick the ideas they deem worthy.
As time goes on, I think PR's will come less and less to the core if more CRAP happens. Maybe that's ok. In other OSS projects I have been involved with (OBS for instance), plugins are usually never meant to be into the core ever. If Umbraco were like OBS, Umbraco might be more stripped down and the plugins would be more of the status quo.
I'm a proponent of packages for just about everything outside of the core functionality. This is completely biased though based on my skills. Umbraco tends to want to favor the "everything out of the box you need" solution. That means in order to get into the core, you have to be sort of watered down and generic.
Just my humble opinion.
Maybe, but when it comes to develop a new site if you use a functionality that's in core then you are at least guaranteed that it will keep working. If you use a package it could get broken at any upgrade (as it happened to DTGE a few minor versions ago) and every time you upgrade you'll have to check manually.
True, but naturally a OSS for-profit company owe something to the community that develop their product and, at the same time, if they make harder for developers to contribute then they'll get less contribution.
Comment author was deleted
Yep, packages incur upgrade risk. 100% agree. But this is give/take. Umbraco can't put all packages in the core either. They simply can't support them all. Also anything in the core also means a fix is dependent upon a future release of Umbraco (longer) than a package release (shorter).
Of course the package dev could simply stop supporting it.
Not a simple solution at all.
No need to pull in "all the packages" thats never been on the cards and why would it be. No need to go to extremes ;)
Comment author was deleted
I never claimed they would. Simply saying since they cannot, they must make decisions based on some sort of litmus test.
Throwing my 2p in too!
Pete, I completely understand your trepidation when receiving the CrAP* message, but upon reading Kevin's succinct reply, I certainly appreciate both points of view.
As a developer who's dabbled in packages, I completely and wholeheartedly agree with Stefano on it's a right PiTa to develop them.
I also agree with Lars, and Shannon indirectly, that we need to move over to Nuget for distribution for a wealth of reasons, like handling dependencies, for example, that will make developing packages a whole lot easier for everyone.
Thanks for putting this up for us all, looking forward to more replies!
*Love this!
Great discussion all and many great points overall. Without re-iterating the above I just want to add a couple of my own thoughts.
My own opinion is that a large part of the challenges mentioned above is related to the package experience and ecosystem and maybe not so much related to whether a feature should exist in Core vs Package. Much of what is mentioned above relates to:
I feel like if those issues are tackled that we would overcome many of the issues voiced here. I would love to see the package ecosystem thrive but in order for that to happen I think these things should be considered:
I'm not saying that this is the solution to everything mentioned above, just noting that I think this is one avenue that would alleviate some of the current problems mentioned with packages.
Hi Shannon, I'm all for making packaging easier, smoother and searchable.
My biggest grumble was with the advice to develop a feature as a package rather than in a branch of the source. I get why it might work for some features but I didn't want it to become the norm. That was my main grumble about package and one I think should been taken on board.
If as a bonus of that we end up with a better package system then great! But what you list sounds like a big chunk of work and effort. A flow chart about when to make something into a package or not might be a quicker first step? :)
Good discussion. I think this was the 2nd time we've ever encouraged anyone to create a package for. It could have been clearer though:
In any case, we will sparingly suggest people who are passionate about a feature that we're not sure about to create a package.
Additionally, we're working on publishing our process for dealing with new incoming issues and PRs so that it doesn't feel as random.
One of the more important parts in that process is that we really would like to get away from stockpiling issues. We'll attempt to classify everything coming in, instead of just letting issues linger forever.
More news to come when the Danish holidays are over and everybody is back in full force again!
Excellent reply Seb. Sorry to have forced you hand on this one but was a good one to talk out and I look forward as ever to some more structure.
Love and hugs xx
Great discussion. As someone who's a bit nervous about approaching package building, I suggested a "Package Building Course" at CG18. I wonder if many would be interested and if it's a flyer? I'm thinking it might help standardise the approach, help with the logistics, and relieve anxiety.
Just a thought :)
For me personally the key point from Seb's reply is "no evidence for large demand of the proposed feature"...
I love Umbraco and the fact that at it's core it is a focussed well built app. You then build what you want, either by installing packages, or via your own cleverness.
It would be disappointing to see Umbraco become like Mr Creosote in Monty Python... https://www.youtube.com/watch?v=rXH_12QWWg8 :-)
Cheers
Nigel
In response to the initial question...
Yes, it is a hindrance. Having been involved with porting a few packages (or features from packages) over to the Umbraco core, I can say that the effort involved is quite significant. This isn't a fault of Umbraco by any means, I'm only highlighting that porting requires additional time & effort, (on both sides).
It involves many discussions that unpick the decisions you made for your own package and how they fit architecturally inside Umbraco core.
(To note, I wouldn't want anyone to think the effort hasn't been worthwhile - the benefits outweigh the effort!)
With my own packages, I've never started to develop a package with the intend of it becoming a feature of Umbraco core. That thinking feels backwards to me. If I felt a feature was right (and justified) for Umbraco core, I would try to put forward my case (via the issue tracker, or sometimes lesser formal channels - note: I appreciate that I'm in a somewhat privileged position here).
If I did get a friendly "CrAP" response, I probably wouldn't develop as a package. In my mind, I'd see it as a failure of my "sales pitch" (so to speak). Like Seb says, there'd be either too many unknowns or it'd be a big chunk of work. To which I'd ask myself can I clarify this feature? / could the feature be broken down incrementally?
It's been good to read everyone's thoughts on this discussion. I do see this more as a "how to sell your feature idea" discussion, rather than highlight the woes of current package development.
Maybe a bit late to the party, but my 2c...
I was part of the open space session at CG18 that talked about all things package-related, and my takeaway from that was there are plenty of punters interested in writing packages, but unsure how to start. No doubt they have the technical nous to do so, it's just a bit of a black box - not because it's overly complicated, just not perhaps as clearly a defined process as it could be.
I enjoy building packages. I can have an idea and turn it into a thing, so long as it fits within Umbraco's constraints (haven't had any issues as yet).
I've worked out a workflow that works for me, around tooling and structure and so forth, but highly likely what I'm doing is completely different from the next package dev. Neither is probably more correct, but it wouldn't hurt to establish a best practice model.
In the open space chat, we touched on the idea of a boilerplate for packages. It's probably unrealistic to standardise completely, which is unfortunate, but some clearer definition around what to do, what not to do, would be really useful. I've thought about creating a git repo with a package framework that fits my approach. A package package.
As for the core vs package question, I think going package-first is an opportunity to prove value. If a proposed feature, delivered as a package, sees great uptake, then sure, maybe it belongs in the core.
But looking at the ecosystems around other CMS platforms, plugins/extensions/packages are an industry unto themselves. People make a living off one or two well-executed ideas. Having a downstream ecosystem is evidence of a mature product.
Build a marketplace for packages and the desire to move into the core would diminish if package development is a financially viable option. That might be difficult given the perception that open source means free, but difficult problems are the rewarding ones to solve.
I think the ideas are important, it's sad when they are lost or waylaid.
I think it's incredibly useful to know up front whether HQ think an idea would be good to develop for the core (or if they are in the process of building something similar) or not, or if it is more niche and would suit a package, you approach the two end results in different ways, so knowing upfront can really save time, and be motivating.
With introversion rife within our industry though, it's sometimes difficult for a person to suggest an idea, without implicitly putting pressure on themselves to build it, either as a PR or a package.
If you are working on something for the core, then you receive feedback and can reach out for help on your PR, if you are building a package then there isn't necessarily anywhere specific to turn to for help, which I guess is why it might feel like a negative thing... when it's not meant to be.
Think we lack a friendly place to discuss, and incubate ideas, or maybe not a focussed place - 'eg there are posts on the forum, slack, the new gitter channel I think will help with this, but be great to have a specific place for ideas, (not necessarily issues, ideas can be half baked... they need to be aired and discussed, and often they are, particularly at meetups and festivals).
I think there is also an obsession in the industry for wanting to be 'the person' that built 'the thing', that doesn't always encourage discussion and collaboration.
It's hard for anyone to make the decision on package vs core, without fully understanding the idea, it's hard to illustrate the idea without building something, have found videos of rough functionality really useful to get across quite what you mean, building a prototype and blogging, but then time passes, and the ideas fade, (and what if someone builds the thing I have prototyped, crikey, I won't be 'the person'!) - Had thought before about a semi regular, webinar with core dev team where ideas are pitched/evaluated, and maybe put into room 101, increase visibility of ideas, and how HQ weigh them up (having to maintain the functionality vs benefits of functionality being there in the core), I don't know how this would work, maybe the gitter channel is enough.
A lot of this is all really a symptom of the scaling of Umbraco, and the shortfall in discussing and responding to issues and pull requests, which everyone is painfully aware of, and experiments are in place to start to address this, which will take time to unfold...
But additionally another thing to consider, switching back to the example of the ability to rotate images in the backoffice (in Pete's opening gambit) an issue was created in September 2016 and is marked as up for grabs, I developed the uSpinMeRound package largely to make the joke about the title, unaware of the issue, then discovered it http://issues.umbraco.org/issue/U4-9028 and now that issue has a comment (May 2017):
see https://github.com/marcemarc/uSpinMeRightRound would happily work on a pull request if this is considered on the right track...
Now I know I'd need to rework this to put this into the core, and I'd be happy to do this, but if you see my introversion around how I've phrased my comment... I'd do this if you thought it was right... having no reply on this issue makes me conclude 'yeah, we don't like what you've done'.
When in reality I sort of know 'yeah, sorry, we haven't seen this, is there an issue on the tracker about it?'.
So eventually Seb will wade through and discover this, and we'll pick up on it again later, and we just have to accept the shortfall was there, and it will take time to recover -
But I use the example to also raise the point from a community perspective the issue only has 4 votes... and the package only has 4 hearts.. so it's hard to consider this as 'a belter that should be in the core' - Why haven't you voted for it Pete?, why has't anyone voted for it? (It's shit Marc, please let it go) - or is this the bit that is also a little bit broken?,
mechanisms are there for people to provide feedback on issues and packages - but we've somehow lost the culture around doing so, which ties into Shannon's observations on sorting out and then re-promoting the package ecosystem, which I won't repeat.
Sorry if you read this far, I got over excited searching for an old reply with my name in and this was the first post that came up.
Up voting stuff. I've got an issue with that. We were warned off it during the death of V5..."You all up vote packages when you've not even downloaded them" and we've also been trained that up voting an issue isn't a sign of something needing to be done:
https://twitter.com/thechiefunicorn/status/939507330781274112
So yes, that part of it is broken currently. I'd happily have some guidance on if or how we should get faith back in up voting stuff.
is working on a reply...