Note: This is discussion is about the build process used within the Umbraco Core solution and not what you might use for your website :)
I have been looking at the core frontend build process that currently uses a combination of NPM, Bower, Gulp and a whole host of frontend tools, a lot of which are now very outdated and some of which show security vulnerabilities. (These are not used on the front end, so shouldn't really be an issue, but it's still nice to see a build with no warnings)
If you work with frontend tools, you will probably know that Bower is near the end of it's life and they are recommending people move off the platform to use something like Yarn instead ( https://bower.io/blog/2017/how-to-migrate-away-from-bower/ )
However, there are other new tools out there and I was interest to know what those in the community and HQ are currently using and if there are any preferences to how this is updated?
I am happy to do the update and make the pull request, but I didn't want to do the work only to find that HQ or others disagreed with the new tools :)
I look forward to hearing everyone's opinions and hopefully coming to a useful conclusion for the way forward.
The last time I spoke to Mads about it he mentioned that NPM itself has progressed loads over the years and he is under the impression that it could do everything we need it to do out of the box. I don't know if you have any experience with this but it would be good to investigate. Less dependencies is better, right!
On the other hand, I hear a lot about WebPack recently and maybe that is a better way to go?
Disclaimer: I am completely ignorant on the topic of frontend build tools, I don't understand them, I don't know how to configure them and I only use them through automated scripts. In other words, I have only a faint idea what I'm talking about, so bear that in mind. 😉
I'll also point Mads to this topic for some context.
Thanks for raising the topic. Yes, the current front-end build is a bit outdated, and it would be wonderful to see how we can improve it.
The most obvious thing you point out is to move away from Bower and find another package manager. Bower recommends Yarn, but after NPM 5 is out I am not sure about the benefits of Yarn anymore? NPM seems to have implemented most of the same features as Yarn so a dependency less would be excellent. There might be some fancy stuff in Yarn I don't know about?
I think moving away from Bower would be an excellent first step.
Another tool we need to consider is WebPack. Unfortunately, I am no WebPack expert but I can tell you the main ideas of the current gulp setup and then we might be able to find a good solution for moving forward.
Less
All .less files get imported into a "belle.less"-file and processed. We use auto-prefixer with postcss. We have a couple of different bundles for styling. "Umbraco", "preview", "installer" and "no-nodes" (Empty Umbraco installation screen).
Javascript
Currently, we bundle the files based on a naming convention. All the controllers are in one bundle, all services in one bundle, all directives in one bundle, etc. We also have bundles for the installer, preview, and probably some more. For javascript, we also have a bit of linting.
Views
We copy over all HTML views.
Images/Fonts
Are just copied over. There is a bit of minification on the images when the copy happens.
Libs
We have a bunch of libraries which doesn't exist on bower. They are too old. These also get copied over, and most of them are part of the initial load. It would, of course, be amazing to get these updated to a version that exists on npm/Yarn/whatever. This might be a bigger task and maybe something that should be handled as a separate task.
In the ideal world, we would ship with smaller bundles, so we only load the dependencies that are needed for a page to render and therefore speed up load times but It might make sense to start simpler to get the ball rolling.
File structure
Currently, the file structure is also a mess. Over the last couple of years more and more of the back office has been componentized and with the upgrade to Angular 1.7 we have now started using the "component" syntax sugar. I would love to see a file structure based on components/functionality instead of file type. A component's js, HTML, styling, and tests live next to each other instead of in entirely separate locations. This is also something we need to consider when setting up the new build.
v7 vs. v8 build setup
Where would these updates make sense to make? Some might be able to fit into v7, and others are breaking changes for v7 and has to be done in v8. Are there any challenges in having different build setups for v7 and v8?
That was a lot of rambling. I guarantee I have forgotten something.
I am very much open to input and all the help anybody would like to assist with so we can find the best solution. Both so we and end users can benefit from a better build setup
Thank you for the detailed reply, most if this I had already figured out, but it's great to have it all written down in one place for reference.
From what I have read, I think the benefits of NPM scripts removing the dependency on Gulp don't outweigh the advantages Gulp gives us. If you want to do complex build steps with NPM, you end up having to put the functionality into separate JS scripts, I think having the whole build process in the single Gulp file is more advantageous.
Gulp also now supports parallel building, which I intend to see if we can utilise for the Umbraco build to improve the performance.
Have you seen the new kid on the block: Parcel JS ?
In a separate test branch I have started to see if I can get ParcelJS to build the project as it seems to be significantly faster than Gulp. In my local tests on another project the build time goes from 8-15 seconds to about 300ms, when you are working on changes to the frontend code, this can really be useful.
I will update this post when I have my various test branches working to let you know my findings :)
My thought was to make all these changes in the V8 branch and leave V7 un-touched, either way, once I have things working, I can easily port it back to a V7 branch if required.
+1 to the file structure being component focused - makes it very easy to find the appropriate files (html, css, js) all in one place.
Looks like Chris has taken some considerable time to audit the fed tools. I'd be happy to join in the conversation as well, but I'm looking at the fed project for the second, maybe third time. I do wonder how quickly I can take the v8 branch and reorganize at least the file structure.
BTW - Any issue with mixing support for LESS / SASS syntax in the project?
Any chance we could establish a "working group" for FED? A simple channel on slack or whatever you use at HQ?
Yes, I have seen Parcel but never tried it. It does look promising but my problem is that it is very much "The new kid on the block". WebPack has become the standard for almost anything so the future for Parcel is hard to tell at the moment. Will they win over WebPack or will WebPack improve to match Parcel?
If we look at npm downloads WebPack has more than 2.8 million weekly downloads compared to Parcel's 2500.
The defacto for us is to simply use npm and we write npm scripts for complicated build processes while a webpack config takes care of compiling entry points for JS/CSS.
I'm glad somebody else is recognizing the benefits of overhauling the build process for front end, but I'm afraid the conversation will have twice as many possible solutions per person involved. We all have different backgrounds, experience, and comfort levels with tools.
I'd stress that re-work of the FED and JS loading strategy would make the UI much more responsive. I know how to do this with webpack and using entry points to separate what's critical to load synchronously (js and css) and what can be asynchronous / deferred.
There are savings to be realized at build time yes, but also client-side.
My two cents are that the way to go is "webpack". I'm using it for several projects and I had a look at what Umbraco does, it could totally do all of that with less configuration than gulp.
Discussion about the Frontend build process used in the core project.
Hi All,
Note: This is discussion is about the build process used within the Umbraco Core solution and not what you might use for your website :)
I have been looking at the core frontend build process that currently uses a combination of NPM, Bower, Gulp and a whole host of frontend tools, a lot of which are now very outdated and some of which show security vulnerabilities. (These are not used on the front end, so shouldn't really be an issue, but it's still nice to see a build with no warnings)
If you work with frontend tools, you will probably know that Bower is near the end of it's life and they are recommending people move off the platform to use something like Yarn instead ( https://bower.io/blog/2017/how-to-migrate-away-from-bower/ )
However, there are other new tools out there and I was interest to know what those in the community and HQ are currently using and if there are any preferences to how this is updated?
I am happy to do the update and make the pull request, but I didn't want to do the work only to find that HQ or others disagreed with the new tools :)
I look forward to hearing everyone's opinions and hopefully coming to a useful conclusion for the way forward.
The last time I spoke to Mads about it he mentioned that NPM itself has progressed loads over the years and he is under the impression that it could do everything we need it to do out of the box. I don't know if you have any experience with this but it would be good to investigate. Less dependencies is better, right!
On the other hand, I hear a lot about WebPack recently and maybe that is a better way to go?
Disclaimer: I am completely ignorant on the topic of frontend build tools, I don't understand them, I don't know how to configure them and I only use them through automated scripts. In other words, I have only a faint idea what I'm talking about, so bear that in mind. 😉
I'll also point Mads to this topic for some context.
Hi Chris,
Thanks for raising the topic. Yes, the current front-end build is a bit outdated, and it would be wonderful to see how we can improve it.
The most obvious thing you point out is to move away from Bower and find another package manager. Bower recommends Yarn, but after NPM 5 is out I am not sure about the benefits of Yarn anymore? NPM seems to have implemented most of the same features as Yarn so a dependency less would be excellent. There might be some fancy stuff in Yarn I don't know about?
I think moving away from Bower would be an excellent first step.
Another tool we need to consider is WebPack. Unfortunately, I am no WebPack expert but I can tell you the main ideas of the current gulp setup and then we might be able to find a good solution for moving forward.
Less
All .less files get imported into a "belle.less"-file and processed. We use auto-prefixer with postcss. We have a couple of different bundles for styling. "Umbraco", "preview", "installer" and "no-nodes" (Empty Umbraco installation screen).
Javascript
Currently, we bundle the files based on a naming convention. All the controllers are in one bundle, all services in one bundle, all directives in one bundle, etc. We also have bundles for the installer, preview, and probably some more. For javascript, we also have a bit of linting.
Views
We copy over all HTML views.
Images/Fonts
Are just copied over. There is a bit of minification on the images when the copy happens.
Libs
We have a bunch of libraries which doesn't exist on bower. They are too old. These also get copied over, and most of them are part of the initial load. It would, of course, be amazing to get these updated to a version that exists on npm/Yarn/whatever. This might be a bigger task and maybe something that should be handled as a separate task.
In the ideal world, we would ship with smaller bundles, so we only load the dependencies that are needed for a page to render and therefore speed up load times but It might make sense to start simpler to get the ball rolling.
File structure
Currently, the file structure is also a mess. Over the last couple of years more and more of the back office has been componentized and with the upgrade to Angular 1.7 we have now started using the "component" syntax sugar. I would love to see a file structure based on components/functionality instead of file type. A component's js, HTML, styling, and tests live next to each other instead of in entirely separate locations. This is also something we need to consider when setting up the new build.
v7 vs. v8 build setup
Where would these updates make sense to make? Some might be able to fit into v7, and others are breaking changes for v7 and has to be done in v8. Are there any challenges in having different build setups for v7 and v8?
That was a lot of rambling. I guarantee I have forgotten something.
I am very much open to input and all the help anybody would like to assist with so we can find the best solution. Both so we and end users can benefit from a better build setup
Hi Mads,
Thank you for the detailed reply, most if this I had already figured out, but it's great to have it all written down in one place for reference.
From what I have read, I think the benefits of NPM scripts removing the dependency on Gulp don't outweigh the advantages Gulp gives us. If you want to do complex build steps with NPM, you end up having to put the functionality into separate JS scripts, I think having the whole build process in the single Gulp file is more advantageous.
Gulp also now supports parallel building, which I intend to see if we can utilise for the Umbraco build to improve the performance.
Have you seen the new kid on the block: Parcel JS ?
In a separate test branch I have started to see if I can get ParcelJS to build the project as it seems to be significantly faster than Gulp. In my local tests on another project the build time goes from 8-15 seconds to about 300ms, when you are working on changes to the frontend code, this can really be useful.
I will update this post when I have my various test branches working to let you know my findings :)
My thought was to make all these changes in the V8 branch and leave V7 un-touched, either way, once I have things working, I can easily port it back to a V7 branch if required.
+1 to the file structure being component focused - makes it very easy to find the appropriate files (html, css, js) all in one place.
Looks like Chris has taken some considerable time to audit the fed tools. I'd be happy to join in the conversation as well, but I'm looking at the fed project for the second, maybe third time. I do wonder how quickly I can take the v8 branch and reorganize at least the file structure.
BTW - Any issue with mixing support for LESS / SASS syntax in the project?
Any chance we could establish a "working group" for FED? A simple channel on slack or whatever you use at HQ?
Yes, I have seen Parcel but never tried it. It does look promising but my problem is that it is very much "The new kid on the block". WebPack has become the standard for almost anything so the future for Parcel is hard to tell at the moment. Will they win over WebPack or will WebPack improve to match Parcel?
If we look at npm downloads WebPack has more than 2.8 million weekly downloads compared to Parcel's 2500.
Based on this I would pick WebPack right now.
The defacto for us is to simply use npm and we write npm scripts for complicated build processes while a webpack config takes care of compiling entry points for JS/CSS.
I'm glad somebody else is recognizing the benefits of overhauling the build process for front end, but I'm afraid the conversation will have twice as many possible solutions per person involved. We all have different backgrounds, experience, and comfort levels with tools.
I'd stress that re-work of the FED and JS loading strategy would make the UI much more responsive. I know how to do this with webpack and using entry points to separate what's critical to load synchronously (js and css) and what can be asynchronous / deferred.
There are savings to be realized at build time yes, but also client-side.
My two cents are that the way to go is "webpack". I'm using it for several projects and I had a look at what Umbraco does, it could totally do all of that with less configuration than gulp.
is working on a reply...