Hi everyone.
I'm undertaking a bit of a proof of concept project to create a migrations framework/api for umbraco. Taking inspiration from Fluent Migrator.
I'm wondering if anyone has tried this before or if there's something else that could do what I want.
We've used uSiteBuilder in the past quite heavily, and we're looking for something else... simpler? faster? more control?
I've not use uSync, it seems simpler and faster, but not more control (this is probably what we'll use if my migrations project fails, or is just too much work.)
The idea is we have some code which runs for each version (checking a version sequence number in the DB) the code could be arbitrary but usually will contain simple API calls to manipulate the doctype/datatype info in the database.
This code can be pretty verbose, so I'd like to wrap it in a bit of a fluent API.
[UmbracoMigration(1)]
public class ModifyLandingPage : UmbracoMigration
{
public override void Up()
{
migrator.ContentType("Landing Page")
.DeleteProperty("Bolox")
.DeleteProperty("oldProperty");
migrator.ContentType("Landing Page")
.Property("Alpha")
.Alias("beta")
.Name("Beta")
.Description("Mandatory")
.Mandatory();
migrator.Save();
}
}
The advantage here, is we can run any code we need for the migration, we can delete properties, we can re-name properties.
I am thinking on start any un-run migrations will run.
also each start the latest migration will ALWAYS run (you could configure that off if you like eg, for production) the API would encourage idempotent code. Idem-potency makes developing your migrations easier.
I probably won't implement the 'down' feature that FluentMigrator has simply because we probably wouldn't use that.
Morning guys, as Benjamin said we have a EF code first migration inspired nuget package for umbraco we are using for our current build. Its solid and in use in our live environment, in need of a little love for it to be feature complete and it lacks proper documentation but its not far off. As always before we release this we need to get Legal to do their stuff, but the proposal has been submitted. Fingers crossed it not too far away.
Quick example.
public override void Migrate()
{
var basePageDt = DocumentType.Get("BasePage");
var zoneHomepageTemplate = Template.Create("ZoneHomepage", "Zone Homepage");
var zoneHomepageDt = DocumentType.Create("ZoneHomepage", "Zone Homepage")
.SetParent(basePageDt)
.AddAllowedTemplate(zoneHomepageTemplate)
.SetDefaultTemplate(zoneHomepageTemplate)
.Save();
zoneHomepageDt
.CreateTab("Sidekick")
.AddProperty("SidekickImage", "Sidekick Image", DataTypes.MediaPicker, "Sidekick", false, "Hero Image")
.AddProperty("SidekickHeader", "Sidekick Header", DataTypes.TextString, "Sidekick", false, "Intro text Header")
.AddProperty("SidekickRichtext", "Sidekick Wysiwyg Content", DataTypes.RichtextEditor, "Sidekick", false, "WYSIWYG content")
.AddProperty("SidekickCallToAction", "Sidekick Call To Action", CustomDataTypeConsts.UrlPicker, "Sidekick", false)
.AddProperty("SidekickWidgets", "Sidekick Widgets", CustomDataTypeConsts.WidgetPicker, "Sidekick", false, "Widgets to display below sidekick text area")
.Save();
}
Keep an eye on @MagDevsUk as we'll no doubt announce it through that.
@Tom looks interesting, how does it support versioning & running? I'm not familiar with EF's migrations.
What operations does the fluent API support? rename, edit, delete?
I guess that those operations it doesn't support we can just fall back to the raw API :-)
@James - I don't see why not, its just a hoop we need to jump through, I imagin it will just go under one of the usual open source licences.
@Murray - The main reason we moved away from uSync/Courier (we tried both) was so we could refactor and CI our application, so we have made sure it allows us to modify everything we have had a need to so far (which is most things). It runs on app start and checks a migration history table to see what migrations it has already ran and executes any it hasn't.
Yep awesome, are your fluent operations idempotent? (can be run twice with the same outcome as running once)?
I think this is important while in development to re-run the last migration over and over so we don't end up with hundreds if micro migrations.
The most important thing to us is having 100% confidence that when migrations execute there is no way anything unexpected can happen. To ensure we are happy to let this run wild and free in our production environments it will immediately bomb out at the first sign of something unexpected and roll back any instructions in the migration that have already been executed. Once a migration has ran, that's it, it won't run again against a db that has it already executed.
This is not ideal for dev, but early on we opted for robustness over ease of development, that being said other than a few steps we take to ensure our migration is sound there is nothing overly tedious about it. I build up my migration step by step verifying each one has done what I expected. I write some instructions, run the migration, check results, comment out previous instruction then add more. Pretty much rinse and repeat that until all the changes for the piece of work I'm doing is done.
Our current application has 91 migrations and is probably going to have about 150 before the end of the project.
MigrationRunner and MigrationBase are for database schema changes and the metadata they use implies upgrading or downgrading between differing Umbraco versions.
You would have to write your own versioning utilities to override Umbraco's versioning to handle using these classes - which would be less than ideal (i.e. I'm using Umbraco 7.1.4 on a project, I don't like the idea of having to bump version to 7.1.5, 7.1.6 etc. just to get my migrations to run, cause then I'm hacking a version number inconsistent with my runtimes).
Yeee we did look at that but it was no go, it wasn't quite what we where looking for. We looked into all the well tried and tested tools people currently use for this kind of thing but they where all dead ends. When we decided to write our own we looking into Umbracos source for any existing functionality we could leverage. We decided in the end to keep it simple and drive the public Umbraco services.
Not date yet unfortunately, we've started some basic docs and all the paperwork is submitted. Hopefully we'll be attending the upcoming Manchester Umbraco meetup, which from its title 'Why code first is bad', sounds like it could be interesting.
I have created a github repo with a ubr short README with code example https://github.com/dev-mag/uMigrate. Once released it will be available from there.
Thanks for the update! The 'readme' in the repository looks promising...
I'm assuming the answer to the "Why code first is bad" topic is, "Because Umbraco sell Courier"?!
My personal experience of Courier has ranged from disappointing to disasterous - other people's mileage may vary, of course - but I'm looking forward to having a 'proper', robust way to deploy changes out to Umbraco sites in different environments!
Honestly... Our deployment system right now is shambolic.
We've been using uSiteBuilder but it's dog slow and fraught with concurrency errors that I don't have time to fix. I hate that we have to manually edit the web.config file also.
We have a system to automatically deploy templates, css, images and config files direct from version control using Jenkins - but we don't have any automation for deployment of changes to document types, etc.
Feels like the fluent API mentioned in this topic could be the answer!
We have ours completely automated from TeamCity to AWS test/live environments. A package is created for each push/checkin and manually selected for deployment to a given environment, if the package includes any number of new migrations they get executed and jobs a gooden. Working well for us so far, Our current live site has executed 127 migrations without any faults (touch wood).
The key for us has been being able to test the migrations locally, repeatably, so we are sure there will be no issues against any other environment.
Yeeeee, we went through the pain of having to write pages and pages of release instructions for every deployment, and then replay them for every environment. Not fun at all, especially if a typo sneaks in!
Looks like somebody may have stolen the name of our framework :(. Although uMigrate would be the obvious name for such a framework it seems a little too much of a coincidence that it appeared shortly after I mentioned it here. I find it hard to believe that after years without a code first migration style deployment framework for Umbraco 2 with the same name with very similar api can come about within weeks of one another.
Legal have just given us the go ahead to release it too, typical.
uMigrage is the first name I thought when I was thinking of making such a thing. I guess it's an overdue idea.
Now you will be able to come up with a less boring name.:-)
Now I guess there's going to be 2 to evaluate :-]
I really appreciate all the work you've put in, can't wait to try it out.
haha :), our sites are built in 6 as when we started the project 7 had only just been released. Umbracos testing strategy of letting the users test it in production doesn't cut it for us so we stuck with tried and tested. We will upgrade though in future!
Well - it's a shame about the naming conflict, but if we've gone from having no solution, to having two potential solutions - that's a win as far as I'm concerned!
We haven't started using v7 yet for the same reason. Looking forward to trying uMigrate / uGetYourStuffLiveWithoutWantingToKillYourself (or whatever it ends up being called) soon!
we have a name, we considered uDeploy haha. Just doing to do some refactoring over the weekend the namespaces and such match. Keeping it a not so well guarded secret until then though in case somebody else uses it for 'inspiration'.
Slow but sure, we're just splitting some stuff out and other re factorings. Testing the changes with our application over the next week or so then we'll get it public :).
We have finally got round to refactoring it ready for release, we are updating our application to use the public version to ensure everything is still working. Once we're using the public one in production i'll let you guys know. Keep in mind its been made for our internal use so the Api isn't 'feature complete', but we've been using it for months so cant be that lacking lol.
No problem, hope it is of some use in this early form, we've done 3 deployments on this new package so its usable. Apologies for gross lack of documentation, it is on the board, just prioritized beneath project guff.
We started the project that ended up spawning uFluent before 7 was out of beta, there's nothing more to it than that really. We are going to be upgrading to 7 soonish and that will include full testing and fixing anything that's broken against 7. But for now unfortunately we have no idea what it would do running on top of 7, it might even work fine, but until we have thoroughly tested it and have it working on our production environment officially it only supports 6.x
Fluent Migrator style Migrations for Umbraco
Hi everyone.
I'm undertaking a bit of a proof of concept project to create a migrations framework/api for umbraco. Taking inspiration from Fluent Migrator.
I'm wondering if anyone has tried this before or if there's something else that could do what I want.
We've used uSiteBuilder in the past quite heavily, and we're looking for something else... simpler? faster? more control?
I've not use uSync, it seems simpler and faster, but not more control (this is probably what we'll use if my migrations project fails, or is just too much work.)
The idea is we have some code which runs for each version (checking a version sequence number in the DB) the code could be arbitrary but usually will contain simple API calls to manipulate the doctype/datatype info in the database.
This code can be pretty verbose, so I'd like to wrap it in a bit of a fluent API.
The advantage here, is we can run any code we need for the migration, we can delete properties, we can re-name properties.
I am thinking on start any un-run migrations will run. also each start the latest migration will ALWAYS run (you could configure that off if you like eg, for production) the API would encourage idempotent code. Idem-potency makes developing your migrations easier.
I probably won't implement the 'down' feature that FluentMigrator has simply because we probably wouldn't use that.
Any thoughts? :-)
Cheers.
Murray.
Count me in. I'm definitely intrigued.
Follow @MagDevsUK and @Type_TR on Twitter, as they're hoping to OSS their efforts that they've been working on over the last few months.
Morning guys, as Benjamin said we have a EF code first migration inspired nuget package for umbraco we are using for our current build. Its solid and in use in our live environment, in need of a little love for it to be feature complete and it lacks proper documentation but its not far off. As always before we release this we need to get Legal to do their stuff, but the proposal has been submitted. Fingers crossed it not too far away.
Quick example.
Keep an eye on @MagDevsUk as we'll no doubt announce it through that.
Does legal getting involved mean that it wouldn't be open source? As in properly feature complete and open source?
@Tom looks interesting, how does it support versioning & running? I'm not familiar with EF's migrations.
What operations does the fluent API support? rename, edit, delete?
I guess that those operations it doesn't support we can just fall back to the raw API :-)
@James - I don't see why not, its just a hoop we need to jump through, I imagin it will just go under one of the usual open source licences.
@Murray - The main reason we moved away from uSync/Courier (we tried both) was so we could refactor and CI our application, so we have made sure it allows us to modify everything we have had a need to so far (which is most things). It runs on app start and checks a migration history table to see what migrations it has already ran and executes any it hasn't.
Awesome! I'm really looking forward to seeing the fruits of your labour.
Yep awesome, are your fluent operations idempotent? (can be run twice with the same outcome as running once)?
I think this is important while in development to re-run the last migration over and over so we don't end up with hundreds if micro migrations.
The most important thing to us is having 100% confidence that when migrations execute there is no way anything unexpected can happen. To ensure we are happy to let this run wild and free in our production environments it will immediately bomb out at the first sign of something unexpected and roll back any instructions in the migration that have already been executed. Once a migration has ran, that's it, it won't run again against a db that has it already executed.
This is not ideal for dev, but early on we opted for robustness over ease of development, that being said other than a few steps we take to ensure our migration is sound there is nothing overly tedious about it. I build up my migration step by step verifying each one has done what I expected. I write some instructions, run the migration, check results, comment out previous instruction then add more. Pretty much rinse and repeat that until all the changes for the piece of work I'm doing is done.
Our current application has 91 migrations and is probably going to have about 150 before the end of the project.
Hi guys,
don't know if it's important for you but Umbraco ships with its own migration system where you can do a lot of stuff.
Haven't tryied to change DocumentType or something like this with it but think that could be possible.
Just search for MigrationRunner or MigrationBase in the Umbraco Source Repository on GitHub.
Maybe you can find some usefull stuff there.
Besides of that I'm interested in the outcome of this project.
David
MigrationRunner and MigrationBase are for database schema changes and the metadata they use implies upgrading or downgrading between differing Umbraco versions.
You would have to write your own versioning utilities to override Umbraco's versioning to handle using these classes - which would be less than ideal (i.e. I'm using Umbraco 7.1.4 on a project, I don't like the idea of having to bump version to 7.1.5, 7.1.6 etc. just to get my migrations to run, cause then I'm hacking a version number inconsistent with my runtimes).
Yeee we did look at that but it was no go, it wasn't quite what we where looking for. We looked into all the well tried and tested tools people currently use for this kind of thing but they where all dead ends. When we decided to write our own we looking into Umbracos source for any existing functionality we could leverage. We decided in the end to keep it simple and drive the public Umbraco services.
EDIT - ye, what Benjamin said :P
Just followed @MagDevsUK and @Type_TR on Twitter - can't wait to hear some news on this!
Something like this has been on our, "If only we had time to build..." list for ages now.
The MigrationRunner can be used to Run Migrations outsite the Umbraco versions.
I'm currently using it to make updates to custom database tables for a package I develope.
But you're right it's more for changing database scheme and not for changing other stuff.
Just wanted to state that there is something in Umbraco.
@Tom, any news on the possibility of releasing what you guys have built, how are the lawyers ;-)
Not date yet unfortunately, we've started some basic docs and all the paperwork is submitted. Hopefully we'll be attending the upcoming Manchester Umbraco meetup, which from its title 'Why code first is bad', sounds like it could be interesting.
I have created a github repo with a ubr short README with code example https://github.com/dev-mag/uMigrate. Once released it will be available from there.
Thanks for the update! The 'readme' in the repository looks promising...
I'm assuming the answer to the "Why code first is bad" topic is, "Because Umbraco sell Courier"?!
My personal experience of Courier has ranged from disappointing to disasterous - other people's mileage may vary, of course - but I'm looking forward to having a 'proper', robust way to deploy changes out to Umbraco sites in different environments!
Thanks,
- Chris
Thanks for the update. Really looking forward to seeing some code. :)
I think "Why Code First is Bad" is a misnomer. It really should be "Why are Current Code First Implementations Bad".
Ahh - yeah - that'd be fair enough.
Always fun to try and stir up a bit of trouble, though! ;o)
As a matter of interest - what do you currently use for your deployments?
Honestly... Our deployment system right now is shambolic.
We've been using uSiteBuilder but it's dog slow and fraught with concurrency errors that I don't have time to fix. I hate that we have to manually edit the web.config file also.
Our process isn't too much better.
We have a system to automatically deploy templates, css, images and config files direct from version control using Jenkins - but we don't have any automation for deployment of changes to document types, etc.
Feels like the fluent API mentioned in this topic could be the answer!
We have ours completely automated from TeamCity to AWS test/live environments. A package is created for each push/checkin and manually selected for deployment to a given environment, if the package includes any number of new migrations they get executed and jobs a gooden. Working well for us so far, Our current live site has executed 127 migrations without any faults (touch wood).
The key for us has been being able to test the migrations locally, repeatably, so we are sure there will be no issues against any other environment.
Sounds a lot less painful than what we're doing!
Yeeeee, we went through the pain of having to write pages and pages of release instructions for every deployment, and then replay them for every environment. Not fun at all, especially if a typo sneaks in!
Looks like somebody may have stolen the name of our framework :(. Although uMigrate would be the obvious name for such a framework it seems a little too much of a coincidence that it appeared shortly after I mentioned it here. I find it hard to believe that after years without a code first migration style deployment framework for Umbraco 2 with the same name with very similar api can come about within weeks of one another.
Legal have just given us the go ahead to release it too, typical.
So this is someone else's eh. http://our.umbraco.org/projects/collaboration/umigrate
uMigrage is the first name I thought when I was thinking of making such a thing. I guess it's an overdue idea.
Now you will be able to come up with a less boring name.:-)
Now I guess there's going to be 2 to evaluate :-]
I really appreciate all the work you've put in, can't wait to try it out.
Cheers.
Murray.
If yours works in v6 also you can have my first born.
haha :), our sites are built in 6 as when we started the project 7 had only just been released. Umbracos testing strategy of letting the users test it in production doesn't cut it for us so we stuck with tried and tested. We will upgrade though in future!
TL;DR.... it works in 6.
Well - it's a shame about the naming conflict, but if we've gone from having no solution, to having two potential solutions - that's a win as far as I'm concerned!
We haven't started using v7 yet for the same reason. Looking forward to trying uMigrate / uGetYourStuffLiveWithoutWantingToKillYourself (or whatever it ends up being called) soon!
Thanks,
- Chris
uRollout
uDeploy
uNoGUI
So, Tom,
When do you think we'll be able to see your v6 friendly version?
Thanks,
- Chris
we have a name, we considered uDeploy haha. Just doing to do some refactoring over the weekend the namespaces and such match. Keeping it a not so well guarded secret until then though in case somebody else uses it for 'inspiration'.
goose. (it migrates)
@Tom, any news to share? :-)
Slow but sure, we're just splitting some stuff out and other re factorings. Testing the changes with our application over the next week or so then we'll get it public :).
Thanks Tom,
Keep up the good work - let me know if you need a beta tester! (not that I'm keen, or anything...)
- Chris
We have finally got round to refactoring it ready for release, we are updating our application to use the public version to ensure everything is still working. Once we're using the public one in production i'll let you guys know. Keep in mind its been made for our internal use so the Api isn't 'feature complete', but we've been using it for months so cant be that lacking lol.
Awesome - looking forward to seeing it!
Will you be accepting community contributions? If so then I guess anything it doesn't do we can add!
Thanks,
- Chris
Its open source on GitHub so I don't see why not :)
Morning guys,
I'll start a new thread in a week or 2 but for now I'll bury these links here to limit visibility for now.
Source : https://github.com/dev-mag/uFluent
Api : https://www.nuget.org/packages/uFluent/ Runner : https://www.nuget.org/packages/uFluent.Migrate/
Any and all feedback is welcomed!
Documentation is lacking atm, that's on the todoasap list!
2 important details for uFluent.Migrate are...
Docs are being worked on, this is very much an early glance for you guys.
Hi Tom,
Thanks for posting this - sorry I didn't jump on it sooner but I must've missed the e-mail alert from this site!
We'll take a look at this and let you know how we get on!
Thanks,
- Chris
No problem, hope it is of some use in this early form, we've done 3 deployments on this new package so its usable. Apologies for gross lack of documentation, it is on the board, just prioritized beneath project guff.
No worries - you've done loads already by the look of it!
If I get anywhere with it, I'll try and put together some blog posts to help people get started!
Evening guys, there's some basic documentation on github if you want to take a look. https://github.com/dev-mag/uFluent
Feedback and/or questions welcome.
Awesome, a section for 'Install' which includes the nuget links would be handy. :-)
I also note that nuget says:
Dependencies
UmbracoCms.Core (≥ 6.1.6 && < 7.0.0)
What's the story with v7 support?
Cheers.
Murray.
We started the project that ended up spawning uFluent before 7 was out of beta, there's nothing more to it than that really. We are going to be upgrading to 7 soonish and that will include full testing and fixing anything that's broken against 7. But for now unfortunately we have no idea what it would do running on top of 7, it might even work fine, but until we have thoroughly tested it and have it working on our production environment officially it only supports 6.x
Oh and added Nuget Links to readme
is working on a reply...