So I wanted to deploy the new extension I'm working on to a running compiled instance. I already copy my DLLs and files over in my project's post build, but It needs some custom sections so I had to create the trees. No problem with *how* to do it as such, there's obviously no UI to do this so I rolled up an installer class, but still I needed a way of running my installer code.
I was thinking I could build a quick user control and drop the code in there, then create a page and hit it on the web [1]... bit awkward, but it'll have to do.
That's when it struck me. I don't know Python at all, but I only had to make a single call to MyFeature.Installer.Install() - It couldn't be too hard to work out, and then I could do the above [1] without having to drop back to VS.
So I looked up a little python, saw stuff about def-ing procedures, figured I don't need that, it;s a one time thing, and I wrote this (sort of):
import clr clr.AddReference("MyCoolExtension") from MyCoolExtension import Installer
Installer.Instance.Perform()
As I say, I don't know python, so those of you that do can probably see where I went wrong. Still, imagine my surprise when, the OnSave test only goes and runs the bally thing!
It strikes me that a) saving this particular script was a bit pointless, but b) I can run it again, simply by saving it again! and c) Actually I could have a whole host of maintenance scripts in here and just re-save them to run them again.
I don't know about saving, might it be worth swapping the icons about on this and having a "Run" action ? because as far asI can see, this is just another phenomenally useful feature, albeit discovered by accident, andpossibly a bug.
I don't care if it's a bug. Active live scripting! Can we keep it? And can we have something similar for C#?
Would anyone like me to do this? Admins only of course
The reason this happens is that by default scripts (DLR and XSLT) are evaluated on save. This is used to try and detect runtime errors before you actually run the script.
A side effect of this is that if you're writing scripts in the manner you are then it will be executed on save.
I wouldn't necessarily advise this practice, creating run-once scripts but having them in an environment where you can't ensure they are run-once is a risky process.
I was surprised actually. I know this happens with the XSLT editor, but while I don't know Python as I say, I'd have thought the compile step would be enough to flag errors? That said I did have a look at the source from trunk to check it out, and didn't spot the execute - short of stepping through it looks like the PythonEngine class just fires off CompileFile on IronPython via reflection, so presumably the execute's buried in there?
I hear you on the danger of exposing such things, of course such a thing should really be an admin only privilege, but it's nonetheless extreemely useful *for* an admin. It seems only a short step away from Powershell maintenance. The benefit obviously being that the above approach is remote and still in-process.
I have performed numerous significant deployments against live umbraco instances, adding sections,running database work, iterating and re-organising documents in batch, all sorts. My life would have been made a *lot* easier in these cases with some kind of programmable shell, an enumerable over a document search for instance or direct access to the v4 Application and ApplicationTree APIs - well, this approach is close as I'm able to execute arbitrary code against a running instance, once tested in dev of course, but having more flexibility, like the Immediate window in VS, to do a quick search or non-destructive test first for instance, I'd say that was phenomenally useful.
Given the way this works, be that by accident or not, it's only a couple of features short from what I'd like - it needs to display Out and Error streams. Maybe it's too much to expose this in the admin app, and it should be an RDP job, but I think that's too restrictive, and not all hosting would permit such things.
I looked at this re C# scripting too and was wondering what sort of CAS policy would be sensible? Though to be fair I'm not even sure a medium trust environment would let me compile a C# class on the fly.
IronPython (and IronRuby) is a dynamic language, no compilation happens, the save + error checking does an execution of the script, nothing more.
If you want a coding shell you'd be much better ff with a dashboard component IMO. You can then have better control over the hosting environment of your script than using the script saving.
Running a CLR language (C#, F# or VB.NET) is entirely different since these are compiled language. There isn't a hosting environment like there is with a dynamic language, they require the compiler.There's two ways you can go about doing this for a compiled language. Way one what you need to do is save out the code file and then pass it through to csc.exe (and the equivalent others) and then you need to compile out an executable to invoke (alternatively you could do an assembly which then has a public method you can invoke). Alternatively you can get an instance of the CodeDomProvider to generate the file output file from and attempt to do the same thing. Having working with CodeDom though is it's really hard.
Until the compiler is available as a service there's limits to how you can do dynamic compiled languages (Anders has hinted that they are working on doing it though).
I beg to differ, the CSharpCodeProvider isn't that hard to use at all, calling CompileAssemblyFromSource with GenerateInMemory = true basically gives you an in-memory assembly you can invoke dynamically as you see fit without needing the intermediate FS and Process.Start("csc") steps. I'm not sure about the performance of this approach in a real-time environment though so it's worth testing, but from what I've picked up from a quick look at the umbraco PythonEngine, running over the directory and generating classes keyed off against the filename on startup in a similar way is eminently doable.
The challenge really is providing such a feature in a way that's easy and intuitive to use, and of course useful - I've tried a couple of things out here. There seem to be a handful of approaches
wrap a namespace declaration and add and additional property of "startup object" or "startup method" so that you could invoke a script like a console app via API - most power, but too much to go wrong
wrap a namespace and class declaration so that nested structs/classes could be declared - though this also needs a picker for the "startup method"
wrap a namespace, class and Execute() method signature (off an interface) so that code written could be executed consistently. This approach opens up the possibility of a number of useful Execute() signatures, passing various parameters in dependent on the purpose, say the current Node or Document.
As to whether such a thing would be genuinely useful? Well, that's a whole other question.
Every time you dynamically create an assembly (or even load one) you add it to the app domain. If you start trying to treat Umbraco as a .NET CLR hosting environment you can easily blow out the memory of the app domain (unless you spin up an app domain just for running each script, but there's different overhead concerns in there.
I think it's a bad idea for the same reason I think the SQL XSLT extension method is a bad idea. Umbraco isn't a coding environment. Give someone the tools to shoot themselves in the foot and they'll blow off their whole leg. Code should be source controlled, if it's not source controlled there's no tracking of what was done, and no way to blame someone when shit goes bad.
Creating a code editor/runner application for running DLR scripts in Umbraco 'could' be useful so long as it was limited to people who really know what they are doing and it was used more or less for maintenance tasks, etc.. . Having DLR scripts run on the 'save' command to execute your script is definitely not ideal, it wouldn't take much to write a DLR script runner application if you really wanted one. As Aaron mentioned c# is not a dynamic language and each time you compile it to be run in your app pool it will use more and more memory which you can't get rid of which is why you would have to do this in a temporary application pool and then when processing is complete you would need to delete that application pool.
If you really want to run c# scripts against Umbraco, you can easily just use LinqPad... you don't have to worry about memory consumption in your web aplication, you can import any DLLs you want in to it, etc.. IMO, this is a much better way to script against Umbraco than having any script editor/runner built in to the web ui, even if it is DLR.
I don't know Python, and having been a C# programmer since day 1, and a java programmer before that I'm not too bothered about learning it. I think it's certainly realistic to have a number of maintenance scripts in place that can be run "on save" for common tasks, and if I were to do that I'd write those tasks in c# and fire them off through the python editor as above. I imagine with some work a useful maintenance API for the environment in question could be put together either solely in Pythong or as a combination of python/c#. However, given what has been said about the risks of exposing the metal as it were, I'd be inclined to migrate the scripting tree to a new section so it can be enabled wore exclusively - or perhaps the permissions code could be extended to support access rights for trees within sections ? (unless this already exists? I know the code has changed a lot since 3.0.3).
That all said, if we're talking about static, standard tasks for an environment that don't require changing, then an a custom section or dashboard would probably be better.
Doing this from LinqPad is interesting. I'm new to LinqPad, but similarly to powershell, wouldn't that still require some session to log in to and a more comprehensive remote API ? The reason I got here and started this topic in the first place was that I was trying to create a custom Application and ApplicationTree and there was no way to do this in the UI, I had to be in-process, or at least in a full environment with db access to do it. So I wrote an installer class to ensure the objects and ran it from the python window, which worked perfectly. I'm not sure there's an easier way to do this save for manual TSQL, hence this post.
Its def more than possible and quite easy to create your own application + trees, and permissions for applications are already handled by Umbraco. I would still not recommend running a c# compiler on your web platform, especially on a production environment.
Having a DLR editor would be pretty easy to make for Umbraco UI if you wanted. LinqPad is pretty easy to integrate with Umbraco, you can just import the Umbraco DLLs and you can do whatever you want with the API in c#.
Using LinqPad like that on the server is definitely something I'll look in to - I have used SnippetCompiler similarly in the past. Not quite the same, but effective. The difference with exposing this sort of thing in UI though is that it can be done remotely without an RDP session, and that's appealing. Obviously I know with power comes great responsibility, but I'm quite sure it's possible to bring a server down with XSLT too (or a number of other ways), should you have admin access, so I'm not sure I buy the risk argument.
Incidentally, I have in the past, with 3.0.3 run console exes as part of large install/upgrade procedures, by doing precisely this, referencing the dlls and coding away. I found I had to re-code a number of things in the 3.0.3 source to get this to work as there were a number of dependencies on HttpContext.Current that obviously wouldn't work out of process. I couldn't say if those issues still exist so this may be a moot point, but running in process still has advantages.
Ultimately I'm a developer, and yeah it's a little selfish, because I know C# very well but not python, but for me if I can log on to a customer's installation and fix their problems inside SLA *response* time without having to build/test/deploy etc, I have a happy customer. I know that's not necessarily a junior's job, and I am of course fully aware of the need for source control and change management, but the two things are not the same and can be addressed independently.
Considering this discussion and points raised, it would seem fair to keep a log of scripts run against the server for change audits - and that's not something one could do with an out of process client. But an in process scripting shell? totally doable, and come v5, that sure beats trawling though NHibernate log4net logs trying to work out what happened.
The risk factor is not because people can do malicious/accidental things, it's because compiling c# on the fly on your production server will inadvertently use up a lot more memory for your web app than you want. The only way to avoid this is to run up a new app domain on the fly, run your c# code in there and then remove the app domain. Doing this process in itself will also use up additional resources on your production server... this is why I'm saying there is a risk. Running a DLR like python on the fly does not run this risk.
I suspect that can be mitigated. The problem is going to be if resource usage is left unchecked, not that resources are being used at all, I mean off the shelf webservers these days come with gigabytes of memory, but I digress. If we know the usage profile it ought to be possible to set thresholds. I looked a little at the means of doing this, compiling on the fly, and keeping a running process going for it makes sense - that is, that with each expression evaluated, or script compiled we add our object code to a static container or something similar. This would mean we could dump a log out of band, or on the DomainUnload event as a last chance, assuming that's reliable. At the very least it should be possible to manage resource usage.
My initial thoughts would be that we'd build stored scripts into a single assembly on startup or first access, and that ought to be relatively safe to leave be, but should any dynamic evaluation be offered on top, then this should have a lifetime, much like asp.net app pools have, as spinning up and destroying an appdomain per evaluation is too expensive. Per user session? That could work.
This has of course now had me looking at the internals of the DLR and Microsoft.Scripting.Ast - interesting stuff... but sadly a mile away from anything I've got time to get stuck into now! So yeah it might be untidy compiling on the fly, but it's certainly doable is my point, if not necessarily as easily as using existing DLR languages.
Blimmin computers - I always get lost in academia when I should be solving business problems... grrrr.
Python as a live scripting shell
So I wanted to deploy the new extension I'm working on to a running compiled instance. I already copy my DLLs and files over in my project's post build, but It needs some custom sections so I had to create the trees. No problem with *how* to do it as such, there's obviously no UI to do this so I rolled up an installer class, but still I needed a way of running my installer code.
I was thinking I could build a quick user control and drop the code in there, then create a page and hit it on the web [1]... bit awkward, but it'll have to do.
That's when it struck me. I don't know Python at all, but I only had to make a single call to MyFeature.Installer.Install() - It couldn't be too hard to work out, and then I could do the above [1] without having to drop back to VS.
So I looked up a little python, saw stuff about def-ing procedures, figured I don't need that, it;s a one time thing, and I wrote this (sort of):
import clr
clr.AddReference("MyCoolExtension")
from MyCoolExtension import Installer
Installer.Instance.Perform()
As I say, I don't know python, so those of you that do can probably see where I went wrong. Still, imagine my surprise when, the OnSave test only goes and runs the bally thing!
It strikes me that a) saving this particular script was a bit pointless, but b) I can run it again, simply by saving it again! and c) Actually I could have a whole host of maintenance scripts in here and just re-save them to run them again.
I don't know about saving, might it be worth swapping the icons about on this and having a "Run" action ? because as far asI can see, this is just another phenomenally useful feature, albeit discovered by accident, andpossibly a bug.
I don't care if it's a bug. Active live scripting! Can we keep it? And can we have something similar for C#?
Would anyone like me to do this? Admins only of course
The reason this happens is that by default scripts (DLR and XSLT) are evaluated on save. This is used to try and detect runtime errors before you actually run the script.
A side effect of this is that if you're writing scripts in the manner you are then it will be executed on save.
I wouldn't necessarily advise this practice, creating run-once scripts but having them in an environment where you can't ensure they are run-once is a risky process.
I was surprised actually. I know this happens with the XSLT editor, but while I don't know Python as I say, I'd have thought the compile step would be enough to flag errors? That said I did have a look at the source from trunk to check it out, and didn't spot the execute - short of stepping through it looks like the PythonEngine class just fires off CompileFile on IronPython via reflection, so presumably the execute's buried in there?
I hear you on the danger of exposing such things, of course such a thing should really be an admin only privilege, but it's nonetheless extreemely useful *for* an admin. It seems only a short step away from Powershell maintenance. The benefit obviously being that the above approach is remote and still in-process.
I have performed numerous significant deployments against live umbraco instances, adding sections,running database work, iterating and re-organising documents in batch, all sorts. My life would have been made a *lot* easier in these cases with some kind of programmable shell, an enumerable over a document search for instance or direct access to the v4 Application and ApplicationTree APIs - well, this approach is close as I'm able to execute arbitrary code against a running instance, once tested in dev of course, but having more flexibility, like the Immediate window in VS, to do a quick search or non-destructive test first for instance, I'd say that was phenomenally useful.
Given the way this works, be that by accident or not, it's only a couple of features short from what I'd like - it needs to display Out and Error streams. Maybe it's too much to expose this in the admin app, and it should be an RDP job, but I think that's too restrictive, and not all hosting would permit such things.
I looked at this re C# scripting too and was wondering what sort of CAS policy would be sensible? Though to be fair I'm not even sure a medium trust environment would let me compile a C# class on the fly.
All very interesting
IronPython (and IronRuby) is a dynamic language, no compilation happens, the save + error checking does an execution of the script, nothing more.
If you want a coding shell you'd be much better ff with a dashboard component IMO. You can then have better control over the hosting environment of your script than using the script saving.
Running a CLR language (C#, F# or VB.NET) is entirely different since these are compiled language. There isn't a hosting environment like there is with a dynamic language, they require the compiler.There's two ways you can go about doing this for a compiled language. Way one what you need to do is save out the code file and then pass it through to csc.exe (and the equivalent others) and then you need to compile out an executable to invoke (alternatively you could do an assembly which then has a public method you can invoke).
Alternatively you can get an instance of the CodeDomProvider to generate the file output file from and attempt to do the same thing. Having working with CodeDom though is it's really hard.
Until the compiler is available as a service there's limits to how you can do dynamic compiled languages (Anders has hinted that they are working on doing it though).
I beg to differ, the CSharpCodeProvider isn't that hard to use at all, calling CompileAssemblyFromSource with GenerateInMemory = true basically gives you an in-memory assembly you can invoke dynamically as you see fit without needing the intermediate FS and Process.Start("csc") steps. I'm not sure about the performance of this approach in a real-time environment though so it's worth testing, but from what I've picked up from a quick look at the umbraco PythonEngine, running over the directory and generating classes keyed off against the filename on startup in a similar way is eminently doable.
The challenge really is providing such a feature in a way that's easy and intuitive to use, and of course useful - I've tried a couple of things out here. There seem to be a handful of approaches
As to whether such a thing would be genuinely useful? Well, that's a whole other question.
Every time you dynamically create an assembly (or even load one) you add it to the app domain. If you start trying to treat Umbraco as a .NET CLR hosting environment you can easily blow out the memory of the app domain (unless you spin up an app domain just for running each script, but there's different overhead concerns in there.
I think it's a bad idea for the same reason I think the SQL XSLT extension method is a bad idea. Umbraco isn't a coding environment. Give someone the tools to shoot themselves in the foot and they'll blow off their whole leg.
Code should be source controlled, if it's not source controlled there's no tracking of what was done, and no way to blame someone when shit goes bad.
Creating a code editor/runner application for running DLR scripts in Umbraco 'could' be useful so long as it was limited to people who really know what they are doing and it was used more or less for maintenance tasks, etc.. . Having DLR scripts run on the 'save' command to execute your script is definitely not ideal, it wouldn't take much to write a DLR script runner application if you really wanted one. As Aaron mentioned c# is not a dynamic language and each time you compile it to be run in your app pool it will use more and more memory which you can't get rid of which is why you would have to do this in a temporary application pool and then when processing is complete you would need to delete that application pool.
If you really want to run c# scripts against Umbraco, you can easily just use LinqPad... you don't have to worry about memory consumption in your web aplication, you can import any DLLs you want in to it, etc.. IMO, this is a much better way to script against Umbraco than having any script editor/runner built in to the web ui, even if it is DLR.
I made some experiments using a simple textarea fire inserted DLR-code.
Here's a small cast http://screenr.com/fHn showing it getting info from nodes + creating new documents.
Fun, but most for experimenting / learning with Python I must admit. Could very well be extended with a toolset of useful scripts.
Regards
Jonas
I don't know Python, and having been a C# programmer since day 1, and a java programmer before that I'm not too bothered about learning it. I think it's certainly realistic to have a number of maintenance scripts in place that can be run "on save" for common tasks, and if I were to do that I'd write those tasks in c# and fire them off through the python editor as above. I imagine with some work a useful maintenance API for the environment in question could be put together either solely in Pythong or as a combination of python/c#. However, given what has been said about the risks of exposing the metal as it were, I'd be inclined to migrate the scripting tree to a new section so it can be enabled wore exclusively - or perhaps the permissions code could be extended to support access rights for trees within sections ? (unless this already exists? I know the code has changed a lot since 3.0.3).
That all said, if we're talking about static, standard tasks for an environment that don't require changing, then an a custom section or dashboard would probably be better.
Doing this from LinqPad is interesting. I'm new to LinqPad, but similarly to powershell, wouldn't that still require some session to log in to and a more comprehensive remote API ? The reason I got here and started this topic in the first place was that I was trying to create a custom Application and ApplicationTree and there was no way to do this in the UI, I had to be in-process, or at least in a full environment with db access to do it. So I wrote an installer class to ensure the objects and ran it from the python window, which worked perfectly. I'm not sure there's an easier way to do this save for manual TSQL, hence this post.
Its def more than possible and quite easy to create your own application + trees, and permissions for applications are already handled by Umbraco. I would still not recommend running a c# compiler on your web platform, especially on a production environment.
Having a DLR editor would be pretty easy to make for Umbraco UI if you wanted. LinqPad is pretty easy to integrate with Umbraco, you can just import the Umbraco DLLs and you can do whatever you want with the API in c#.
Using LinqPad like that on the server is definitely something I'll look in to - I have used SnippetCompiler similarly in the past. Not quite the same, but effective. The difference with exposing this sort of thing in UI though is that it can be done remotely without an RDP session, and that's appealing. Obviously I know with power comes great responsibility, but I'm quite sure it's possible to bring a server down with XSLT too (or a number of other ways), should you have admin access, so I'm not sure I buy the risk argument.
Incidentally, I have in the past, with 3.0.3 run console exes as part of large install/upgrade procedures, by doing precisely this, referencing the dlls and coding away. I found I had to re-code a number of things in the 3.0.3 source to get this to work as there were a number of dependencies on HttpContext.Current that obviously wouldn't work out of process. I couldn't say if those issues still exist so this may be a moot point, but running in process still has advantages.
Ultimately I'm a developer, and yeah it's a little selfish, because I know C# very well but not python, but for me if I can log on to a customer's installation and fix their problems inside SLA *response* time without having to build/test/deploy etc, I have a happy customer. I know that's not necessarily a junior's job, and I am of course fully aware of the need for source control and change management, but the two things are not the same and can be addressed independently.
Considering this discussion and points raised, it would seem fair to keep a log of scripts run against the server for change audits - and that's not something one could do with an out of process client. But an in process scripting shell? totally doable, and come v5, that sure beats trawling though NHibernate log4net logs trying to work out what happened.
The risk factor is not because people can do malicious/accidental things, it's because compiling c# on the fly on your production server will inadvertently use up a lot more memory for your web app than you want. The only way to avoid this is to run up a new app domain on the fly, run your c# code in there and then remove the app domain. Doing this process in itself will also use up additional resources on your production server... this is why I'm saying there is a risk. Running a DLR like python on the fly does not run this risk.
What's life without challenges ;)
I suspect that can be mitigated. The problem is going to be if resource usage is left unchecked, not that resources are being used at all, I mean off the shelf webservers these days come with gigabytes of memory, but I digress. If we know the usage profile it ought to be possible to set thresholds. I looked a little at the means of doing this, compiling on the fly, and keeping a running process going for it makes sense - that is, that with each expression evaluated, or script compiled we add our object code to a static container or something similar. This would mean we could dump a log out of band, or on the DomainUnload event as a last chance, assuming that's reliable. At the very least it should be possible to manage resource usage.
My initial thoughts would be that we'd build stored scripts into a single assembly on startup or first access, and that ought to be relatively safe to leave be, but should any dynamic evaluation be offered on top, then this should have a lifetime, much like asp.net app pools have, as spinning up and destroying an appdomain per evaluation is too expensive. Per user session? That could work.
This has of course now had me looking at the internals of the DLR and Microsoft.Scripting.Ast - interesting stuff... but sadly a mile away from anything I've got time to get stuck into now! So yeah it might be untidy compiling on the fly, but it's certainly doable is my point, if not necessarily as easily as using existing DLR languages.
Blimmin computers - I always get lost in academia when I should be solving business problems... grrrr.
is working on a reply...