A quick question though? Why do you go to the trouble of having the CDN source type as Blob storage?
When we've done this setup - we've just pointed the CDN at the Umbraco site, where it requests the image/asset on the original URL the first time and subsequently caches it.
In this approach, you don't have to upload your CSS, Javascript etc to CDN as the CDN fetches them from Umbraco.
With images the only modification you have to make is to prefix the original /media URL with the CDN domain and the image processor stuff still works.
I'm new to this and I am in the process of trying to implement an Azure based CDN at the moment and your approach seems far easier so wondered if you can point towards more information on that setup? Is this something that can be done for an Umbraco instance not hosted on the Azure platform?
I wish I'd seen your post before I just transferred 5GB of media to my Azure CDN test account :(
This was super simple to setup with HTTP - so thanks! Currently I can't get it to work with HTTPS due to a gateway error which I am yet to resolve. Some blog posts I have been reading seem to indicate it might be related to the fact we have SNI enabled on the server but that is a discussion for a separate thread.
Is this optional? It seems to still work without this step?
Initially I thought that this approach was going to copy the assets into my blob storage container but now it seems the storage container is not even required - am I correct? If so, even more reason for this approach to be favoured! It's such a light and easy integration.
It would be great if Jeavon could contribute to this thread with reasons to favour his approach over yours.
If you upload to blob you get some advantages if you have multiple developers/editors/installed instances I believe.
The assets are stored in one place so if you pull the site down to develop on it you don't need to get all the media too. This works with Darren's setup too of course but you end up with one Umbraco site that has the data and all the others that don't so you now have a tricky situation if anyone deletes the wrong site, by having the assets stored shared but separate from Umbraco you keep a nice separation.
If you then work locally and add some assets for a new feature they get stored locally on your local version not the live site that the CDN will ask for the assets from. If you then forget to copy them over you end up breaking things. Its an easy mistake to make and can be a tricky one to remember and fix come deployment time.
Both setups work, Darrens is easier (if you only have one developer/site on the whole) but Jeavons is more belt and braces with the complete separation of the assets from Umbraco.
Centralised media is a bonus, especially as this site, in particular, has over 5Gb of media currently. Say all of your environments are using the shared blob storage, how do you manage the possibility that media will be uploaded from a test/staging environment that is not intended to be used in production? Or, is this where the CDN toolkit comes in handy?
To remove any confusion, right now I have just setup the origin to point to the staging site and can get media returned via http://
I understand that once I add UmbracoFileSystemProviders.Azure into the mix that any media uploaded will then go to the blob storage - correct? But, without this step I'm curious where any media is being drawn from? Is it simply cached across Azure CDN Servers with no visible storage after the first request?
Sorry for the noob questions but this is my first time so please be gentle :)
Azure CDN will store copy of the media on it's own servers.
Upon first request - if it doens't hold a copy of the item it will get it from Umbraco.
If you don't set up Azure filesystem providers - it will get the copy to cache from the local file system where umbraco lives, otherwise it would read it from Blob storage.
Thanks for the clarification. So in the case of an existing site, I need to still push all existing media up to the blob storage container if I am to enable the Azure filesystem provider?
Oh I've already been through the pain of that piece of software - it should be READ-ONLY! It could not handle upload of the existing media, looked like it was creating multiple background processes for each item and just hanging!
What I used in the end was AZCopy and the following command and it was super quick in comparison:
Thanks for this Darren, uploading nice and quickly now. I made a small pull request to set the MIME type for files before uploading and it seems to be working well for me so far.
First things first, Darren's approach is called origin pull, my approach (or the approach of the CDN toolkit) is called origin push.
We originally planned to use origin pull as it's very simple however we hit some security issues during testing.
Therefore the reason we created the CDN Toolkit an was to enable a CDN push approach which would allow us to secure what is placed in the blob storage and therefore onto the CDN.
Thanks Jeavon, are you able to share any details of the security issues you encountered so that we can make an assessment of whether we are likely to be impacted by it or not?
Questions!
Hi Jeavon,
Thanks for piecing this all altogether.
A quick question though? Why do you go to the trouble of having the CDN source type as Blob storage?
When we've done this setup - we've just pointed the CDN at the Umbraco site, where it requests the image/asset on the original URL the first time and subsequently caches it.
In this approach, you don't have to upload your CSS, Javascript etc to CDN as the CDN fetches them from Umbraco.
With images the only modification you have to make is to prefix the original /media URL with the CDN domain and the image processor stuff still works.
I'm (probably) missing something!
Hi Darren,
I'm new to this and I am in the process of trying to implement an Azure based CDN at the moment and your approach seems far easier so wondered if you can point towards more information on that setup? Is this something that can be done for an Umbraco instance not hosted on the Azure platform?
I wish I'd seen your post before I just transferred 5GB of media to my Azure CDN test account :(
Cheers, Simon
I've been meaning to blog about it!
You use: https://our.umbraco.org/projects/collaboration/umbracofilesystemprovidersazure/
To use the backoffice for media in blob storage.
You set up a CDN endpoint pointing at your umbraco site (Custom endpoint).
In your templates you put the CDN URL on the front of the URL:
e.g. http://mycdn.azureedge.net/media/32983/what.jpg
You need to setup that CDN endpoint to cache every unique request.
The CDN caches hard, so you may want to put the update date of the media item into the URL e.g.
http://mycdn.azureedge.net/media/32983/what.jpg?ud=201708091032
So that you get the new version each time the media item is saved.
Hi Darren,
This was super simple to setup with HTTP - so thanks! Currently I can't get it to work with HTTPS due to a gateway error which I am yet to resolve. Some blog posts I have been reading seem to indicate it might be related to the fact we have SNI enabled on the server but that is a discussion for a separate thread.
Is this optional? It seems to still work without this step?
Initially I thought that this approach was going to copy the assets into my blob storage container but now it seems the storage container is not even required - am I correct? If so, even more reason for this approach to be favoured! It's such a light and easy integration.
It would be great if Jeavon could contribute to this thread with reasons to favour his approach over yours.
Thanks, Simon
Hi Simon,
We've not had issues with https....
I guess you could use the approach without blob and Azure filesystem providers - but Pete makes a good point, now I've read it properly.
Let us know about the SNI issue, either here or on another thread.
Thanks Darren, I'll be sure to try this as it seems so much more straightforward with less config to manage in Umbraco.
If you upload to blob you get some advantages if you have multiple developers/editors/installed instances I believe.
The assets are stored in one place so if you pull the site down to develop on it you don't need to get all the media too. This works with Darren's setup too of course but you end up with one Umbraco site that has the data and all the others that don't so you now have a tricky situation if anyone deletes the wrong site, by having the assets stored shared but separate from Umbraco you keep a nice separation.
If you then work locally and add some assets for a new feature they get stored locally on your local version not the live site that the CDN will ask for the assets from. If you then forget to copy them over you end up breaking things. Its an easy mistake to make and can be a tricky one to remember and fix come deployment time.
Both setups work, Darrens is easier (if you only have one developer/site on the whole) but Jeavons is more belt and braces with the complete separation of the assets from Umbraco.
Centralised media is a bonus, especially as this site, in particular, has over 5Gb of media currently. Say all of your environments are using the shared blob storage, how do you manage the possibility that media will be uploaded from a test/staging environment that is not intended to be used in production? Or, is this where the CDN toolkit comes in handy?
You need a blob storage container per umbraco environment/database e.g. dev/stage/prod.
You can configure the container that is used in the Azure filesystem provider config.
You probably also want a CDN endpoint per environment.
Hi Pete - my solution uses blob storage too??
Oh, from Simons description of it not copying the assets to blob (is it maybe storing resized image in the blob storage?) it sounded like it wasn't.
sorry i'll read it properly this time and reply to simon :)
To remove any confusion, right now I have just setup the origin to point to the staging site and can get media returned via http://
I understand that once I add
UmbracoFileSystemProviders.Azureinto the mix that any media uploaded will then go to the blob storage - correct? But, without this step I'm curious where any media is being drawn from? Is it simply cached across Azure CDN Servers with no visible storage after the first request?Sorry for the noob questions but this is my first time so please be gentle :)
Azure CDN will store copy of the media on it's own servers.
Upon first request - if it doens't hold a copy of the item it will get it from Umbraco.
If you don't set up Azure filesystem providers - it will get the copy to cache from the local file system where umbraco lives, otherwise it would read it from Blob storage.
Thanks for the clarification. So in the case of an existing site, I need to still push all existing media up to the blob storage container if I am to enable the Azure filesystem provider?
Yup - there is a tool called Azure storage explorer.
We also wrote a console app that does it with multiple threads - and byte size comparison which is quite fast for large amounts of media:
https://github.com/darrenferguson/Moriyama.BlobCopy
Oh I've already been through the pain of that piece of software - it should be READ-ONLY! It could not handle upload of the existing media, looked like it was creating multiple background processes for each item and just hanging! What I used in the end was AZCopy and the following command and it was super quick in comparison:
Since I've got to do it again now I will try out your app - thanks!
Thanks for this Darren, uploading nice and quickly now. I made a small pull request to set the MIME type for files before uploading and it seems to be working well for me so far.
First things first, Darren's approach is called origin pull, my approach (or the approach of the CDN toolkit) is called origin push.
We originally planned to use origin pull as it's very simple however we hit some security issues during testing.
Therefore the reason we created the CDN Toolkit an was to enable a CDN push approach which would allow us to secure what is placed in the blob storage and therefore onto the CDN.
Thanks Jeavon, are you able to share any details of the security issues you encountered so that we can make an assessment of whether we are likely to be impacted by it or not?
Thanks, Simon
Can you Slack me on the community channel?
I will try my best to do that as soon as I can - thanks
I have sent you some info on there already :)
is working on a reply...