You can place them in a folder on your web server and use the File Picker datatype for selection. You're not required to keep them in the media section.
IMHO the standard Umbraco media library will be able to handle this, however it depends mainly on how you want to organize your media and what functionality you require, e.g.:
- do you need to loop over your media items regularly and you can't reasonably limit the number of items per loops e.g. by re-organizing your media items (categories, etc)?
- Will you end up with a large number of media items in one folder (e.g. > than 100 or 200 items), which can't for whatever reason be further organized, e.g. by using an uploaded date folder structure or similar?
- Searching can be made comfortable easy when the items are in the library with Examine, yet you might need to think of writing a custom picker as selecting a thumbnail image out of the available 10k ones will be quite a strenuous task for an editor.
There are a number of tools out there to automatically sort your items in certain folders so they become easier to handle in the backend. As Umbraco only loads sub nodes when you requests them you should be okay with performance. So I guess the biggest issue here is functionality. If all you are doing is let the editor associate e.g. a product that you want to display on a page with a couple of product images that should be absolutely fine as it doesn't depend at all on the number of items you have in the library. If however say you tag each image and on the frontend need to display all the images with a certain tag then that might get a bit problematic. On the other hand, that will get quite problematic independently of the system you are using, so a custom implementation using Examine might be a good bet here.
There was a demo of Umedial at the Umbraco BE Festival and it's already pretty far. You can probably contact Martijn Beumers if you want a demo. No idea about pricing...
sorry for late reply. Each item in the media library has it's own folder (named after the id of the file upload property of that media node), so looking up the file shouldn't be a problem, however all folders which wrap the actual files are on one level, so with 10k media items Windows has to find your folder out of 10k folders. That is obviously not ideal, though to be absolutely honest I don't know how Windows searches the file/folder structure, we might be talking about 1/1000 seconds here. Nevertheless during that search the server will have a high load, and with enough concurrent users/requests that will be a proper knockout.
The way the file upload/media items work in the backend is the following: after selecting a file in the upload and hitting save the item will be copied to the media folder (aka a folder named after the id of the file upload property), it might also create other versions e.g. for an image a thumbnail. That's the physical part of it. After that the value of the file upload property will be set to the relative path to the item, starting at '/media', e.g. '/media/12345/myFile.pdf'. For the standard implementation that property is 'umbracoFile'. So if you use that e.g. in an Xslt it will return the relative location of the physical file which is enough for the browser to fetch the image. All I'm saying here is that the physical file is just linked to the media node via the relative path entry in 'umbracoFile'.
Now my way to solve your issue would be to create my own file upload data type (as a user control and then using the user control wrapper to create an instance of it in Umbraco), which is nothing more than a file upload control and a Value property. Upon save you are now in charge where you want to upload the physical file to and what you want the value of the control (aka the relative link to the file) to be. So you could e.g. create a date folder structure in your /media folder (or any other folder) like this for a file uploaded today:
/media /2011 /03 /23 /123456 (aka the property id of your custom data type for this media node) /myFile.pdf
The value of the field would then be set to '/media/2011/03/23/123456/myFile.pdf'. The Windows file system would have to do 6 look-ups instead of 3, however I highly doubt there will be more than 50 items in each folder.
Might be worth though having a bit of a go at seeing how Windows does on say 1000 look-ups in sequence of a directory out of 10k or even 100k, that would actually be interesting to know.
I've actually turned to the cloud. I signed up for Rackspace CloudFiles. Looks pretty cheap, 15 cents per GB for storage and 18 cents per GB down. They have a .NET api too. They have a CDN deal with akamai too so it should be much faster/better then anything I can do.
If anyone is interested I can report back in a few months after I've given it some heavy use.
This should handle my linear lookup problem though.
I'd be interested in hearing back about this! I have something similar coming up and have been looking at cloud based file hosting, so I'd be interested to hear your experiences!
Yah, I think it would work well with umbraco. You could hook up the media section to it and mimic the folder structure in containers on the cloud file side.
It looks to be pretty much free unless you have a HUGE site. They also have a data center in the US and UK plus the akamai CDN replication.
Sounds too good to be true, we will see if I get hit with some hidden fees after a couple billing cycles.
Did you ever set anything up with CDN Nate? I have a project coming up that will have hundreds of thousands of media resources and considering my options.
Short of using the cloud, I like Sascha's suggestion of creating your own media upload handler that 'categorises' the media in the /media folder.
Has anyone successfully achieved either approach and do they have any detailed instructions on what they did?
For Umbraco 5 you could try to write your own hive provider to store the media somewhere else. Only not sure if selecting images in TinyMCE supports a hive provider.
Yes, I used a combination of Rackspace Cloudfiles and Azure Blob Storage. I started with CloudFiles but migrated to Azure Blob Storage to get some of the files closer to my server.
I really like this approach. I have over 400k files in my storage already. That would not work well on a single server/folder structure.
Everything gets backed up automatically (3 locations with Azure) and it's pretty cheap. I'm on a bizspark membership and I'm only paying a couple bucks a month. Maybe nothing for storage.
Anyways, I really like this approach. You can switch web servers without a huge headache. What if you put 400k files on a web sever? What happens if the hard drive crashes?
You can do backups but it's just much much easier with the cloud stuff.
Check out Cloud Berry. That's a good tool to view your storage account. Both CloudFiles and Azure have a really good .NET library to interact with their library.
So in conclusion, I'm really really really glad I went with this approach.
Large number of images in media library?
I'm creating a new website that will have a lot of user generated images on it. Is it purdent to put these in the media library or another strategy?
Is there a point where the media library shouldn't be used?
10,000 pictures?
100,000 pictures?
1,000,000 pictures?
Thanks!
Nate
You can place them in a folder on your web server and use the File Picker datatype for selection. You're not required to keep them in the media section.
You could also try umedial. This is created for situations like this. For more information go to http://umedial.com/.
Is umedial even ready for release.. haven't heard much since last year.
hmm, my last two posts didn't post, so hopefully this one does.
Jeroen, when will that be released? Is it free or is there a fee?
Daniel, I'm worried about big linear searches against a file directory.
Hi Nate,
IMHO the standard Umbraco media library will be able to handle this, however it depends mainly on how you want to organize your media and what functionality you require, e.g.:
- do you need to loop over your media items regularly and you can't reasonably limit the number of items per loops e.g. by re-organizing your media items (categories, etc)?
- Will you end up with a large number of media items in one folder (e.g. > than 100 or 200 items), which can't for whatever reason be further organized, e.g. by using an uploaded date folder structure or similar?
- Searching can be made comfortable easy when the items are in the library with Examine, yet you might need to think of writing a custom picker as selecting a thumbnail image out of the available 10k ones will be quite a strenuous task for an editor.
There are a number of tools out there to automatically sort your items in certain folders so they become easier to handle in the backend. As Umbraco only loads sub nodes when you requests them you should be okay with performance. So I guess the biggest issue here is functionality. If all you are doing is let the editor associate e.g. a product that you want to display on a page with a couple of product images that should be absolutely fine as it doesn't depend at all on the number of items you have in the library. If however say you tag each image and on the frontend need to display all the images with a certain tag then that might get a bit problematic. On the other hand, that will get quite problematic independently of the system you are using, so a custom implementation using Examine might be a good bet here.
Hope that helps,
Sascha
There was a demo of Umedial at the Umbraco BE Festival and it's already pretty far. You can probably contact Martijn Beumers if you want a demo. No idea about pricing...
Jeroen
@Sacha I'm worried about a large linear look up on a windows file system, not the nodes in the backend of umbraco.
Hi Nate,
sorry for late reply. Each item in the media library has it's own folder (named after the id of the file upload property of that media node), so looking up the file shouldn't be a problem, however all folders which wrap the actual files are on one level, so with 10k media items Windows has to find your folder out of 10k folders. That is obviously not ideal, though to be absolutely honest I don't know how Windows searches the file/folder structure, we might be talking about 1/1000 seconds here. Nevertheless during that search the server will have a high load, and with enough concurrent users/requests that will be a proper knockout.
The way the file upload/media items work in the backend is the following: after selecting a file in the upload and hitting save the item will be copied to the media folder (aka a folder named after the id of the file upload property), it might also create other versions e.g. for an image a thumbnail. That's the physical part of it. After that the value of the file upload property will be set to the relative path to the item, starting at '/media', e.g. '/media/12345/myFile.pdf'. For the standard implementation that property is 'umbracoFile'. So if you use that e.g. in an Xslt it will return the relative location of the physical file which is enough for the browser to fetch the image. All I'm saying here is that the physical file is just linked to the media node via the relative path entry in 'umbracoFile'.
Now my way to solve your issue would be to create my own file upload data type (as a user control and then using the user control wrapper to create an instance of it in Umbraco), which is nothing more than a file upload control and a Value property. Upon save you are now in charge where you want to upload the physical file to and what you want the value of the control (aka the relative link to the file) to be. So you could e.g. create a date folder structure in your /media folder (or any other folder) like this for a file uploaded today:
/media
/2011
/03
/23
/123456 (aka the property id of your custom data type for this media node)
/myFile.pdf
The value of the field would then be set to '/media/2011/03/23/123456/myFile.pdf'. The Windows file system would have to do 6 look-ups instead of 3, however I highly doubt there will be more than 50 items in each folder.
Might be worth though having a bit of a go at seeing how Windows does on say 1000 look-ups in sequence of a directory out of 10k or even 100k, that would actually be interesting to know.
Hope that makes sense,
Sascha
You can tweek physical path using media events, like in this video.
http://umbraco.com/help-and-support/video-tutorials/developing-with-umbraco/events/friendly-media-item-paths
I tweek it a little (to move thumbnail and files, not only images) and try use it in my new project.
Petr
I've actually turned to the cloud. I signed up for Rackspace CloudFiles. Looks pretty cheap, 15 cents per GB for storage and 18 cents per GB down. They have a .NET api too. They have a CDN deal with akamai too so it should be much faster/better then anything I can do.
If anyone is interested I can report back in a few months after I've given it some heavy use.
This should handle my linear lookup problem though.
Opps, double post
I'd be interested in hearing back about this! I have something similar coming up and have been looking at cloud based file hosting, so I'd be interested to hear your experiences!
:)
Yah, I think it would work well with umbraco. You could hook up the media section to it and mimic the folder structure in containers on the cloud file side.
It looks to be pretty much free unless you have a HUGE site. They also have a data center in the US and UK plus the akamai CDN replication.
Sounds too good to be true, we will see if I get hit with some hidden fees after a couple billing cycles.
Did you ever set anything up with CDN Nate? I have a project coming up that will have hundreds of thousands of media resources and considering my options.
Short of using the cloud, I like Sascha's suggestion of creating your own media upload handler that 'categorises' the media in the /media folder.
Has anyone successfully achieved either approach and do they have any detailed instructions on what they did?
Thanks in advance, Rich.
Hmm how about using the Universal Media Picker for Umbraco 4? Would that work?
For Umbraco 5 you could try to write your own hive provider to store the media somewhere else. Only not sure if selecting images in TinyMCE supports a hive provider.
Jeroen
It's v4 the project so Universal Media Picker looks like a very good choice.
Is it possible to configure a single property to allow selection from multiple sources.
eg... a property that an editor can select from either S3, Flickr or YouTube?
Thanks,
Rich
Hey Rich,
Yes, I used a combination of Rackspace Cloudfiles and Azure Blob Storage. I started with CloudFiles but migrated to Azure Blob Storage to get some of the files closer to my server.
I really like this approach. I have over 400k files in my storage already. That would not work well on a single server/folder structure.
Everything gets backed up automatically (3 locations with Azure) and it's pretty cheap. I'm on a bizspark membership and I'm only paying a couple bucks a month. Maybe nothing for storage.
Anyways, I really like this approach. You can switch web servers without a huge headache. What if you put 400k files on a web sever? What happens if the hard drive crashes?
You can do backups but it's just much much easier with the cloud stuff.
Check out Cloud Berry. That's a good tool to view your storage account. Both CloudFiles and Azure have a really good .NET library to interact with their library.
So in conclusion, I'm really really really glad I went with this approach.
-Nate
Brilliant, thanks for the feedback. I look forward to tackling this in the near future.
Rich.
is working on a reply...