Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Chris Lord 58 posts 211 karma points
    Feb 27, 2014 @ 22:13
    Chris Lord
    1

    Media and Amazon S3

    Hi All,

    I am using v7.0.3 and a requirement is to store all media in an S3 bucket, not on the web server at all! So, my current solution (which I am not sure is the best solution at this stage), is to map a windows drive to the bucket, using CloudBerry Drive and then set Umbraco to use this "virtual" drive to store the meda.

    This is all looking good, and I changed my FileSystemProviders.config to look like this;            

       <FileSystemProviders>  

      <!-- Media -->

      <Provider alias="media" type="Umbraco.Core.IO.PhysicalFileSystem, Umbraco.Core">

        <Parameters>

          <!--<add key="virtualRoot" value="~/media/" />-->

          <add key="rootPath" value="E:\"/>     

          <add key="rootUrl" value="https://bucketname.s3-external-3.amazonaws.com/" />      

        </Parameters>

      </Provider>   

    </FileSystemProviders>

    So, when I add media in Umbraco, sure enough, it ends up in S3. When I use that media using the RTE, it renders on the site perfectly and the src of the image is the fully qualified URL to the bucket! Great!

    My issue is, in the admin backend, I cannot see a preview of my images.....having inspected the element in the browser, it is requesting the following URL;

    /umbraco/backoffice/UmbracoApi/Images/GetBigThumbnail?originalImagePath=https%3A%2F%2Fbucketname.s3-external-3.amazonaws.com%2F1003%2Fpic3.jpg, which is returning a 500 error (see below);

    An error has occurred.
    The given path's format is not supported.
    System.NotSupportedException
    at System.Security.Util.StringExpressionSet.CanonicalizePath(String path, Boolean needFullPath) at System.Security.Util.StringExpressionSet.CreateListFromExpressions(String[] str, Boolean needFullPath) at System.Security.Permissions.FileIOPermission.AddPathList(FileIOPermissionAccess access, AccessControlActions control, String[] pathListOrig, Boolean checkForDuplicates, Boolean needFullPath, Boolean copyPathList) at System.IO.FileInfo.Init(String fileName, Boolean checkHost) at Umbraco.Core.IO.PhysicalFileSystem.GetLastModified(String path) at Umbraco.Web.Editors.ImagesController.GetResized(String imagePath, Int32 width, String suffix) at lambda_method(Closure , Object , Object[] ) at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass13.b__c(Object instance, Object[] methodParameters) at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments) at System.Threading.Tasks.TaskHelpers.RunSynchronously[TResult](Func`1 func, CancellationToken cancellationToken)

    So, it seems like Umbraco doesn't like the fully qualifed URL....which is a shame, but I am assuming it's trying to create a thumbnail from the original image or something??

    Is there anything I can do here, or am I approching this the wrong way? My goal is that no media is on my server at all, it's sent to S3 and then requested from the client from S3 directly?

    Cheers
    Chris
  • Matt Whetton 19 posts 61 karma points
    Feb 28, 2014 @ 11:24
    Matt Whetton
    100

    Hi Chris (we spoke last night on twitter)

    I've tracked down the source of the problem - there's a bug in Umbraco.Core or Umbraco.Web causing this issue. I'm not sure whether its in PhysicalFileSystem class or the ImagesController. (I don't know where its supposed to be dealt with)

    The problem is that the originalImagePath is url encoded, and it doesnt get decoded anywhere. So when the PhysicalFileSystem class tries to find the file it doesnt strip off the rootUrl.

    The problem could be solved in either the Images controller in the GetResized method, or in the PhysicalFileSystem at the GetRelativePath method. I think it should probably be happening in the ImagesController.

    I know this doesn't solve the problem, but at least it gives a description.

    Matt

  • Chris Lord 58 posts 211 karma points
    Feb 28, 2014 @ 11:39
    Chris Lord
    0

    Thanks Matt,

    This is useful, and when I manually strip this out the image renders. I am not sure if this is a bug, or I am trying to use it in the wrong way (am fairly new to Umbraco), so will have a play and log it as a request to fix if I can't find a way of dealing with it!

    Cheers

    Chris

  • Michael Law 7 posts 37 karma points
    Mar 17, 2014 @ 16:00
    Michael Law
    2

    Hey Man,

    I just started using a DFS path for our media items save url, and serving them up from a site in IIS, and came across this issue too, so thought i'd log how we fixed it. The PhysicalFileSystem provider expects a local file so it can use the File.IO to get the dates. This obviously wont work with remote files.

    public DateTimeOffset GetLastModified(string path)
        {
            return DirectoryExists(path) 
                ? new DirectoryInfo(GetFullPath(path)).LastWriteTimeUtc 
                : new FileInfo(GetFullPath(path)).LastWriteTimeUtc;
        }
    

    If you create your own FileSystem provider and just use the PhysicalFileSystem to implement everything as usual with the exception of the GetLastModified method, that should work. See below implementation.

    public class YomegoFileSystemProvider : IFileSystem
    {
        private IFileSystem _provider { get; set; }
    
        public YomegoFileSystemProvider(string virtualRoot)
        {
            _provider = new PhysicalFileSystem(virtualRoot);
        }
    
        public YomegoFileSystemProvider(string rootPath, string rootUrl)
        {
            _provider = new PhysicalFileSystem(rootPath, rootUrl);
        }
    
        #region physicalfilesystem
    
        public IEnumerable<string> GetDirectories(string path)
        {
            return _provider.GetDirectories(path);
        }
    
        public void DeleteDirectory(string path)
        {
            _provider.DeleteDirectory(path);
        }
    
        public void DeleteDirectory(string path, bool recursive)
        {
            _provider.DeleteDirectory(path, recursive);
        }
    
        public bool DirectoryExists(string path)
        {
            return _provider.DirectoryExists(GetFullPath(path));
        }
    
        public void AddFile(string path, Stream stream)
        {
            _provider.AddFile(path, stream);
        }
    
        public void AddFile(string path, Stream stream, bool overrideIfExists)
        {
            _provider.AddFile(path, stream, overrideIfExists);
        }
    
        public IEnumerable<string> GetFiles(string path)
        {
            return _provider.GetFiles(path, path);
        }
    
        public IEnumerable<string> GetFiles(string path, string filter)
        {
            return _provider.GetFiles(path, filter);
        }
    
        public Stream OpenFile(string path)
        {
            return _provider.OpenFile(path);
        }
    
        public void DeleteFile(string path)
        {
            _provider.DeleteFile(path);
        }
    
        public bool FileExists(string path)
        {
            return _provider.FileExists(path);
        }
    
        public string GetRelativePath(string fullPathOrUrl)
        {
            return _provider.GetRelativePath(fullPathOrUrl);
        }
    
        public string GetFullPath(string path)
        {
            return _provider.GetFullPath(path);
        }
    
        public string GetUrl(string path)
        {
            return _provider.GetUrl(path);
        }
    
        public DateTimeOffset GetCreated(string path)
        {
            return _provider.GetCreated(path);
        }
    
        #endregion physicalfilesystem
    
        #region ours
    
        public DateTimeOffset GetLastModified(string path)
        {
            // ML - Work around for using a UNC DFS path
    
            if (Regex.IsMatch(path, "^(http|https)://.*$"))
            {
                var request = WebRequest.Create(path) as HttpWebRequest;
    
                if (request != null)
                {
                    request.Method = "HEAD";
    
                    var response = request.GetResponse() as HttpWebResponse;
    
                    if (response != null)
                    {
                        if (response.StatusCode == HttpStatusCode.OK)
                        {
                            return response.LastModified;
                        }
                    }
                }
            }
    
            return _provider.GetLastModified(path);
        }
    
        #endregion ours
    }
    

    What would be much nicer if the methods on the PhysicalFileSystem were marked as virtual so we could just override the ones we want after making our class inherit from PhysicalFileSystem, but suppose we cant have everything. You could just use the 'new' keyword before your implementation of GetLastModified but that's not a very transparent pattern i suppose.

  • Michael Law 7 posts 37 karma points
    Mar 17, 2014 @ 16:07
    Michael Law
    1

    Our FileSystemPRoviders config looks like this

    <FileSystemProviders>

      <!-- Media -->

      <Provider alias="media" type="Website.Core.Plugins.Umbraco.Providers.YomegoFileSystemProvider, Website.Core.Plugins.Umbraco">

        <Parameters>

          <add key="rootPath" value="\\[unc]\[client]\[project]\[environment]\media\" />

          <add key="rootUrl" value="http://our.image.url/[client]/[project]/[environment]/media/" />

        </Parameters>

      </Provider>

    </FileSystemProviders>

Please Sign in or register to post replies

Write your reply to:

Draft