Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Mac McDell 73 posts 148 karma points
    Apr 18, 2016 @ 21:33
    Mac McDell
    0

    imageProcessor for AzureCloudStorage breaks website

    Nutshell:

    • I can confirm I can see media and cache containers in my storage container.
    • I can confirm I see the cache container populating with correctly scaled images.

    Umbraco however will not server the correct cache location url for images.

    ImageProcessor.AzureBlobCache breaks cdn functionality. I am trying to get an Azure CDN to work with AzureMedia storage as per the recommendations from umbraco folks. On the video as soon as Jeavon does the CDN config and builds, he is now delivering over CDN. This does not work on multiple new install tests I have now done (3 now). My situation is that I have images uploading to blob but the cdn is not reflected AND any image that is querystring'ed (http://localhost:56918/media/1058/img20130804123040.jpg?height=200) will 404. 100% of my network traffic is 200. No 304 or CDN anywhere.

    Was trying to follow the youtube tutorial with Jeavon Leopold to get Umbraco using blob storage and CDN. https://www.youtube.com/watch?v=5Qp-ZGaKFaY

    Test Scenario Version: Umbraco V 7.4.3. Scenario: Brand new fanoe site using SQL CE via nuget. Steps to reproduce:

    1. From VS: Install new empty web application 4.5.2.
    2. Install 0.5.1 from UmbracoFileSystemProvider. https://our.umbraco.org/projects/collaboration/umbracofilesystemprovidersazure/
    3. Set Azure Cloud Storage setting
    4. Build and Test. --uploads should work to the blob storage and files should be getting server from blob with vpp http://localhost:56918/media/1058/img20130804123040.jpg
    5. Run the Nuget package UmbracoFileSystemProviders.Azure v 0.5.1-beta -- he mentions you need to do this for CDN setup.
    6. Build and test (should be same behaviour)
    7. Setup a CLASSIC CDN on manage.windowsazure.com (the video says it doesn't work with newer one) and link it to your storage account.
    8. Enable querystrings on the CDN.
    9. Install-Package ImageProcessor.web.Plugins.AzureBlobCache v 1.0.8. --> This is where things break down
    10. Set the azure storage keys and CDN path. on ImageProcessor/Cache.config.
    11. Set currentCache = "AzureBlobCache"
    12. Build (according to the video that should be it.
    13. Test

    Result:

    • Image are still pointing with 200 to http://localhost:56918/media/1058/img20130804123040.jpg.
    • Images with any querystring param are 404ing.
    • cdn path is not showing in network traffic anywhere.

    config settings :

    imageprocessor/cache.config    
    <caching currentCache="AzureBlobCache"><caches>
            <cache name="AzureBlobCache" type="ImageProcessor.Web.Plugins.AzureBlobCache.AzureBlobCache, ImageProcessor.Web.Plugins.AzureBlobCache" maxDays="365">
              <settings>
                <setting key="CachedStorageAccount" value="DefaultEndpointsProtocol=https;AccountName=myStorageAccountName;AccountKey=123456789NVOvyV/Wku0a123456" />
                <setting key="CachedCDNRoot" value="http://mycdnEndpoint.vo.msecnd.net/" />
              </settings>
            </cache>
          </caches>
        </caching>
    
    imageprocessor/security.config
      <service name="LocalFileImageService" type="ImageProcessor.Web.Services.LocalFileImageService, ImageProcessor.Web" />
        <service prefix="media/" name="CloudImageService" type="ImageProcessor.Web.Services.CloudImageService, ImageProcessor.Web">
          <settings>
            <setting key="MaxBytes" value="8194304" />
            <setting key="Timeout" value="30000" />
            <setting key="Host" value="http://myStorageAccountName.blob.core.windows.net/media/" />
          </settings>
        </service>
      </services>
    
    FileSystemProviders.config
    <?xml version="1.0"?>
    <FileSystemProviders>
    
      <!-- Media -->
      <Provider alias="media" type="Our.Umbraco.FileSystemProviders.Azure.AzureBlobFileSystem, Our.Umbraco.FileSystemProviders.Azure">
        <Parameters>
            <add key="containerName" value="media"/>
            <add key="rootUrl" value="http://myStorageAccountName.blob.core.windows.net/"/>
            <add key="connectionString"
                value="DefaultEndpointsProtocol=https;AccountName=myStorageAccountName;AccountKey=g7r7MUE0/yBN="/>
            <!--
            Optional configuration value determining the maximum number of days to cache items in the browser.
            Defaults to 365 days.
          -->
            <add key="maxDays" value="365"/>
        </Parameters>
      </Provider>
    
    </FileSystemProviders>
    
  • Jeavon Leopold 3072 posts 13628 karma points MVP 10x admin c-trib
    Apr 18, 2016 @ 22:18
    Jeavon Leopold
    0

    Hi Mac,

    You are missing two keys in cache.config

    e.g.

        <setting key="CachedBlobContainer" value="cache" />
        <setting key="UseCachedContainerInUrl" value="true" />
    

    Everything else is good, please note on step 7, there is no problem using the new portal and new type of CDN (in fact, I would recommend it now).

    Hope that sorts it.

    Jeavon

  • Mac McDell 73 posts 148 karma points
    Apr 18, 2016 @ 22:41
    Mac McDell
    0

    Hi Jeavon... I do actually have these settings in .. (my bad... i should have stated that) . I was just truncating my code... full file looks like this:

    <?xml version="1.0" encoding="utf-8"?>
    <caching currentCache="AzureBlobCache">
      <caches>
        <cache name="DiskCache" type="ImageProcessor.Web.Caching.DiskCache, ImageProcessor.Web" maxDays="365">
          <settings>
            <setting key="VirtualCachePath" value="~/app_data/cache" />
          </settings>
        </cache>
        <cache name="AzureBlobCache" type="ImageProcessor.Web.Plugins.AzureBlobCache.AzureBlobCache, ImageProcessor.Web.Plugins.AzureBlobCache" maxDays="365">
          <settings>
            <setting key="CachedStorageAccount" value="DefaultEndpointsProtocol=https;AccountName=MyStorageAccountName;AccountKey=/Wku0aWumPtG7MUE0/yBNL" />
            <setting key="CachedBlobContainer" value="cache" />
            <setting key="UseCachedContainerInUrl" value="true" />
            <setting key="CachedCDNRoot" value="http://MyEndpoint.vo.msecnd.net/" />
            <setting key="SourceStorageAccount" value="" />
            <setting key="SourceBlobContainer" value="" />
            <setting key="StreamCachedImage" value="false" />
          </settings>
        </cache>
      </caches>
    </caching>
    

    I can see the image getting written to the blob cache but I don't get the url serving to me. i can see this is getting uploaded: https://tdreditorials.blob.core.windows.net/cache/8/9/4/4/f/f/8944ffa7b6336130c0f9f08d09630e6fa02f1e37.jpg

    ...and i have this uploaded: https://tdreditorials.blob.core.windows.net/media/1042/logo.jpg

    BUT .. when i run the site: I get 404 on page looking for this: http://localhost:56918/media/1042/logo.jpg?height=65&width=205&bgcolor=000

    The site is not taking the /media standard urls and converting them to the cdn.

    PS: Thanks so much for getting back to me .. I am on day 3 of trying to get this to work.

    PPS: Your Security.Config settings say to disable the LocalFileImageService on line 5 and enable cloudImageService so I did that as well... RemoteImageService is still enabled.

  • Jeavon Leopold 3072 posts 13628 karma points MVP 10x admin c-trib
    Apr 18, 2016 @ 22:44
    Jeavon Leopold
    0

    Ok, one step at a time, can you change the currentCache to DiskCache and see if the images show up?

  • Jeavon Leopold 3072 posts 13628 karma points MVP 10x admin c-trib
    Apr 18, 2016 @ 22:45
    Jeavon Leopold
    1

    Also how long ago did you create the CDN, it says 90 minutes but it can take more like 12 hours in my experience and until it's ready you get 404s

  • Mac McDell 73 posts 148 karma points
    Apr 18, 2016 @ 22:51
    Mac McDell
    0

    Hi. Switched to DiskCache and 100% success. I get the umbraco logo with querystring param serving fine along with everything else. CDN was created ~7 hours ago now. When I tried uploading in media after CDN switch, the images were all 404 due to qs extension. enter image description here

  • Jeavon Leopold 3072 posts 13628 karma points MVP 10x admin c-trib
    Apr 18, 2016 @ 23:11
    Jeavon Leopold
    0

    Ok, so FileSystemProvider is work fine.

    If you request an image directly from Cdn, e.g. http://MyEndpoint.vo.msecnd.net/media/1042/logo.jpg do you get anything (assuming media and cache are in the same account)

  • Mac McDell 73 posts 148 karma points
    Apr 19, 2016 @ 02:44
    Mac McDell
    0

    Hi. I get 404 when I try and hit http://endpoint891634.vo.msecnd.net/media/1042/logo.jpg

    Should the urls be resolving with that address or should they be resolving as the cached path?

  • Jeavon Leopold 3072 posts 13628 karma points MVP 10x admin c-trib
    Apr 18, 2016 @ 23:13
    Jeavon Leopold
    0

    One other check to do is to ensure that the cache container is set to public not private

  • Mac McDell 73 posts 148 karma points
    Apr 19, 2016 @ 04:59
    Mac McDell
    0

    As per your suggestion I switched to the new portal.azure.com and setup a new endpoint using the cdnProfile and its new configuration. I created both a new storage container and endpoint. I updated my keys and ran the site. Right away I got media and cache folders created on the storage containers. At that point I set the media and cache folder permissions to BLOB / Public read access. After doing this I had the same behavior (broken images with querystring and no endpoint). I fussed around a bit more and then suddenly I noticed that things started to speed up. I went back to the home page and noticed that the logo now had a 302 redirect from http://localhost:56918/media/1042/logo.jpg?height=65&width=205&bgcolor=000 to http://umb743.azureedge.net/cache/c/d/0/5/5/0/cd0550ba17e83d032005aa8fab4f33b09dc289ce.jpg WIN!

    The other images however like the bacci one on the bottom don't get pushed to cdn for some reason.. I can't explain why http://localhost:56918/media/1046/14386124825d43f359900hcropped.jpg If I add a height querystring onto it then immediately I am redirected to a cached instance http://localhost:56918/media/1046/14386124825d43f359900hcropped.jpg?height=200 302s to http://umb743.azureedge.net/cache/d/d/4/c/a/3/dd4ca3d51c8e0c686beeeb6815f3431217a37b7c.jpg

    Is this expected behaviour? I notice the same thing with an image I uploaded into the 3 wide colum. I uploaded a 480X270 tall gif and then stuck into one of the 3 wide columns in fanoe home page. It won't get a 302 either even though it is scaled to 276 wide.

    Either way it seems to be working I am just not sure if the anomalies are expected.

    Thank you for your help in running this down.
    FYI - other pages (http://localhost:56918/explore/our-umbraco/) that have images with qstring are getting pushed to cdn correctly.

  • James Jackson-South 489 posts 1747 karma points c-trib
    Apr 19, 2016 @ 04:34
    James Jackson-South
    0

    Hi Mac,

    Hi. I get 404 when I try and hit http://endpoint891634.vo.msecnd.net/media/1042/logo.jpg

    Should the urls be resolving with that address or should they be resolving as the cached path?

    Is /media/1042/logo.jpg an actual image on your website?

    It seems to me that you might be getting confused by the pipeline process.

    UmbracoFileSystemProviders.Azure works directly against the blob container, not against the CDN. Only Imageprocessor.Web will work against the CDN. (New or Classic, it doesn't matter).

    I'm going to layout some config files (you'll probably have all this set out already.)

    Your FileSystemProviders.config file should be as follows:

    <?xml version="1.0"?>
    <FileSystemProviders>
      <Provider alias="media" type="Our.Umbraco.FileSystemProviders.Azure.AzureBlobFileSystem, Our.Umbraco.FileSystemProviders.Azure">
        <Parameters>
          <add key="containerName" value="media" />
          <add key="rootUrl" value="http://[myAccountName].blob.core.windows.net/" />
          <add key="connectionString" value="DefaultEndpointsProtocol=https;AccountName=[myAccountName];AccountKey=[myAccountKey]"/>
          <!--
            Optional configuration value determining the maximum number of days to cache items in the browser.
            Defaults to 365 days.
          -->
          <add key="maxDays" value="365" />
        </Parameters>
      </Provider>
    </FileSystemProviders>
    

    Where rootUrl points to the url to blob container.

    You then need to insure that your ImageProcessor security.config file is set as following:

    <?xml version="1.0"?>
    <security>
      <services>
        <service prefix="media/" name="CloudImageService" type="ImageProcessor.Web.Services.CloudImageService, ImageProcessor.Web">
          <settings>
            <setting key="MaxBytes" value="8194304"/>
            <setting key="Timeout" value="30000"/>
            <setting key="Host" value="http://[myAccountName].blob.core.windows.net/media/"/>
          </settings>
        </service>
      </services>  
    </security>
    

    Remove the LocalImageService leaving only the CloudImageService The Host value points to the same root url as the prior config but with media/ appended to the end. Your prefix setting should be media/ also.

    Your cache.config file should be set as follows:

    Depending on how you set the root of your CDN endpoint in Azure you may have to set the UseCachedContainerInUrl to false.

    So how do we go about testing this?

    First off, are you able to post a link to an image in the media folder and the cached folder? It would be great if you can post both the CDN and direct storage urls. (I find CloudBerry Explorer for Azure Blob Storage a great tool for grabbing the correct url from the blob storage account).

    That will let us know whether the cache folder name is being used in the CDN path.

    Once you can supply that we will be able to ensure the config is set up appropriately and get you up and running.

    I would additionally check your Azure setting to make sure everything has the right permissions (Jeavon was right to ask about that).

    • Your blob container should have the access policy blob
    • Your CDN should have origin type Storage, you shouldn't need to set an querystring setting since the final url won't have any.

    Hope this helps

    James

  • Mac McDell 73 posts 148 karma points
    Apr 19, 2016 @ 05:24
    Mac McDell
    0

    Hi James thanks for your reply. I was writing while you were. Please see my response above to Jeavon. It works... but not as I would expect. You guys keep asking if something like this works: original url:

       <img src="/media/1049/9027510123_92a91b5cf4_b_dark.jpg?width=1920&amp;height=700&amp;center=0.340823970037453,0.54&amp;mode=crop">
    

    does 302 redirect to http://umb743.azureedge.net/cache/3/a/1/f/f/b/3a1ffbe5b6ff7a0011f10eefca03f52e4e2d5f5e.jpg but this does not resolve at all. http://umb743.azureedge.net//media/1049/902751012392a91b5cf4b_dark.jpg?width=1920&height=700&center=0.340823970037453,0.54&mode=crop

    I was thinking maybe it was because it was existing content but this: http://localhost:56918/media/1053/facepalm.jpg does neither a redirect or create a cache instance or create a cdn endpoint like http://umb743.azureedge.net/media/1046/14386124825d43f359900hcropped.jpg

    I am happy to send you guys my setup. It is about 120mb zipped including the dlls.

    PS: CDN was setup as storage, permissions on blob are public, UseCachedContainerInUrl = false makes all images 500 that were previously in cache. With false I get server error requesting http://localhost:56918/media/1042/logo.jpg?height=65&width=205&bgcolor=000: The remote server returned an error: (400) Bad Request The logo is just the fanoe/umbraco logo

  • James Jackson-South 489 posts 1747 karma points c-trib
    Apr 19, 2016 @ 09:44
    James Jackson-South
    0

    Hi Mac,

    The other images however like the bacci one on the bottom don't get pushed to cdn for some reason.. I can't explain why http://localhost:56918/media/1046/14386124825d43f359900hcropped.jpg If I add a height querystring onto it then immediately I am redirected to a cached instance http://localhost:56918/media/1046/14386124825d43f359900hcropped.jpg?height=200 302s to http://umb743.azureedge.net/cache/d/d/4/c/a/3/dd4ca3d51c8e0c686beeeb6815f3431217a37b7c.jpg

    Is this expected behaviour? I notice the same thing with an image I uploaded into the 3 wide colum. I uploaded a 480X270 tall gif and then stuck into one of the 3 wide columns in fanoe home page. It won't get a 302 either even though it is scaled to 276 wide.

    There's no anomaly there. ImageProcessor.Web will only intercept requests that have querystring parameters. It would be incredibly expensive process-wise for it to do otherwise.

    I think you are having trouble understanding how websites work as much as anything.

    Scaling an image in the browser will not change a thing unless you are also changing the image request. A browser will not make additional requests for anything at all unless the requested url changes. Any activity that doesn't make a specific request to the server is client side and is handled purely in the browser.

    When you change the url to append querystring parameters you are sending a request to the server. ImageProcessor.Web will attempt to intercept and parse the request. If it is one it recognises only then will it process, cache and redirect the request.

    http://umb743.azureedge.net//media/1049/902751012392a91b5cf4b_dark.jpg?width=1920&height=700&center=0.340823970037453,0.54&mode=crop

    Will never work. That is as expected. You are making a request to a url outside of your domain and as such there is no way for ImageProcessor.Web to intercept it. You simply cannot intercept a request to an external url.

  • Mac McDell 73 posts 148 karma points
    Apr 19, 2016 @ 16:24
    Mac McDell
    0

    Hi. I got the starter site published up here to see what I am trying to explain: http://umbraco743.azurewebsites.net/

    Before responding I wanted to wait until the morning to ensure the CDN had enough time to propagate. What I found was that finally I was getting cdn content that matched what you stated I should have .. ie blob/media/image.jpg == cdn/media/image.jpg. My cdn appears to be working as it should. Images that are run through the Imageprocessor ,anything querystringed or cropped for eg, are getting served through cdn. The issue though is that images not being run through the processor do not appear to be getting served through the CDN on the site. They are visible in the CDN though. Please see screenshot. They appear to still be getting served through the vpp.
    on the video: https://www.youtube.com/watch?v=5Qp-ZGaKFaY at 23:22, Jeavon makes his cache config setting and then 'boom' declares that ALL images are now being served through cdn. You can see this in his network tab. Am I missing something?

    example 1 --Blob VPP , BLOB , CDN

    html markup in page for eg <img class="wide" src="/media/1046/14386124825_d43f359900_h_cropped.jpg"> <-- This looks like it is serving from storage and not the cdn... ?

    example 2

    BLOB VPP ,BLOB, CDN

    image not being run through the processor appear to be serving directly from blob storage and not cdn

  • James Jackson-South 489 posts 1747 karma points c-trib
    Apr 19, 2016 @ 23:27
    James Jackson-South
    0

    Hi Mac,

    Well yeah?!

    Nothing will get served through the CDN unless you specifically tell ImageProcessor.Web to do so by adding querystring parameters. We discussed this above.

    The FileSystemProvider and ImageProcessor.Web are two separate entitities. Neither know about each other at all.

    You configured your FileSystemProvider to use the blob storage account and it doesn't know about the CDN.

    I'm not sure how I can explain it any better.

  • Mac McDell 73 posts 148 karma points
    Apr 20, 2016 @ 04:14
    Mac McDell
    0

    OK Thanks James.

    I already understood that the ImageProcessor is not called if the image does not have any querystring params attached to it. Clearly your package is behaving exactly as it should. I was however under the impression that if we didn't add the querystring param it would instead attempt to serve the image through the CDN. That was a major misunderstanding on my part.
    I guess if there are non processed images I want behind a cdn the easiest thing is to just write a custom extension method or similar. Thanks for taking the time to help out.

  • James Jackson-South 489 posts 1747 karma points c-trib
    Apr 20, 2016 @ 04:44
    James Jackson-South
    1

    No worries, really glad I was able to give you any assistance. It's a very complicated system overall so difficult to explain at times.

    There might, might, be a way I can help you further though.

    There is a setting in the configuration interceptAllRequests that was added in ImageProcessor.Web 4.5.0 that tells the system to intercept all requests.

    <!--interceptAllRequests - Added version 4.5.0 to allow intercepting requests with no querystring.-->
    <processing preserveExifMetaData="false" interceptAllRequests="true">
    

    http://imageprocessor.org/imageprocessor-web/configuration/#processingconfig

    This makes things a little more expensive though as your image will now be processed and encoded by System.Drawing before caching and redirecting to the CDN. That can make images actually larger than they originally were. That would serve all images from the CDN.

    Having a look through my FilesystemProvider code though it looks like there is no reason why I could not use the CDN root url rather than the blob container url in the configuration. If I'm right that would ensure that all processed images come from the CDN.

    e.g

    <?xml version="1.0"?>
    <FileSystemProviders>
      <Provider alias="media" type="Our.Umbraco.FileSystemProviders.Azure.AzureBlobFileSystem, Our.Umbraco.FileSystemProviders.Azure">
        <Parameters>
          <add key="containerName" value="media" />
         <!-- Replace the value here -->
          <add key="rootUrl" value="http://umb743.azureedge.net/" />
          <add key="connectionString" value="DefaultEndpointsProtocol=https;AccountName=[myAccountName];AccountKey=[myAccountKey]"/>
          <!--
            Optional configuration value determining the maximum number of days to cache items in the browser.
            Defaults to 365 days.
          -->
          <add key="maxDays" value="365" />
        </Parameters>
      </Provider>
    </FileSystemProviders>
    

    I suggests you give both a try to see if that aids you.

    Cheers

    James

  • Mac McDell 73 posts 148 karma points
    Apr 21, 2016 @ 17:39
    Mac McDell
    1

    Good news. Forcing all requests through the imageProcessor reduces the image request time almost a whopping 50% after subsequent request.

    Full image load times from the storage account ranged around: 573ms,988ms,1760ms

    Full image load times from the cdn after processing: 228ms, 339ms, 348ms, 371ms

    All of this is with browser caching off. If we compare the two worst times the improvement is roughly 5 times faster. It is worth eating the cost of the processor time loss on the initial request in my opinion.

    For what it is worth just changing the rootUrl setting to the cdn didn't work.

    Thanks for the setting tip! Golden!

Please Sign in or register to post replies

Write your reply to:

Draft