Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Justin Spradlin 139 posts 347 karma points
    Feb 26, 2012 @ 00:44
    Justin Spradlin
    1

    S3 Provider does not get more than 1000 items from bucket

    Morten,

    I sent you a G+ message, but I'll share here in case anyone else has this problem. I was having a problem where my bucket was not returning all of the items listed. So I created a quick console app and added a reference to your Sitereactor.CloudProvider.Amazon library. I did a quick call to GetAllObjectsByBucketName. It showed that I was only getting back 1000 items. So I did a little digging and found that S3 returns paged results if you have more than 1000 items. 

    I used dotPeek to decompile your .dll and saw that it would be very easy to update the code for the StorageFactory class. I added a little while loop to keep requesting items as long as there were more results.

    Can you add this to the next version of the S3 provider?

     

     

    public Dictionary<stringlong> GetAllObjectsByBucketName(string bucketName)
            {
                var s3Objects = new List<S3Object>(); 
    
                var results =this._client.ListObjects(new ListObjectsRequest()
                {
                    BucketName = bucketName
                });
    
                s3Objects.AddRange(results.S3Objects);
    
                // Keep getting results until there are no more. 
                // The default implementation only returns 1000 objects.
                while (results.IsTruncated)
                {
                    results = this._client.ListObjects(new ListObjectsRequest()
                        {
                            BucketName = bucketName,
                            Marker = s3Objects.Last().Key
                        });
                    s3Objects.AddRange(results.S3Objects);
                }
    
                return s3Objects.ToDictionary((o => o.Key), (Func<S3Objectlong>)(o => o.Size));
            }

     

    Thanks,

    Justin

  • Morten Christensen 596 posts 2773 karma points admin hq c-trib
    Feb 26, 2012 @ 10:52
    Morten Christensen
    0

    Thanks Justin!

    I have to push the provider to bitbucket anyway, so I'll add your fix when doing so.

    I'll report back when its ready.

    - Morten

  • Morten Christensen 596 posts 2773 karma points admin hq c-trib
    Mar 31, 2012 @ 15:11
    Morten Christensen
    0

    Hi Justin,

    I pushed the source for the various cloud providers to github (incl. your fix):
    https://github.com/sitereactor/Cloud-Proivders-for-Universal-Media-Picker

    If you have a chance to test the updated S3 provider please let me know.

    - Morten

  • Justin Spradlin 139 posts 347 karma points
    Mar 31, 2012 @ 15:49
    Justin Spradlin
    0

    Sure, I'd be glad to. It may be later this afternoon before I can do it though. Do you have a precompiled version or do I need to compile from the source?

    Justin

  • Morten Christensen 596 posts 2773 karma points admin hq c-trib
    Mar 31, 2012 @ 16:31
    Morten Christensen
    0

    You'd have to get the source and compile, as I haven't added build scripts yet.

    - Morten

  • Justin Spradlin 139 posts 347 karma points
    Apr 01, 2012 @ 04:56
    Justin Spradlin
    0

    Morten,

    I just cloned the repo and built the source. The first issue is that the lib folder doesn't have the Amazon or Azure SDK dlls. I was able to build the Amazon providers by using the Amazon dll I had in the bin folder of my dev site. 

    The issue I saw was that "folders" in my bucket only displayed at the top level, any folders nested deeper than that showed the regular file icon and you couldn't expand them like the a folder. A second issue that happens in CloudBerry Explorer too, is that the items are not sorted by name with folders first. 

    I hope this helps. 

    Thanks,

    Justin

  • Morten Christensen 596 posts 2773 karma points admin hq c-trib
    Apr 01, 2012 @ 09:17
    Morten Christensen
    0

    Hi Justin,

    Sorry about that! I had an entry in my .gitignore to exclude packages folder from source control. I removed that entry and added the used nuget packages, so references should be good again.

    I will look into the issue with the nested folders as I'm sure I had fixed that a while ago. The sorting should be fairly simple to fix.

    Thanks,

    Morten

  • Dan Diplo 1554 posts 6205 karma points MVP 6x c-trib
    Apr 12, 2013 @ 13:24
    Dan Diplo
    0

    I just came across this issue and the source code from Git Hub solves the problem. So, Morten, maybe you need to release a new version with this included?

Please Sign in or register to post replies

Write your reply to:

Draft