S3 Provider does not get more than 1000 items from bucket
Morten,
I sent you a G+ message, but I'll share here in case anyone else has this problem. I was having a problem where my bucket was not returning all of the items listed. So I created a quick console app and added a reference to your Sitereactor.CloudProvider.Amazon library. I did a quick call to GetAllObjectsByBucketName. It showed that I was only getting back 1000 items. So I did a little digging and found that S3 returns paged results if you have more than 1000 items.
I used dotPeek to decompile your .dll and saw that it would be very easy to update the code for the StorageFactory class. I added a little while loop to keep requesting items as long as there were more results.
Can you add this to the next version of the S3 provider?
publicDictionary<string, long> GetAllObjectsByBucketName(string bucketName)
{
var s3Objects = newList<S3Object>();
var results =this._client.ListObjects(newListObjectsRequest()
{
BucketName = bucketName
});
s3Objects.AddRange(results.S3Objects);
// Keep getting results until there are no more. // The default implementation only returns 1000 objects.while (results.IsTruncated)
{
results = this._client.ListObjects(newListObjectsRequest()
{
BucketName = bucketName,
Marker = s3Objects.Last().Key
});
s3Objects.AddRange(results.S3Objects);
}
return s3Objects.ToDictionary((o => o.Key), (Func<S3Object, long>)(o => o.Size));
}
Sure, I'd be glad to. It may be later this afternoon before I can do it though. Do you have a precompiled version or do I need to compile from the source?
I just cloned the repo and built the source. The first issue is that the lib folder doesn't have the Amazon or Azure SDK dlls. I was able to build the Amazon providers by using the Amazon dll I had in the bin folder of my dev site.
The issue I saw was that "folders" in my bucket only displayed at the top level, any folders nested deeper than that showed the regular file icon and you couldn't expand them like the a folder. A second issue that happens in CloudBerry Explorer too, is that the items are not sorted by name with folders first.
Sorry about that! I had an entry in my .gitignore to exclude packages folder from source control. I removed that entry and added the used nuget packages, so references should be good again.
I will look into the issue with the nested folders as I'm sure I had fixed that a while ago. The sorting should be fairly simple to fix.
I just came across this issue and the source code from Git Hub solves the problem. So, Morten, maybe you need to release a new version with this included?
S3 Provider does not get more than 1000 items from bucket
Morten,
I sent you a G+ message, but I'll share here in case anyone else has this problem. I was having a problem where my bucket was not returning all of the items listed. So I created a quick console app and added a reference to your Sitereactor.CloudProvider.Amazon library. I did a quick call to GetAllObjectsByBucketName. It showed that I was only getting back 1000 items. So I did a little digging and found that S3 returns paged results if you have more than 1000 items.
I used dotPeek to decompile your .dll and saw that it would be very easy to update the code for the StorageFactory class. I added a little while loop to keep requesting items as long as there were more results.
Can you add this to the next version of the S3 provider?
Thanks,
Justin
Thanks Justin!
I have to push the provider to bitbucket anyway, so I'll add your fix when doing so.
I'll report back when its ready.
- Morten
Hi Justin,
I pushed the source for the various cloud providers to github (incl. your fix):
https://github.com/sitereactor/Cloud-Proivders-for-Universal-Media-Picker
If you have a chance to test the updated S3 provider please let me know.
- Morten
Sure, I'd be glad to. It may be later this afternoon before I can do it though. Do you have a precompiled version or do I need to compile from the source?
Justin
You'd have to get the source and compile, as I haven't added build scripts yet.
- Morten
Morten,
I just cloned the repo and built the source. The first issue is that the lib folder doesn't have the Amazon or Azure SDK dlls. I was able to build the Amazon providers by using the Amazon dll I had in the bin folder of my dev site.
The issue I saw was that "folders" in my bucket only displayed at the top level, any folders nested deeper than that showed the regular file icon and you couldn't expand them like the a folder. A second issue that happens in CloudBerry Explorer too, is that the items are not sorted by name with folders first.
I hope this helps.
Thanks,
Justin
Hi Justin,
Sorry about that! I had an entry in my .gitignore to exclude packages folder from source control. I removed that entry and added the used nuget packages, so references should be good again.
I will look into the issue with the nested folders as I'm sure I had fixed that a while ago. The sorting should be fairly simple to fix.
Thanks,
Morten
I just came across this issue and the source code from Git Hub solves the problem. So, Morten, maybe you need to release a new version with this included?
is working on a reply...