I know the S3 API quite well through hacking on rclone and I have to say it is pretty good as object storage APIs go.
If I could add one thing to it, it would be a way of listing a bucket to get the objects and the metadata back at once - it is another HTTP request to get the metadata otherwise.
The large object handling is pretty good (unlike, lets say the OpenStack Swift API which has 2 ways of uploading large objects both of which basically leave everything hard to the caller!). Large object handling (say objects over 5GB) is the Achilles heel of all the cloud storage systems.
The myriad ways of authenticating with S3 (which have grown like topsy over the years) are a significant complication. However that is the same for all of the Enterprisey cloud storage solutions (eg Azure blob storage).
There are quite a few re-implementation of the S3 API, CEPH, Wasabi, Minio, Alibaba cloud, etc all of which are more or less 100% compatible (though the rclone integration tests have winkled out a few small differences mostly to do with handling of characters like LF and CR).
So impressive work AWS, making an API which has stood the test of time and become a de-facto industry standard.
Thanks for your work! I'm currently using rclone to backup my camera photos. It's lovely software
Also thanks for implementing sync to different storage providers. It's probably tempting to just implement the big players, like S3, but little ol' people like me appreciate the rest :)
It has been a voyage implementing so many cloud providers and keeping them working. You might be surprised how much code is tests which get run against the cloud providers every night in the integration tests!
and then you have GCS implementing their own version of multi-part upload API...
A thing I wish they have is a way to get a consistent view of the object (similar to X-Newest in swift). Charge it 10x or something. Sometimes you really need this.
If I could add one thing to it, it would be a way of listing a bucket to get the objects and the metadata back at once - it is another HTTP request to get the metadata otherwise.
The large object handling is pretty good (unlike, lets say the OpenStack Swift API which has 2 ways of uploading large objects both of which basically leave everything hard to the caller!). Large object handling (say objects over 5GB) is the Achilles heel of all the cloud storage systems.
The myriad ways of authenticating with S3 (which have grown like topsy over the years) are a significant complication. However that is the same for all of the Enterprisey cloud storage solutions (eg Azure blob storage).
There are quite a few re-implementation of the S3 API, CEPH, Wasabi, Minio, Alibaba cloud, etc all of which are more or less 100% compatible (though the rclone integration tests have winkled out a few small differences mostly to do with handling of characters like LF and CR).
So impressive work AWS, making an API which has stood the test of time and become a de-facto industry standard.