­
Lock logo 70px
ExpeditedSSL - Fastest+Easiest AWS SSL


Aws AWS S3 Buckets
of Objects


If you're brand new to Amazon's Simple Storage Service (S3) start with our AWS S3 in Plain English - which will cover what and how to think about S3.

Now that you're back and presumably a Cloudified Architecture Solutions Management Certified Professional - lets dig into the two big concepts of Amazon's Simple Storage Service: Buckets and Objects.


Bucket of Bits

Werner vogels buckets

Pictured above: Artists interpretation of Werner Vogels checking S3 status.

If you’ve dabbled at all with S3 you’ve almost immediately come across the term “bucket” - which is generally described as a kind of top level folder under your S3 AWS account.

And, while that’s true, it’s much easier to get to grips with how to set things up if you consider each “bucket” as if it were a distinct file server.

  • Got an web app staging server? Then you probably need a staging bucket.
  • All of your users in New York? Then you should probably put your bucket in the datacenter closest to New York.
  • Similarly, much of the security and (very cool) features of S3 are set at the bucket level.
  • Additionally, each of your buckets are given a distinct subdomain under the s3.amazonaws.com domain, ex:
    https://thisismybuckettherearemanyotherslikeitbuthtisoneismine.s3.amazonaws.com

Lies, Lies, Lies and Objects

So, here’s the thing. I’ve been saying “files” up until now and it’s really “objects” that are stored in S3. In my defense, even when you’re actually using S3 you and all the other developers will still say to each other: “Hey, did those files copy over to that bucket?""

But S3 isn’t really a file system - and it’s a little hard to get your mind around the fact that it’s not.

Inside of a bucket it’s just a list of unique names (that typically look like filename paths) and then there are the actual bits (which are the objects we call files) stored on the whirring hard drives of AWS.

So when you’re looking at a file in a bucket with a name like:

/avatars/2015/01/22/aang.png

That’s simply the name of that file, in the same way as if the file were called:

_avatars_2015_01_22_aang.png

Thankfully, in the same way that the human mind cannot grasp the infinite expanse of the cosmos, we’re all also incapable of handling more than a dozen files laying around without wanting to organize them in some way.

So, EVERY S3 client participates in the polite fiction that a forward slash means that Oh Boy! there’s a folder here, even when there’s really not one.


Where the Problems Start

First, imagine you have 100 files in your avatar “folder” and you then rename that folder. If it were a normal file system that would be one operation and take a few milliseconds. With S3 it would be 100 separate painful steps as each and every file would have to be touched to rename it (you’re actually moving each file).

The second big problem is that different S3 clients have developed different workarounds for the lack of true folders. So it’s quite common to see weird extra files that are storing information and for different clients to create issues with each other as they both vie for control.

What to Use

It's hard to recommend a singular S3 client as you all of the popular ones work rather well. So it's best to try a few out and see for yourself:


Hey, this is sponsored by SendCheckIt - and by "sponsored" I mean that's what I've been working on instead of updating this list. If you send email newsletters you'll probably like it.