AWS S3: Buckets, Security, Pre-signed URLs & Lifecycle

Q: Why is S3 "not a filesystem"?

There are no real directories — keys like reports/2026/report.pdf just look hierarchical; the slashes are part of the key string. You can't append to an object in place (you overwrite it), and operations are per-object. Treat it as a flat key→object store with prefix-based listing, not a mounted disk.

Q: Should I use a bucket policy or IAM policy?

Both, for different jobs. IAM policies define what your identities (users, roles) can do across AWS. Bucket policies attach to the bucket and govern access to it specifically (including cross-account or conditional access). For most app access, IAM roles are the primary tool; reach for bucket policies for bucket-level rules.

AWS S3

Amazon S3 (Simple Storage Service) is object storage for files, images, backups, logs, and static assets. It's extremely durable and scales effectively to any volume — but it's object storage, not a filesystem, and using it safely means understanding its access model. Misconfigured S3 buckets are one of the most common sources of public data leaks.

The core idea: you store objects (a file plus metadata) under string keys in buckets, and you control access through IAM and bucket policies. Get the security model right and the rest follows.

TL;DR

S3 stores objects in buckets — it's not a filesystem.
Keep buckets private by default; block public access.
Use pre-signed URLs for client uploads/downloads.
Enable versioning and lifecycle policies for safety and cost control.

Quick Example

Upload and fetch an object with the CLI:

Core Concepts

Bucket — the top-level container (globally unique name).
Object — a file plus its metadata.
Key — the object's identifier, a path-like string (reports/2026/report.pdf).

S3 is strongly consistent for new puts and deletes, but design with distributed behavior in mind for high-scale listing and write patterns.

Security Best Practices

Block public access unless you explicitly need public objects.
Use IAM roles/policies with least privilege; prefer short-lived credentials.
IAM policies control what identities can do; bucket policies allow/restrict at the bucket level — combine them carefully and test every access path.

Encryption

At rest — enable default encryption (SSE-S3 or SSE-KMS).
In transit — require HTTPS.

Pre-signed URLs

Let clients upload/download directly without exposing AWS keys:

Client requests an upload URL from your backend.
Backend returns a time-limited pre-signed URL.
Client uploads directly to S3.
Backend records the object metadata.

See File Upload.

Versioning & Lifecycle

Versioning lets you recover from accidental deletes/overwrites.
Lifecycle policies expire old objects or transition them to cheaper storage classes (Standard → IA → Glacier).
For compliance, add object retention policies; for disaster recovery, add cross-region replication when required.

Events & Integrations

S3 can emit events (e.g. s3:ObjectCreated) to trigger downstream processing. A common pipeline:

This pairs naturally with AWS Lambda and Message Queues.

Performance Tips

Use multipart uploads for large files (and resumability).
Avoid a single hot key prefix when writing at massive scale.
Put a CDN (CloudFront) in front of public assets.

Best Practices

Block Public Access on by default — opt into public only per-object via CloudFront where needed.
Encrypt at rest and require HTTPS.
Pre-signed URLs for browser uploads — never ship AWS credentials to clients.
Versioning + lifecycle for safety and cost.
Configure CORS explicitly for browser uploads.

Common Mistakes

Accidentally public bucket

Missing CORS for browser uploads

FAQ

Why is S3 "not a filesystem"?

There are no real directories — keys like reports/2026/report.pdf just look hierarchical; the slashes are part of the key string. You can't append to an object in place (you overwrite it), and operations are per-object. Treat it as a flat key→object store with prefix-based listing, not a mounted disk.

How do I let users upload files directly to S3?

Use pre-signed URLs: your backend generates a short-lived URL granting permission for one specific upload, the client PUTs the file straight to S3, then notifies your backend to record metadata. This keeps AWS credentials server-side and offloads bandwidth from your servers. Remember to configure CORS.

Should I use a bucket policy or IAM policy?

Both, for different jobs. IAM policies define what your identities (users, roles) can do across AWS. Bucket policies attach to the bucket and govern access to it specifically (including cross-account or conditional access). For most app access, IAM roles are the primary tool; reach for bucket policies for bucket-level rules.

How do I control S3 storage costs?

Use lifecycle policies to transition infrequently accessed objects to cheaper tiers (S3-IA, Glacier) and expire obsolete data. Enable Intelligent-Tiering for unpredictable access patterns, clean up incomplete multipart uploads, and watch data-transfer/egress charges — fronting public assets with CloudFront often reduces both cost and latency.