Object Storage & Signed URLs for Uploads/Downloads

Object Storage and Signed Download URLs

Almost every application eventually stores files: avatars, invoices, exports, attachments, videos. Putting them in the database bloats backups, and keeping them on application servers breaks the moment you run two instances or redeploy. Object storage — Amazon S3, Google Cloud Storage, Azure Blob Storage, Cloudflare R2, and S3-compatible services — is the standard answer: virtually unlimited, durable, cheap storage addressed by key over HTTP.

The design question is how users get files in and out without either making the bucket public or streaming every byte through your servers. Signed URLs solve both: your backend authorizes the request and issues a short-lived URL that lets the client upload or download one specific object directly from storage.

TL;DR

Keep buckets private; never make user content publicly listable or readable by default.
Store metadata (owner, tenant, original name, status) in your database and generate object keys on the server.
Presigned upload URLs or POST policies let clients upload directly, with limits on size, type, and expiry.
Presigned download URLs let authorized users fetch files directly — keep them short-lived.
Validate after upload: check real file type and size, scan for malware, then mark files available.
Serve downloads with Content-Disposition: attachment and from a separate domain to avoid cross-site scripting through uploaded content.

Quick Example

A backend using boto3 that issues a constrained upload policy and a short-lived download URL:

The browser uploads straight to storage using the returned policy:

Core Concepts

Objects, Keys, and Buckets

Object storage holds objects — bytes plus metadata — under keys in buckets. There are no real directories; prefixes such as tenants/42/uploads/ simply organize keys. Objects are replaced, not edited in place. Amazon S3 provides strong read-after-write consistency, and other major services offer similar guarantees for single objects.

Signed URLs

A signed URL embeds the operation, bucket, key, expiry, and a cryptographic signature made with the backend's credentials. Storage verifies the signature and expiry and allows exactly that operation. Anyone holding the URL can use it until it expires, so expiry is your main control — individual URLs cannot be revoked without rotating the signing credentials.

Upload Patterns

Browser uploads require a CORS configuration on the bucket that allows your app's origin, the method used, and the needed headers.

The Upload Lifecycle

The client asks your API to start an upload; the API authorizes, validates the declared type and size, generates a key, and records a pending file.
The client uploads directly to storage with the signed URL or policy.
A storage event (or a client confirmation) triggers processing: verify size, detect the real type from file contents, scan for malware, generate thumbnails.
The file becomes clean and visible — or rejected and deleted.
Lifecycle rules remove abandoned pending uploads and incomplete multipart uploads.

See File Upload.

Serving Downloads Safely

Authorize every request in your API, then redirect (HTTP 302) to a fresh signed URL or return it to the client.
Use short expiries — seconds to minutes for sensitive files.
Set Content-Disposition: attachment so browsers download rather than render untrusted content, and X-Content-Type-Options: nosniff.
Serve user content from a separate domain (not your main app's origin), so an uploaded HTML or SVG file cannot run scripts with your app's cookies.
For high-traffic public or semi-public files, use CDN signed URLs or signed cookies (for example CloudFront) so edge caches serve content while access stays controlled. See Amazon CloudFront.

Operations and Cost

Block public access at the account or bucket level, and use bucket policies that deny anything not intended.
Versioning protects against accidental overwrites and deletions; pair it with lifecycle rules to expire old versions.
Storage classes move rarely accessed data to cheaper tiers; watch retrieval costs and minimum durations.
Egress is often the largest cost; CDNs and providers with low or no egress fees change the math.
Encryption at rest is on by default for major providers; use customer-managed keys where compliance requires them.

Best Practices

Never Trust Client-Supplied Names or Types

Generate keys on the server, keep the original filename only as sanitized metadata, and determine the real type by inspecting content after upload.

Isolate Tenants

Prefix keys by tenant, check tenant ownership on every request, and consider separate buckets or keys for customers with strict isolation needs. See Multi-Tenant Application Design.

Keep Signing Credentials Narrow

Sign with a role whose permissions cover only the bucket and prefix it needs, so a leaked URL or credential cannot reach other data.

Clean Up Automatically

Lifecycle rules for pending uploads, incomplete multipart uploads, temporary exports, and old versions keep storage costs and exposure down.

Common Mistakes

Public Buckets for Convenience

Long-Lived Signed URLs

Rendering Uploaded Files Inline on the Main Domain

FAQ

What is a presigned URL?

A URL containing a signature and expiry, created with your backend's credentials, that grants temporary permission to perform one operation — such as uploading or downloading a specific object — without making the bucket public or sharing credentials.

How long should a signed URL last?

As short as practical: seconds to a few minutes for downloads of private files, and minutes for uploads. Longer expiries increase the risk if a URL is shared or leaked, because it cannot be individually revoked.

Should uploads go through my server?

Usually not. Direct uploads to object storage with signed URLs or POST policies avoid tying up servers and bandwidth. Route uploads through your server only when you must transform or inspect content during transfer.

How do I restrict upload size and type?

Use presigned POST policies (or equivalent) with content-length ranges and content-type conditions, validate declared values in your API, and verify the actual size and type after upload before making the file available.

Can signed URLs be revoked?

Not individually. They stop working when they expire or when the credentials that signed them are rotated or lose permission. Short expiries and authorization at the moment of issuing are the practical controls.