File sync (Google Drive/Dropbox)

01 · CONTEXT AND REQUIREMENTS

What a file service needs to do

A cloud file service, in the style of Dropbox or Google Drive, stores each person's files and makes them accessible from any device. The starting point is the requirements gathering. The core functional requirements are uploading a file, downloading it, sharing it with other people, and keeping files synchronized across devices. Editing files and viewing them without downloading are out of scope for this analysis.

Among the non-functional requirements, three drive the design: high availability, with availability prioritized over consistency; support for very large files, up to 50 GB in the source; and uploads, downloads, and sync that are as fast as possible. It must also be possible to recover lost or corrupted files.

Three entities are enough: the file itself (the bytes), the file metadata (name, size, type, who uploaded it), and the user. Separating bytes from metadata guides the rest of the design, because each one calls for a different kind of storage.

02 · UPLOAD AND DOWNLOAD

Getting the bytes out of the backend's path

The first idea is to send the file to the application server and write it to disk. That works for a small application, but it does not scale and is not reliable: storage grows along with the files, and if the server goes down, access to all of them is lost. Object storage solves both problems, since it holds a virtually unlimited number of files and handles scaling on its own.

Passing the file through the backend to object storage still sends the bytes twice. The way out is for the client to upload directly, using a presigned URL, which grants permission to write to a specific location. The file service becomes a control plane: it generates the URL, records the metadata, and updates the state when the storage notifies that the upload has finished. One case remains to handle: a file uploaded without its metadata saved, and the reverse.

1. The client asks the file service for a signed URL to upload. The service records the file metadata with the upload still in progress.

2. The client sends the bytes directly to object storage, without going through the backend.

3. The storage notifies the service that the upload has finished, and the service marks the file as uploaded in the metadata.

4. On download, the client requests the URL from the service, which checks in the metadata whether the person has access and returns a signed CDN URL.

5. On the first request, the CDN fetches the file from storage; on later ones it serves the copy closest to whoever is downloading.

The same logic applies to download: downloading from storage to the backend and then to the client is two transfers, slow and expensive. The client should download directly. For people far from the storage region, a CDN stores copies of the files on servers spread around the world and serves from the closest one.

03 · METADATA AND SHARING

Who can see what

The metadata is loosely structured, has few relations, and the main query is by user, which fits a NoSQL database. The source notes that a SQL database would work just as well, so the choice matters less than it seems.

For sharing, the first impulse is to put the list of who has access in each file's metadata. Listing a user's files is easy, but listing the files shared with them is slow, because it requires looking at each file's list. A cached inverted index speeds up the query, but forces you to keep two copies of the same information in sync.

The alternative is to normalize: a shares table that links user to file, with the user as the partition key and the file as the sort key. The files shared with a person come out of a single query, and there is no copy to keep in sync. The cost is a query slightly less efficient than a direct lookup.

04 · SYNC

Two directions and one source of truth

Each device keeps a local copy and the server keeps the remote copy, so there are two directions of sync. The remote server is the source of truth, and so it needs to become consistent as soon as possible so the other devices know what to pull. In the local-to-remote direction, an agent on the client watches the folder for file system events, queues what changed, and sends it through the upload API.

In the remote-to-local direction, polling is simple, but slow to detect changes and wastes bandwidth when nothing has changed. Open WebSocket or SSE connections give real time, but they can drop and lose messages. The hybrid solution uses one connection per device, not per file, to push notifications, and periodically queries for changes since the last timestamp as a safety net.

1. The agent on device A notices that a file in the local folder changed and queues the upload.

2. Device A sends the content to storage and reports the change to the file service.

3. The service records the change in the metadata: which file, what kind of change, and the updated data.

4. The service notifies device B over the connection it keeps open.

5. Device B queries what changed and downloads the new content. If the notification were lost, the periodic query would catch the change anyway.

If two users edit the same file, the source resolves the conflict with last write wins: the most recent edit is the one that stays. On versions, the source observes that you normally do not overwrite the only file: you add a new one and update a version pointer.

05 · LARGE FILES

Chunks, fingerprint, and resume

Uploading a large file in a single request runs into limits. With a 50 GB file and a 100 Mbps connection, that is about 4000 seconds, more than an hour with no response. The API Gateway used in the design accepts only 10 MB per request. On top of that, a network drop forces you to start over from scratch, and the person sees no progress.

The solution is chunking: split the file into pieces and send them one by one or in parallel. The split must be done on the client; splitting on the server defeats the purpose, since the whole file would already have been sent. Typical chunks are 5 to 10 MB. To resume, the state of each chunk is kept in the file's metadata record.

What remains is deciding who updates that state. Letting the client report on its own is risky, because the server ends up trusting something the client can fake. It is better for the client to report each chunk's ETag and for the backend to verify it with the storage's ListParts API before marking the chunk as uploaded: trust, but verify.

To know whether an upload has already been attempted, the file name will not do. You use a fingerprint computed over the content, for the whole file and for each chunk. It lets you detect duplicates and resume an upload in progress. Because it identifies the content and not the record, the fileId remains a separate unique identifier.

1. The client splits the file into chunks and computes the fingerprint of each one and of the whole file.

2. The client asks the service whether an upload with this fingerprint already exists. If it does and is in progress, it resumes from the state of the chunks.

3. If it does not exist, the service starts the multipart upload in storage, generates a URL for each chunk, and writes the metadata with the upload in progress.

4. The client sends the chunks directly to storage, in parallel, and reports each one's ETag to the service.

5. The service checks the parts against storage before marking each chunk as uploaded in the metadata.

6. With all chunks marked, the service asks storage to assemble the final object and only then marks the file as uploaded.

Download does not need chunks: after assembly, storage has a single object, and the client downloads through a normal URL. For very large files, Range requests let you download byte ranges in parallel or resume an interrupted download.

06 · SPEED AND SECURITY

Fewer bytes, protected bytes

With chunks, sync only needs to send the ones that changed. But with fixed-size chunks, inserting a single byte near the start shifts all the following boundaries and changes the fingerprint of every chunk after the edit, which makes delta sync nearly useless. Content-defined chunking solves this: the boundaries come from a rolling hash over the content, and a small edit affects only the surrounding chunks.

Compression, done on the client, only pays off if the gain in transfer exceeds the cost of compressing and decompressing. On already-compressed media the gain is very small; on text it is usually larger. When encryption is also involved, compress first, because encryption introduces randomness and hurts compression.

On security, data goes over HTTPS in transit and is encrypted at rest in storage. Access control comes from the shares table. For the case where someone passes along a download link, signed URLs are valid for only a short time, about five minutes in the source. But they are bearer tokens: anyone with a valid URL downloads the file. The short expiry limits exposure without eliminating it, and extra restrictions, such as binding the URL to an IP, harden access.