Pastebin

01 · CONTEXT AND REQUIREMENTS

What a pastebin needs to do

A pastebin lets someone paste a block of text and get a link to share it; whoever opens the link sees the text. The design is a cousin of the URL shortener, with one central difference: here the system stores the content of the text, not just the original address.

No scope and requirements of the guidebook include four things: creating the paste and getting the link, opening the link and viewing the content, deleting expired pastes, and tracking visit statistics. The paste does not expire by default, but the person can set a deadline. The user is anonymous, and choosing your own alias is out of scope.

Among the assumptions that shape the design: opening a link must be fast, traffic is uneven, the service must be up almost all the time, and the content is text only. Visit analytics can be monthly and does not need to be real time, which takes it off the critical path.

02 · SCALE ESTIMATION

Back-of-the-envelope math

The guidebook assumes ten million writes and one hundred million reads per month, a ratio of ten reads for every write. Since a month has about two and a half million seconds, that works out to about four writes and forty reads per second, on average.

Storage comes from the paste size. With 1 KB of content per paste, the guidebook arrives at about 12.7 GB of new content per month and 360 million short links in three years. These are values from an example assumption; what matters is the method of starting from the monthly volume and arriving at the per-second rate.

03 · API AND DATA

Metadata in the database, text in the object store

A API has two uses. To create, the client sends the paste content and, optionally, the expiration deadline; the response returns the short link. To read, the client provides the link and receives the content, the creation date, and the deadline.

No data model, each row of the pastes table stores the short link, which is the primary key, the expiration time, the creation date, and the content path. The primary key creates an index that the database uses to enforce uniqueness. The text itself is not in the row: it lives elsewhere, and the row points to it.

Separating metadata and content is a choice of database selection: the database answers small key lookups quickly, and the object store is better suited to text blobs than table rows.

1. The browser sends the text; the web server forwards it to the write API.

2. The write API generates the unique link and checks in the database whether it already exists.

3. With the link free, it writes the metadata row to the database and the text to the object store, and returns the link.

4. Later, whoever opens the link lands on the read API, going through the web server.

5. The read API looks up the link in the database; if found, it fetches the text from the object store; otherwise, it returns an error.

04 · GENERATING THE UNIQUE LINK

Hash, base62, and the duplicate check

The guidebook generates the link in two steps. First it applies MD5, a hash function that produces 128 bits, to the user's IP plus the time; a variant uses random data instead. Then it encodes the result in base62 and keeps the first seven characters.

Base62 uses letters and digits, which works well in URLs without needing escaping. Base64 is another popular encoding, but it brings extra characters that get in the way in URLs. With seven characters there are 62 to the power of 7 combinations, which the guidebook considers enough for the 360 million links in three years. The encoding is deterministic: the same input always gives the same link.

Cutting the hash down to a few characters has a price: two distinct pastes may end up with the same link. That is why the write API does not trust the hash alone. It queries the database for duplicates and, if the link already exists, generates another one; the primary key reinforces uniqueness.

05 · EXPIRATION AND ANALYTICS

TTL, cleanup, and statistics

The TTL, the paste's time to live, becomes two columns: the creation date and the deadline in minutes. The expiration time is the sum of the two. The default is not to expire, and the deadline is optional.

To delete what has expired, the guidebook proposes scanning the database for entries whose expiration time is in the past and then deleting them or marking them as expired.

1. The routine queries the database for pastes whose expiration time has passed.

2. The expired rows are deleted or marked as expired.

3. As an extension of the design, the corresponding text is also removed from the object store.

For the statistics, since real-time analytics is not a requirement, it is enough to batch-process the web server logs to generate access counts per period. This result can go to a data warehouse, away from the read path.

06 · SCALING

Cache, replicas, and what still stays on the primary

The guidebook starts from the basic design and goes after the bottlenecks. For reads, traffic for popular content should be served by an in-memory cache which also helps with uneven traffic and spikes. Read replicas handle cache misses, as long as write replication does not overload them.

For writes, the average rate is feasible for a single SQL primary with a standby; if that is not enough, other SQL scaling patterns come in. The object store handles the monthly volume of new content with room to spare.

1. A read request arrives at the read API.

2. If the paste is popular, the in-memory cache responds and the database is not even queried.

3. If the cache misses, the API queries a read replica and fetches the text from the object store.

4. Writes go to the primary, which replicates to the replicas; too many writes in replication can delay them.