Open a web series that you love on a shaky train connection. The picture goes a little soft for a few seconds, sharpens again, and the video never stops. Nobody thinks about that. I didn't either, until my team had to build it.

We'd been using Cloudflare Stream, which does all of this behind one upload button. When we moved to Google Cloud, that button became my problem.

This blog has two halves. First, the few ideas behind video streaming, with no background needed. Then, how we built it and what went wrong. Once you have the ideas, the architecture will feel obvious.

Why you can't just serve the MP4

Say an instructor uploads a one-hour lecture, a 2 GB MP4 file. The simplest plan is to hand that file to the browser. Two things go wrong.

The first is a quirk of the format. An MP4 has an index that tells the player where everything is, and many tools write it at the very end of the file. Imagine a book with its contents page on the last page, delivered to you one page at a time. The browser may need the whole file before it can play the first second.

The second is bigger: everyone gets the same file. A learner on office Wi-Fi and a learner on a train both get full quality, and the one on the train watches a spinner. You pay for all that data too. At 1,000 learners watching 15 hours a month, our rough estimate was 18 TB of data, every month.

Everything below exists to solve that second problem.

Transcoding: making several copies

Transcoding means re-encoding a video into new sizes and qualities. Think of saving one photo as small, medium, and large.

The number that matters is bitrate, how much data each second of video uses. Higher means sharper, and heavier to download. Here's the set we make from each upload:

360p     0.7 Mbps    small, survives a bad network
480p     1.2 Mbps
720p     2.5 Mbps
1080p    4.5 Mbps    sharp, needs a good network

This set is called a ladder, and each line is a rung.

HLS: cutting the video into pieces

Four copies don't help much if switching means starting the lecture again. So we cut every copy into small pieces, a few seconds long.

HLS (HTTP Live Streaming) is the format for this. It adds a small text file called a playlist, ending in .m3u8, that works like a restaurant menu: "this video comes in 360p, 480p, 720p and 1080p, and here's where to find the pieces of each."

The browser never downloads "the video". It reads the menu, then downloads one small piece after another.

So a single lecture becomes hundreds of small files. Remember that. It comes back later.

ABR: the player changes gear

ABR (adaptive bitrate) is what the player does with all those pieces, and it works like an automatic gearbox.

The player downloads a piece and times it. Came quickly? The next piece can come from a higher rung. Took too long? Drop down a rung. It decides again for every piece, so when your train enters a tunnel, the picture softens instead of stopping.

That's the moment from the first paragraph. HLS is the format. ABR is the behaviour it makes possible.

CDN: warehouses near your customers

Imagine an online shop with one warehouse in one city. Every order crosses the country, slowly and expensively. So the shop opens local warehouses. The first customer in your city waits while an item comes from the main warehouse; the local one keeps a copy, and the next customer gets it almost immediately.

A CDN (content delivery network) does this with files. Its servers around the world are the local warehouses, and our storage is the main one, called the origin. When the second learner watches a lecture, the pieces come from a nearby copy, which is faster for them and cheaper for us.

Signed URLs and cookies: who gets in

Course videos aren't public, so every request needs some proof of permission. There are two common kinds, and the difference is easiest to picture at a theme park.

A signed URL is a ticket for one ride: a link that opens one file and expires. It can't be altered without breaking it.

A signed cookie is a wristband. You get it once at the gate, it's valid for one area until closing time, and the browser automatically shows it on every request.

With hundreds of pieces per video, you can see which one we'll want.

That's all the vocabulary

Here it is in one breath: we transcode a video into several qualities, HLS cuts them into pieces with a menu, the player uses ABR to choose each piece, a CDN delivers it from nearby, and a signed cookie decides who gets in.

If that sentence makes sense, the rest of this post will too.

What I built

Let's follow one lecture through the system. It has three stages:

1. UPLOAD    Instructor's browser --> private cloud storage
 
2. PROCESS   "Upload finished" --> our API --> processing job
               - makes a playable MP4 (watchable from here on)
               - asks Google to transcode --> HLS pieces
               - asks Google to transcribe --> captions
 
3. WATCH     Learner --> our API: allowed? --> signed cookie
             Player --> CDN --> video pieces

Stage 1: Upload

A 2 GB file shouldn't pass through our servers, so the browser sends it straight to Cloud Storage using a resumable upload. If the connection drops, it carries on from where it stopped. Our API only creates a database record for the video and hands the browser a one-time upload link.

There's one catch. The bytes never touch our servers, so if a browser claims "this file is 100 MB", we can't check that at the start. The first honest number arrives when the upload finishes. That's where we enforce the 2 GB limit, deleting anything oversized before spending money on processing it.

Stage 2: Process

When the upload finishes, Google sends our API an event through Eventarc or Pub/Sub. Here's the first lesson every backend engineer learns about events: they can arrive twice. If each copy started processing, we'd pay to do the work twice.

Checking the status first doesn't help, because two copies can check at the same moment and both see "not started". So the update itself does the checking:

UPDATE course_videos
SET upload_status = 'PROCESSING'
WHERE id = :id AND upload_status IN ('PENDING', 'UPLOADING');

The database lets only one caller change that row. If zero rows changed, this event was a duplicate and it stops there.

The winner starts a Cloud Run job, a container that handles one video and then exits. It uses ffmpeg, the standard command-line video tool, and it starts by fixing the contents-page problem from earlier:

ffmpeg -i input.mp4 -c copy -movflags +faststart playable.mp4

+faststart moves the index to the front of the file, and -c copy means the video itself isn't re-encoded, so this takes seconds and loses no quality. The video is now watchable, so we mark it READY. It also grabs a poster image from 5 seconds in, since the first frame is usually black.

Everything else happens in the background, through two Google APIs:

• Transcoder API builds the ladder and the HLS pieces. We skip rungs above the original's quality, because upscaling 720p to 1080p costs money and adds no detail.

• Speech-to-Text v2 creates the captions. We pull out just the audio, send it for batch transcription, and turn the resulting word timings into a WebVTT file, the subtitle format browsers understand. Its dynamic batching mode is slow but cheap, about 16 times cheaper than the Video Intelligence API by our numbers at the time.

Both of these run after the video is already watchable. Playback never waits for captions. A slow transcription should never make a lecture unwatchable.

One small design choice saved us from a classic disaster. Uploads and processed files live in separate storage buckets, and the job only reads from one and writes to the other. Its own output can't trigger another run, so it can't loop forever.

Stage 3: Watch

A learner presses play. Our API checks they're allowed to watch (the course's creator, or enrolled, for example), then signs a cookie that covers only that video's folder and lasts a few hours. That's the wristband. The player, built with Video.js by a teammate, reads the menu and starts pulling pieces through Cloud CDN, which checks the wristband on every request.

Because a re-upload always creates a new video ID, the files never change, so the CDN can keep its copies for a long time.

The decision that surprised me

My first plan had no CDN. One MP4, a signed URL, done. A CDN needs a load balancer with a fixed monthly cost, and at development traffic it would have quadrupled our bill to save pennies.

Three days later the requirements changed: adaptive streaming now, and no downloads for anyone.

ABR means hundreds of pieces, and a one-ride ticket can't cover hundreds of pieces whose names you don't sign in advance. There are workarounds, but the wristband was far simpler. Cloud Storage doesn't offer signed cookies, and Cloud CDN does.

So we added the CDN mainly for security. Speed turned out to be the bonus.

What broke

The CDN that couldn't open its own warehouse. The CDN reads private storage using its own Google-managed account, and our organisation's security policy refused to give that account access. The browser just showed AccessDenied, which looks exactly like a bad cookie, so we went looking for a signing bug that didn't exist.

What cracked it was deliberately sending a wrong key. A wrong key got rejected by the CDN with an HTML page, while a correct key got an XML error from storage itself. Different responses proved the cookie was fine and the problem sat further down.

While the policy question was being sorted out with admins, we shipped a switch: one setting served a plain MP4 without the CDN, the other served HLS through it. Videos kept working the whole time, and we flipped it once access was approved.

The transcode that succeeded and produced nothing. Google's Transcoder also runs under its own account, and that account had no storage access. Our job reported success, because all it had done was submit the work. Nothing appeared, and none of our logs mentioned it.

The request that worked and still failed. Browsers have a safety rule: a response to a cookie-carrying request from another domain is thrown away unless it includes a specific permission header, and Cloud Storage doesn't send that one. The network tab showed a 200 with the right data, and the player still failed. We added the header at the load balancer.

What I'd tell another backend engineer

• Separate "playable" from "perfect". Ours is watchable right after that one quick ffmpeg fix, and everything else catches up.

• Assume every event can arrive twice, and plan for one that never arrives.

• Cloud services often run under their own accounts. When a call succeeds and nothing appears, check those permissions first.

• Look at who actually produced an error. The same AccessDenied can come from very different places.

Next time a video softens for a moment on a bad connection and then carries on, you'll know what happened. A player wearing a wristband read a menu, timed its last download, and picked a smaller piece from a server somewhere nearby.

Before this project, I'd have called video "just file upload with a bigger file". I wouldn't now.