Our video monitoring evolution

TL;DR: Cribs stream over WebRTC straight to the phones on the home networks, and go via Janus servers over the internet. We got something reliable after a few iterations. When the bills got high (egress $$$), we prioritized peer to peer whenever possible.

We’re going to go chronologically through different phases of our video monitoring implementation, with emphasis on reliability, cost, and ease of development. We’ll highlight the hunches and bets we made, and the lessons we learned along the way. We’ll use some cake analogies, and no more Star Wars references. Without further ado, let’s dive in.

Figuring out the recipe

These were the early days. We had a few crib prototypes ready to share with the world (a few miles’ radius because we were delivering them ourselves). The use case was clear: get iOS and Android apps to stream the crib’s audio/video feed. No one in our team had prior video streaming experience. We did our research and started developing hunches. One thing felt obvious to us early on, and we still lean on it today:

Parents and nannies are the primary users and they’re going to be at home with the baby most of the time. For them, it made sense to stream entirely over the local network. Relying on the internet here would be an unnecessary waste of bandwidth, and open us up to slow or unstable connections. We’d assumed that the split between LAN and internet streaming would be around 85:15, or even more in favour of the former. So we tackled video over the LAN first. Internally, we started calling streaming over the LAN “local streaming” and streaming via the internet “remote streaming”. We’d deal with internet streaming later, how hard could it be?

We were undecided on what technology to use, but we were pretty clear on what we wanted, in increasing order of priority: a) low time to first frame, b) low latency, c) it should be cool.

We chose WebRTC after flirting with RTSP, HLS and RTMP. The crib runs a Linux stack, and GStreamer just got a brand new WebRTC plugin! Phones already had reliable PeerConnection libraries. We’d already sorted out MQTT to control the crib’s bounce and music. The secure signalling plane needed by WebRTC was free lunch. We were young and we could make bold bets:

Local streaming turned out to be mostly straightforward. For the next phases, we’ll do a brief check on how it’s doing, but focus more time on remote streaming.

We’re done with our first trimester, metaphorically speaking. Let’s prepare our first report card. We’re going to measure this phase in terms of ease and cost.

Report card: first trimester

Ease

The internet is full of cake recipes, just needed to pick the cakiest one.

Cost

What’s that?

Attempting our first bake

How’s local streaming going?: OK-ish? Fails pretty often, but WebRTC is solid and zippy. The bugs lie in our software: the phone-crib discovery, signalling race conditions, etc. We got the ingredients right, we’re just terrible at baking at the moment.

With pilot customers round the corner, we started work on our first internet streaming solution.

We wanted to ‘Get it right the first time’ ™: we wanted it to be stable, scalable and extensible. We wanted it to be as mouth watering as the cake we saw on the display at that bakery store.

Some things stood out for us:

1. A crib’s stream was consumed by anywhere between 1 to a dozen users. Grandparents, guardians etc. needed internet streaming. Essentially, a 1:N use case, N being small.
2. We wanted the option of transcoding, overlaying stuff on the video, sprinkles and other toppings.
3. We were still stabilizing the crib software, we didn’t want to push it too hard with a high uplink. We also didn’t want to surprise users with an expensive internet bill.

We were leaning towards using an intermediate server for internet streaming. Sure, WebRTC’s greatest strength is to get two peers to talk over the internet. Shouldn’t we just set up some STUN and TURN servers and roll that out? We had our doubts at the time. Time to go by hunches and vibes again:

Well, it’s half hunch-half bet, like how crab cake is half crab and half cake.

The next thing was certainly a bet, and whoa, what a bet it was:

We played with popular paid stuff at the time like Wowza, AWS offerings. They were great, a bit black-box-ish for our liking, and still a bit behind when it came to WebRTC. We explored open source WebRTC servers like Janus and Kurento. We really liked Janus (technically and philosophically!). Granted, we’d need to put some more work in if we did our own thing rather than a paid service, but this was a good opportunity to develop some video expertise. Also, why should we pay for something when we could do it ourselves? So, we decided to:

We picked Janus. Janus is a Selective Forwarding Unit (SFU). It’s lightweight, with a plugin based design. It didn’t have transcoding or intermediate processing out of the box, but you could make a plugin for it.

Janus has a streaming plugin, which seemed a good fit for the 1:N use case we were looking for. One side/face of Janus receives SRTP streams from cribs. The other side serves each stream as a mount point, which the phone can view. It was up to us to define these streams and reserve ports for them. Janus took care of serving them as WebRTC streams, to as many viewers as needed.

The cake is ready to be served. Janus is our server. We cut the cake into slices, one per crib. Janus handles the delivery of these slices to the intended recipients. It ensures deliciousness is kept intact.

There was a fancier video room plugin that Janus offered, which was meant more for multi-party calls. It seemed tempting to us because it would enable talk-back, but it seemed like overkill at the time.

“Woohoo!” We patted ourselves on the back. We had a pretty functional and zippy internet streaming solution working. We were serving a few hundred users, and the number was growing every day. Things were taking shape. The world was our oyster, or crab cake. This was our second trimester report card:

Report card: second trimester

Ease

Put the ingredients in a blender, give it a swirl. There’s a weird aftertaste, though.

Cost

AWS gave us free credits. Who cares?

The oven test

How’s cousin LAN doing?: Good. When it works, it works great. When it doesn’t, it doesn’t at all. We realized users can have complex network topologies at their home. They can have firewalls, shared Wi-Fi networks in apartments. Those can block non-standard ports, even WebRTC can’t do much if you block all UDP ports. For all such cases, our apps had to fall back to remote. The other pain point was audio, we thought it would just tag along with video, but it had its own challenges: why is it playing on the earpiece, why is it on the speaker, why is it muting this other app, why is it not muting this other app? WebRTC audio behaves differently when you treat it as call vs media, and getting that right took us a while.

Internet streaming was working well, but we soon realized that the Janus streaming plugin wasn’t the right fit for us. The plugin left the responsibility of port allocation to the user (us in this case). It made more sense for use cases where the set of streams changes on a deployment timescale, and not user-behaviour. For example, it works great for a collection of IP cameras in an organization, or live broadcasts. In such cases, the streams can change, but not that often, and there’s someone trustworthy to manage the changes. In other words: static good, dynamic bad.

In hindsight, our use case was never static at all. The phones (i.e., users) decide when the streams start and stop, and that’s a chaotic dance of streams starting and ending. Things were fine when we had a few hundred customers, we just appended the next new user to our relatively small list. But as the number grew, we had to do something or we would run out of ports on one instance, then the next instance, and so on. We considered a dynamic allocation strategy: When a crib needs to stream, allocate a <server, port> pair for it, when it’s done, release the pair, and let someone else use it. This would work great if each actor in the system was reliable. But what happens if the crib lies to us? What if it says it has stopped streaming, but hasn’t? There were many ways this could go wrong. It became clear to us that we could only create a best-effort solution here, not a foolproof one. Cutting the cake ourselves was way harder than we imagined. This would only work if Janus managed the streams, as it’s the real source of truth. It knows when a WebRTC session really stops, its resources are released, etc.

Some lessons were learned. The first one was in software design. There’s a reason why we try to have layers in software, and in cake. We tried to own a piece of logic (port allocation) which we would never be able to do a good job at. If we don’t layer well, the cake won’t bake well. Crabs, on the other hand, have a shell, which with some imagination can be thought of as a layer. But let’s keep our focus on the dessert class of cakes, not the crab kinds. The lesson here was:

The next lesson was around our choice of the Janus streaming plugin. We chose it because it fit one aspect of our use case really well: the 1:N feed-to-viewers mapping. We didn’t dig deep enough to see that it would be hard to manage streams when they are chaotically created and destroyed. The plugin supports it on paper, but it doesn’t promise anywhere that it will be easy. The lesson learned was that we should cover all our bases when making a design choice:

Report card: third trimester

Ease

Umm, okay, a bit soggy in some places. Might need some more baking.

Cost

We’ll deal with that later! We need to re-architect our internet streaming solution!

The cake’s out of the bag

How’s local streaming holding up?: Much better! The crib and app software is mature now, it’s served many many hours of streaming. Long running sessions still break sometimes, apps drain the battery. But expectations are now high too: our goal was no breaks whatsoever.

Internet video required a massive overhaul at this scale. We were already good at Janus. As mentioned earlier, there was a more advanced Janus plugin called video room. With this design, each crib gets a virtual room, which the crib and its phones join. The crib publishes the stream to the room, and the phones subscribe to it. It’s WebRTC everywhere. No static port allocation anymore: Janus manages the ports, based on ground truths (is the crib/phone still connected via WebSockets/ICE?). We let Janus do the cake slicing since it’s much better at it.

This wouldn’t scale by itself. A single server could only handle so many video streams. Adding more server instances, but tying them to certain cribs would be a sort of static allocation in its own way. Thankfully, the Janus creators had thought of this too (SFU cascading), and gave us the pieces to implement a scalable solution. We came up with an autoscaling cluster of virtual instances, each running Janus. Room state is replicated across the cluster. Through some stream forwarding trickery provided by Janus, a crib and a phone could join different instances as decided by the autoscaling load balancer. We have orchestrator software that makes sure the app receives the stream.

We slowly migrated our cribs to this new architecture, and it worked well. The number of moving parts in the system was now 10-fold, race conditions still occurred, but they were short lived and the next app retry generally worked. It seemed like we’d done the hard part, and the only thing that was left was the gradual fine tuning and fixing that would get us to the reliability at scale we were looking for.

Time to move on to other tasks, right? … Right?

This was the time when we learned a new word: Egress.

Outbound data from servers costs money. $0.0x/GB looked low enough to not notice in our bills when we started, but as we scaled, the number rose faster than we expected: Our original assumption, the 85:15 LAN-internet split, seemed grounded in reality. Most users, after all, would stream from their home, but the reality we saw was 75:25. This was due to several reasons:

1. The home network: The phone and crib need to be on the same LAN, but that doesn’t always happen. A mesh system running as a second router behind the ISP’s box, a guest or IoT network, a router that walls Wi-Fi devices off from each other, or a deliberate firewall or VLAN all leave the phone unable to see the crib.
2. Phone’s privacy controls: iOS asks before an app may talk to devices on your home network; if that’s declined, the crib becomes invisible to the app and nothing on screen says why. VPNs and rotating private Wi-Fi addresses can also route traffic away from the home network or keep changing the phone’s address.
3. Our own software quirks: Our phone-crib discovery can end up being flaky at times, the crib could get assigned new IPs often by the router. Sometimes the crib itself tries to hop between different saved networks to find a better signal. These are things that we’ve been fixing of late.

We questioned our earlier bets and hunches again. Did it really make sense to do all this on our own? Should we have just gone ahead with a paid service? Would a different technology, like HLS, have reduced costs? The answers varied a bit, but all of them leaned towards a ‘No’. You’ve got to pay the money. There are some tricks to bring down the costs, which we will discuss soon.

Our video monitoring baby/cake is now fully baked and out in the world. It still needs coddling, but it’s got some agency now. It crawls around now and throws up sometimes. It smells like seafood sometimes, but don’t all babies? It just burped, aww how adorable!

Report card: post-birthday

Ease

The cake came out delicious, but don’t ask me to make 10 of them at the same time.

Cost

Wait, why does my AWS bill have 2 extra zeros?

Fine tuning our recipe

We’re now close to the present day.

How’s our good ol’ local streaming doing?: We’re ‘close’ to being 100% reliable, and this is for 75-80% of all the video sessions. Not bad! The cribs that struggle, struggle a lot, though. Sometimes it’s the user’s Wi-Fi, other times it is our Wi-Fi stack misbehaving with certain Access Points, sometimes it’s the crib not latching to the best Access Point, or not picking up the best and least noisy Wi-Fi band. There are some things that are in our control, and we continue working towards these.

We’ve been improving our Janus video room solution: We have AI now to smoothen out our testing and deployment by preparing scripts that we were too lazy or busy to prepare earlier. We’re grinding our teeth less while making new releases.

Choosing Janus was a good call. It’s an active community of great developers. We’d found a bug early on, we shared the logs and they were able to fix it quickly. When we had questions, we just had to post them on the forum and someone got back. Trusting Janus and open source software was a good bet.

Getting internet streaming to 99.9% reliability, as with local streaming, is getting increasingly hard. It’s still got a lot of moving pieces, we’re merging the pieces where possible, and making each piece more robust. But it still feels like we’re fixing a leaky balloon filled to the brim with water. You plug one leak, and another one opens somewhere else. Will another re-architecture from scratch fix it?

Some of our early hunches, i.e., looking at local and internet streaming separately (Hunch #1), and choosing an intermediate streaming server (Hunch #2) made us overlook giving P2P streaming over the internet a fair shot. P2P can offer lower latency and time to first frame, and save us a lot of money. We’re correcting that now. We now have a hybrid internet streaming approach, where the phone attempts both P2P and Janus at the same time. If P2P succeeds, we pause Janus (and hence egress costs). If P2P doesn’t work, which it often doesn’t, because of CGNATs, poorly configured IPv6 networks etc, we go back to Janus. If there are too many remote streamers we go back to Janus. Janus is our SFU, and now a fancy relay server too when P2P doesn’t work.

We’re adding more observability to catch issues early, discover trends etc. It’s awkward when the product team asks us how video monitoring is doing, and we only have to say vague stuff like ‘good’, or ‘better than yesterday’. It’s easy to not give it enough attention when you have so many things to do. You’ll have to do it someday, and earlier is always better. Thanks to our product and data teams, we now have a lot of stats on how our video is doing, what’s breaking, which users are facing issues the most. It deserves its own post. It’s strange to see your baking journey reduced to numbers and graphs, but hey, if it gives the cake a better chance to get into a good university, and flour-ish in the real world, then why not?

Yaay! Our video monitoring solution can stand up on its own legs, still eating things it shouldn’t eat, throwing tantrums when you don’t give it attention, but it just framed its first sentence ‘Peer Connection Failed’ awwww.

Report card: toddler

Ease

We’re a bit tired now, but we can call ourselves bakers for the first time.

Cost

Let’s avoid going broke.

The cake is a lie

How’s local streaming doing? And what’s up with remote streaming?: Tracking these as separate metrics was not the right way to go about it. Video should just work for the end user. It should be zippy and it should be shiny. We’re bored of cake analogies. A few of them didn’t even make sense. It’s made us very crabby.

What’s next: Even more emphasis on reliability and security, adding features that users ask for.

We can’t predict our report card for the future. Sharing it here for the sake of completeness:

Report card: the future

Ease

Does it even matter?

Cost

Does it even matter?

Looking back at some of the early hunches we had:

Neither of these are technically wrong. If they were, we would have no video monitoring. But, they still cost us in a few ways: it took us longer to be more reliable than we’d planned, a lot more development effort and churn, and, of course, literal costs ($$$!).

What was our fault, then??

We didn’t realize when our hunches became bets! We were supposed to go by these, and revisit them after our next set of learnings. Instead of doing that, somewhere in our collective minds, these became truths or rules to live by. We forgot that crab cakes aren’t actually cakes, they’re just a type of fish cake. Wait, they are not cakes. Period. An obvious learning, but one worth sharing here:

We’re nonetheless satisfied with our bake: It’s decent cake. If a cake critic tastes a bit of crab, we’ll say that’s how we always intended it to be.

AI Disclaimer: The text was written by humans; the diagrams and charts were made with AI.