The tubes are in parallel

This is the third time I’ve read End-to-end Arguments in System Design (Saltzer, Reed & Clark) and the paper has gotten better every time. The main takeaway for me is that whatever feature you might want to network to offer, it’s usually much better to implement it at the end-point application level than the network level. Ensuring FIFO delivery or ensuring all packets arrive is nice but such features are trouble when all you care about is getting most of the packets there quickly (like in voice calls). The end-points can take care of packet ordering and data integrity by using check sums/hashes of the data and if the application requires both perfect accuracy and timeliness, then network level “solutions” won’t work either because some things can’t be done (networks can get faster and more accurate, but as long as there are errors and the speed of light there will always be a trade-off between speed and reliability). Further, because end-point applications will have to assume some error rate, it will always have to check for errors anyway. The conclusion here is that having the network do more will get in the way of applications that don’t require those features because features inevitably include trade-offs.

I think BitTorrent is a good example of doing things the right way. BitTorrent divides files into “pieces” and checks for errors on each individual “piece”; it doesn’t re-download the whole file because one bit was dropped. It does this in part because all the pieces come from different places, and so it’s important to know that you are getting the right data and not something completely different (when downloading from a single source you may not get what you wanted either, but the only fixable issues are errors, because downloading the wrong file entirely is more a question about life-choices that has nothing to do with networks). I think BitTorrent is a particularly clever protocol because it embraces the weakness of the internet and provides an end-point solution. By weaknesses I mean bottlenecks that happen when a lot of traffic is going to one server (e.g. for downloads), both for the server itself and for its connection to the internet as well as the inevitable reliability issues. There are no bottlenecks when the tubes are all in parallel.

Instead of relying on someone else to fix the system, BitTorrent fixes the problems itself. Organizations that advocate treating packets differently based on what they are might gain from taking this sort of approach. Of course, the subversive nature of BitTorrent might convince them otherwise. Oh well.

Leave a Comment

Log in