Did I make the Internet faster? Perhaps I did! The first prototype of HTTP compression

Well, Elon is no longer a piece of news anymore, I rather call him a daily feature :). However, the ‘Inventor of email` piqued my curiosity, because it reminded me of those early internet days when we all were inventors (just like multiple claims on who invented email) :) The internet speed was slow, browsers didn’t support compression, and my need to have faster browsing gave birth to first prototype of HTTP compression in the browser, although mostly met with mixed reactions (like any other new idea), soon it became the standard.

I wasn’t savvy enough to write about it then, nor would I have ever written about it unless this news prompted me. So here is a short article from my memory of those days and how it happened. It may sound like a tall claim, but the internet never forgets and thankfully, my post about this prototype in 1996 on Linux net archive is still live, making it easy to compare dates with compression support release dates in browsers.

Those Days…

1996, I was a student at the Indian Institute of Technology, Bombay. Not many of us could afford internet then, but we were lucky to have access to a 64kbps internet line (ernet.in). It might feel painfully slow by the current standards but it felt like an expressway in those days. This is true compared to the 9.6kbps connection offered by VSNL (for a whopping Rs. 25,000 for 250 hours).

It did feel slow sometimes though, downloading Slackware took over a day. However, browsing was more painful, especially webpages with images that could take a few minutes, we could even go and pick up tea and it was still loading which made me a tea addict :). This was especially true during peak hours, unfortunately, there were no off-peak hours, should I be reminding you of IRC during those days? :)

Need is the mother of Invention — Creation of the first HTTP compression prototype

In our quest for faster browsing and sending free emails (IIT would charge us Rs. 4 per email back than, pre-Hotmail), some of us would use an alternative — free US-based telnet servers. We would log into those servers, send emails (elm, pine), and also browse using Lynx instead of using overloaded and bandwidth-limited port 80 on the campus.

However, text browsing inside a console on state-of-the-art DEC Alpha and Silicon Graphics IRIX was no fun. I had more addictions as you can see, so I soon worked out a solution — running proxy servers on those telnet servers and setting proxy in Netscape.

Life was much better now — the browsing speed had definitely improved but it was still not good enough. However, seems like some forces wanted me to work on this, soon there was a tipping point, bzip2, which was released in 1996 and created a good buzz. Suddenly I started connecting browsing and the compression — what if all HTML files were compressed on the server and then decompressed by browsers — the browsing would be multifold faster. Wow, I praised myself and started working on a quick prototype!

Note that internet speed was not the only reason for slow browsing, there was also no compression or content negotiation used in HTTP/1.0 at that time. The entire traffic was uncompressed. No browser was supporting “Accept-Encoding:” (required to request compressed content from the server) including the most popular browser then, Netscape. The first Netscape version to support “Accept-Encoding:” was 4.06, released in Aug 1998 (history). Interestingly IE 4 did implement earlier in Sep 1997 and much later by Opera 4 (around the year 2000). Lynx was earliest in 2.6 which was released around Sep 1996 but then working compression support came later, along with the first HTTP/1.1 standard in early 1997.

I started working on combining HTTP and compression, and soon created a prototype where the contents were compressed on the fly on the server and then decompressed upon receiving it. I had to write a separate daemon process since browsers were not supporting compression then. The results were indescribable now, it took the browsing experience to the whole next level.

I made a post on the Linux-net mailing list in 1996 which is still live and undoubtedly proves that it was the first (or one of the first) prototype of HTTP compression even before HTTP/1.1 specification in 1997 and browser support much later. I wrote an email to Netscape but didn’t get any reply.

I did not get any recognition for it but the same holds true for many unsung heroes for their inventions — so it’s cool :) But, it feels nice and makes me proud seeing it being used by millions of people (or billions maybe).

Back to work and tea now :)