In a 1996 essay for Scientific American, Brewster Kahle laid out his plan for preserving the Internet. He noted, “Where we can read the 400-year old books printed by Gutenberg, it is often difficult to read a 15 year-old computer disk.” At the time of his writing, web pages had an average lifespan of just 75 days — most sites lived on temporary personal accounts, university servers, small ISP hosting, or short-lived commercial setups — and storage hardware was “rare and specialized because of costs.” A webpage could vanish the moment a server shut down, or if a file format became obsolete — a single company’s collapse meant a webpage could be lost forever. The Internet was very quickly becoming a record of human knowledge and culture, yet there was no guarantee that this ever-increasing record would survive.
That same year, Brewster Kahle went on to found the Internet Archive, his Library of Alexandria for the digital age. Kahle, a computer science and engineering graduate, had spent his time at MIT immersed in the then nascent field of Artificial Intelligence. He graduated in 1981 and, only two years later, helped Daniel Hillis start the Thinking Machines Company (TMC) in 1983.
TMC was one of the first companies created solely for the purpose of creating artificial intelligence. Hillis’ idea was parallelism: rather than using one fast processor, computer scientists would be better off using thousands of simple processors working in parallel. This contrarian enterprise attracted the likes of Richard Feynman, Marvin Minsky and Stephen Wolfram, but its approach to AI never delivered. TMC filed for bankruptcy in 1994. Still, they showed that parallelism could work at mass scale, effectively laying the groundwork for GPUs and the subsequent AI revolution. During this time at TMC, Kahle also helped invent WAIS (Wide Area Information Server) in 1989 — one of the Internet’s earliest search and document retrieval systems, and a precursor to the modern web search.
After leaving TMC in 1989, Kahle founded WAIS Inc. with Bruce Gilliat to bring the search and document retrieval technology to market. The firm sold Internet publishing tools and services to clients like Encyclopaedia Britannica and The New York Times, helping bring their massive databases of billions of words to the Internet. WAIS Inc. was purchased by America Online in 1995 for $15 million.
In the book Founders at Work: Stories of Startups’ Early Days, Kahle reminisces, “My first startup upon leaving Thinking Machines was a bootstrap. We had no investment at all, and I had no savings, so it was self-funded from the beginning…It was sort of like going from the Roaring ‘20s, when champagne is coming from everywhere, into the depression, where you are washing your baggies and reusing twist ties.”
Kahle was driven by what he foresaw: the early Internet carried with it a looming risk of massive cultural loss, akin to the burning of the Library of Alexandria, the loss of early films (recycled for silver), and discarded early printed works. To keep it all from disappearing, he cofounded two organizations: Alexa Internet, and, again with Gilliat, the non-profit Internet Archive. Alexa crawled the web to provide traffic-based recommendations and was acquired by Amazon in 1999 for $250 million in stock.
The Internet Archive, headquartered in San Francisco, California, is today the world’s largest free public digital library. It preserves the history of the web through its archive of more than a trillion internet pages, and hosts millions of digitized books, texts, videos — along with one of the world’s largest collections of software, including rare, vintage, and obsolete programs.
Kahle’s concern in the face of this ephemerality — where entire websites could vanish overnight when a domain expired or content was overwritten without any backup — was twofold: to safeguard this living trove humanity was creating on the go, and to make it accessible. To combat the web’s average 75-day lifespan and the dreaded “404 Document not found” error, the Internet Archive launched the Wayback Machine in 2001 and began taking snapshots of websites.
