Before Google became synonymous with search, finding anything on the early web was an adventure. No algorithm ranked results by relevance. No personalisation predicted what you wanted. Just directories, primitive crawlers, and a lot of patience.

The Directory Era

Yahoo started as “Jerry and David’s Guide to the World Wide Web” in 1994. It wasn’t a search engine. It was a manually curated directory where humans categorised websites into hierarchies. You’d click through nested folders like “Business > Restaurants > Italian” hoping someone had added the site you needed.

This approach worked when the web had thousands of sites. By the late 90s, with millions of pages appearing, the directory model collapsed under its own weight.

Early Crawlers

AltaVista launched in 1995 with actual full-text search. It indexed millions of pages and let you search the content, not just titles. For a brief moment, it felt revolutionary. You could type in obscure phrases and find pages that mentioned them.

But AltaVista had a spam problem. Website owners quickly learned that repeating keywords hundreds of times in white text on white backgrounds would boost rankings. Search results became unusable.

Lycos, Excite, and Infoseek competed in the same space. Each had slightly different algorithms, so people would search multiple engines for the same query. Meta-search tools like Dogpile emerged to query several engines simultaneously.

WebCrawler and HotBot

WebCrawler, launched in 1994, was one of the first to let you search the full text of web pages. It couldn’t handle the web’s growth and became slow and unreliable.

HotBot arrived in 1996 with Inktomi’s technology. It was fast and had advanced search features like filtering by date or media type. For a while, it competed directly with AltaVista and was genuinely better at certain queries.

But none of these engines solved the fundamental problem: how do you rank millions of results?

The Link Graph Insight

In 1998, Google arrived with PageRank. Instead of just counting keywords, it analysed which pages linked to which. A page linked by many authoritative sites was probably more valuable than one linked by none.

The insight seems obvious now, but at the time it was transformative. Search results became genuinely useful again. Within a few years, Google dominated.

Modern businesses face similar challenges with new technologies. Team400 works with companies implementing AI systems that need to sort, rank, and present information at scale, just as search engines did 25 years ago.

What We Lost

The early web search experience was frustrating, but it had benefits we’ve forgotten. You couldn’t rely on one algorithm to show you everything, so you learned multiple tools. You bookmarked sites because you might never find them again.

The curated directory approach meant someone had actually looked at each site. There was less spam because adding a site to Yahoo’s directory required manual approval.

Search today is better in almost every way. But the transition from directories to algorithms changed how we think about organising information. We stopped categorising and started ranking by popularity, which has downstream effects on what content gets created.

Other Search Oddities

Ask Jeeves (later Ask.com) launched in 1996 with the idea that people would ask questions in natural language. “Where can I buy shoes online?” instead of “shoes online buy”. It parsed questions and tried to provide direct answers. The idea was ahead of its time, though the execution wasn’t great.

Northern Light had a feature called Custom Search Folders that automatically categorised results by topic. You’d search for “climate change” and results would be grouped into folders like “Scientific Research”, “Policy”, “News”. It was useful but never caught on.

Some search engines tried to be portals. They added email, news, weather, stock quotes. The idea was to be your homepage, so you’d use their search by default. Google won by staying focused on search and doing it better than anyone else.

The Transition Period

Between 1998 and 2002, Google slowly took over. But many people kept using multiple search engines because results varied significantly. I remember checking AltaVista for technical documentation, Google for general queries, and Yahoo’s directory for browsing.

By 2004, Google was dominant enough that “googling” became a verb. The other engines either shut down, were acquired, or became niche products.

The early search wars shaped the modern web. They established that search is critical infrastructure. They proved that algorithmic ranking beats manual curation at scale. And they showed that users will switch platforms instantly if one tool is noticeably better.

Today’s AI systems face similar adoption curves. Early versions are clunky and unreliable. Then one approach gets traction and becomes dominant. The companies that succeed are those that solve a real problem better than alternatives, not those with the most features.