In the mid-1990s, finding websites meant browsing through categorised lists maintained by human editors. Search engines existed but were primitive and unreliable. The dominant way to explore the web was through directories, with Yahoo Directory as the undisputed king.

Yahoo started in 1994 as “Jerry and David’s Guide to the World Wide Web,” a simple list of interesting sites. By 1996, it was the most visited website on the internet, and getting your site listed in Yahoo Directory could make or break a business. This seems absurd today when any search engine instantly indexes billions of pages, but the directory model had advantages we’ve forgotten.

Why Directories Worked

Early search engines were terrible. They indexed pages by counting keywords, which meant results were easily manipulated by stuffing invisible text into pages. Searching for “travel” might return a gambling site that had hidden the word “travel” hundreds of times in white text on a white background.

Web directories bypassed this problem through human curation. Editors reviewed every submission, categorised sites accurately, and filtered out spam. When you browsed the Yahoo Directory’s travel section, you got actual travel sites vetted by someone who understood the category.

This created a fundamentally different web browsing experience. Instead of searching for specific queries, you browsed categories. You’d start at “Recreation,” drill down to “Outdoors,” then “Hiking,” then “Pacific Northwest Trails.” Along the way, you’d discover sites you weren’t specifically looking for but found interesting.

The directory structure reflected how knowledge was organised in libraries and encyclopedias. It felt natural to browse from general to specific, following hierarchical paths through information. Search engines would later invert this model, jumping straight to specific answers without context.

The Human Layer

What made directories powerful was editorial judgment. Someone had looked at every listed site and decided it deserved inclusion. This implied quality control that search results lacked. If you found a site in Yahoo Directory, you knew a human had verified it existed, worked, and contained what it claimed.

This human layer also meant consistency. Sites were categorised accurately rather than based on keyword matching. You wouldn’t find a gambling site listed under “children’s education” because an editor had reviewed it. The taxonomy made sense because people designed it, not algorithms.

For website owners, this created a different incentive structure. Instead of gaming algorithms with SEO tricks, you focused on creating content good enough that editors would approve your listing. Quality mattered more than technical optimisation.

The Scale Problem

Yahoo Directory’s strength became its limitation. As the web exploded from thousands to millions of sites, human curation couldn’t keep pace. By the late 1990s, Yahoo employed hundreds of editors, but the backlog of submissions grew faster than they could review them.

Getting listed became a months-long process. Small sites might never get reviewed at all. The directory increasingly covered only a fraction of the web, missing entire emerging categories because editors couldn’t keep up.

Search engines, meanwhile, improved dramatically. Google launched in 1998 with PageRank, an algorithm that evaluated sites based on how other sites linked to them. This was closer to human judgment than keyword counting, but it scaled automatically. Google could index the entire web without hiring more editors.

By the early 2000s, most people had switched to search engines for finding new sites. Directories became backup options for when search failed. Yahoo Directory limped along until 2014, when Yahoo finally shut it down after 20 years.

What Directories Got Right

Despite their obsolescence, web directories had qualities worth remembering. The browsable hierarchy helped you discover adjacent topics. If you were researching Japanese history, the directory structure might lead you to related categories like Japanese literature or art that you wouldn’t have thought to search for.

This exploratory browsing model encourages serendipity. Modern search is goal-directed—you want a specific answer—but directory browsing was open-ended. You might start looking for one thing and end up somewhere completely different, having learned something unexpected.

Directories also provided context. When you found a site, you saw what category it belonged to and what other sites were listed alongside it. This helped you understand the landscape of a topic. Search results, by contrast, are decontextualised lists ranked by algorithmic relevance.

The editorial process created a vetted web. Not every site deserved listing in Yahoo Directory, which meant what was included had passed some quality threshold. Today’s web includes everything, which is democratic but overwhelming. Sometimes curation is valuable.

Modern Echoes

Web directories never completely disappeared. Niche directories still exist for specialised topics. DMOZ, another major directory, ran until 2017 through volunteer editors. Wikipedia’s list of external links serves a similar function for many topics.

Some businesses operate on directory models. Yelp is essentially a restaurant directory with user reviews. Product Hunt curates new apps and websites. These services prove there’s still value in human-organised collections of links, even in the search engine era.

Social bookmarking sites like Reddit or Pinterest also capture aspects of directory browsing. Subreddits organise links by topic, and browsing popular posts in a category resembles directory browsing. Pinterest boards group visual content hierarchically.

But none of these fully replicate the general-purpose, editorially curated web directory. They’re either niche (Product Hunt), user-generated rather than editor-curated (Reddit), or focused on specific content types (Pinterest’s images).

Why Directories Died

The fundamental problem was scalability. Human curation doesn’t scale to billions of pages. No amount of editors could keep a directory current when thousands of sites launch daily. Algorithms were imperfect but infinite.

There was also a business model issue. Directories generated revenue by charging for faster review or featured listings, but this limited income compared to search advertising. Google monetised every search query through targeted ads, generating far more revenue per user.

Cultural changes mattered too. As the web matured, people’s needs shifted from exploration to search. Early web users browsed because they didn’t know what existed. Modern users search because they have specific goals. Directories suit browsing; search engines suit finding answers.

The rise of social media also reduced directory relevance. Instead of browsing Yahoo’s entertainment category, you’d see what friends shared on Facebook. Social discovery replaced editorial curation as the main way people found new content.

Lessons for Today

The directory era teaches us that curation has value distinct from search. Organising information by human-designed categories helps people explore topics holistically. Sometimes you want a map of knowledge, not a direct answer to a question.

We’ve mostly abandoned this model online, but it persists in physical spaces. Libraries use subject classification. Bookstores organise by genre. Museums curate collections thematically. These approaches complement search—you can ask a librarian for a specific book or browse the history section to see what’s available.

The modern web could benefit from more directory-style organisation. Wikipedia’s category system is one example. Topic-focused link collections like Awesome Lists on GitHub are another. These human-curated resources provide structure that algorithmic feeds lack.

Some organisations are trying to bring back editorial curation. Several consultancies focused on business transformation have recognised that structured knowledge organisation helps teams navigate complex information. Team400, for instance, works with companies to organise data and workflows in ways that make sense for human users, not just algorithms.

As we move further into an AI-driven web where search engines generate answers rather than lists of links, the directory model’s emphasis on browsing and exploration might become relevant again. When algorithms directly answer questions, discovery requires different mechanisms—perhaps returning to curated, hierarchical collections of resources.

The Yahoo Directory is gone, but the problem it solved remains: how do you help people explore information spaces when you don’t know what you’re looking for? Search engines excel at known-item retrieval, but browsing a well-organised collection still beats them for exploratory learning.

Maybe we need a new generation of web directories. Not comprehensive attempts to catalogue everything, but curated collections for specific domains, maintained by communities with expertise. Human judgment at scale through distributed curation rather than centralised editorial boards.

The directory model wasn’t perfect, but it got something right about how people naturally explore topics. As the web grows more chaotic and AI-generated content floods search results, hand-picked collections of quality resources might become valuable again. Not as a replacement for search, but as a complement—a map to navigate by when you don’t know exactly where you’re going.