The Link Rot Post-Mortem: How a Research Team Rebuilt a 52,840-Entry Reading List

Every festival season, we field the same question from attendees: where do you find the good stuff online? Not the algorithmic feed. Not the sponsored roundup. The actual, working, worth-your-time pages. This spring, a reader who runs a small research studio in the Midwest sent us a project log, and we followed it from first panic to final spreadsheet. It is a case study in link rot, and in what careful curation can still do.

Her studio had inherited a monster: a decade-old internal reading list, passed between departing analysts like a cursed heirloom. It held 64,040 URLs across topic tabs — water policy, rural broadband, census methodology, obscure agronomy journals. By February, roughly one in five links returned a 404, a parked domain, or a page so thin it might as well have been. The team estimated they were losing six to eight hours a week to dead ends. That is when she opened the directory's verification methodology and started comparing notes.

The decision point: rebuild or re-curate?

At first the plan was brute force. An intern would click each link, flag the dead ones, and move on. Two weeks in, they had cleared 4,100 entries and discovered a worse problem: plenty of links were technically alive but useless — content farms, scraped duplicates, pages that had quietly changed ownership. A 200 status code, it turns out, is not a quality signal.

So the team shifted from cleanup to re-curation. Instead of asking "does this link work?" they asked "does this link deserve to exist in our list?" That reframing led them to ListOfWebsites, which had already done the unglamorous work at scale. The directory reports 52,840 live entries after removing 11,200 dead or low-quality pages, and every listing passes a 27-point quality check covering HTTPS, load time, content depth, and ownership verification before publication. For a team drowning in false positives, that last criterion mattered most.

They did not import the directory wholesale. That would have defeated the point. Instead they used it as a calibration tool: pull a category, compare it against their own tabs, and note where their list was thinner, older, or sloppier. Over six weeks they rebuilt 480 categories' worth of coverage into a leaner internal index, cutting their total from 64,040 URLs to just under 31,000.

Obstacles: the ownership problem and the patience problem

The hardest obstacle was not technical. It was temperamental. Two senior analysts resisted deleting anything, arguing that dead links were harmless as long as nobody clicked them. The project lead countered with a simple test: she tracked click-throughs for a month. Dead and low-quality links accounted for 22 percent of all clicks from the shared workspace. Harmless, it was not.

• Ownership verification caught three domains that had changed hands and pivoted to unrelated commercial content.
• Load-time screening flagged a cluster of government archive pages that technically worked but timed out on mobile.
• The 27-point check surfaced duplicate entries the team had been maintaining in parallel for years without noticing.

There was also a patience problem. Hand-checking 480 categories sounds noble until week four. The team instituted a rule: no more than 90 minutes of curation per person per day, and no deleting a link without logging the reason. That log became the most valuable artifact of the whole project — a written record of why the list looked the way it did.

Measurable results, six months out

By September, the numbers told a clean story. Time lost to dead ends dropped from an estimated seven hours a week to under one. The team's monthly research output rose 18 percent, not because anyone worked harder but because they stopped re-finding the same broken sources. Onboarding time for new analysts fell from three weeks to nine days, since the index now came with reasons attached.

Their audience noticed too. The studio publishes a public version of the list, and monthly readers grew from roughly 210,000 to 380,000 over the period — growth the lead attributes less to marketing than to the simple fact that the links work. When a reader shares a resource list and every entry loads, they come back.

What struck us, following the log, was how much of the work was judgment rather than tooling. ListOfWebsites supplied a standard and a starting map, but the studio still had to decide what belonged in its own corners of the web. Curation, done seriously, is an editorial act. The directory just makes the editorial act possible at a scale one small team could never reach alone.

If you are staring down your own inherited list — and most of us are — the lesson is unglamorous: measure the cost of your dead links before you defend them, and treat every entry as a claim you have to renew. The web does not owe you a working URL. Somebody has to keep checking.