“Amazon Will Destroy You” says, among other things: “Amazon is going to destroy the Big 6, destroy bookstores, destroy 95% of all agents, destroy distributors (Ingram, Baker & Taylor), and revolutionize the publishing industry by becoming the dominant force. . . . Amazon INNOVATES. That’s the thing you whiners don’t understand.”

My own (very limited) adventures in publishing make me inclined to think this is basically correct; having now dealt a fair amount with agents, I’ve observed enough about their behavior to be suspicious: most appear to simply be judging submissions based on “gut instincts” of dubious predictive value. An increasing body of research shows how horseshit gut instincts are as a guide to most kinds of behavior, and how little we know about the decisions we make. You only have to read Daniel Kahneman and Dan Ariely to understand this. You can see the same principles in Moneyball, book or movie. Virtually every field that has pitted empirical data or tests against the beliefs of supposed “experts” has found that the data win. In the case of Amazon, the company is basically running a massive experiment by letting anyone who wants to publish, publish, and letting readers sort out who’s good.With the gates open, entrants are numerous—and more numerous than conventional publishers could ever hope to match.

Literary agents and publishers are now in the same position as baseball managers were in the pre-Moneyball era: completely missing the data revolution going on around them.

The gatekeepers are just wrong too much. You can tell as much from the innumerable stories about beloved, or at least popular, books that were rejected dozens or hundreds of times before finding a publisher. This isn’t the fault of agents or publishers directly: if I were an agent and saw the manuscripts for Harry Potter, Twilight, or The Da Vinci Code, I’d have rejected them—and I’d have been wrong, at least as measured by revenue. When publishers had a lock on distribution, however, they could afford to be wrong, because it was virtually impossible for writers to use alternate mechanisms to sell their books. A wrongly rejected book went in the drawer. A lucky break just happened to be a lucky break.

Increasingly, though, writers aren’t going to rely on gatekeepers. They’re going to let the market decide; publishers and agents will step in after a writer is popular. The problem is, once a writer is sufficiently successful via Internet metrics, I’m not sure that agents or publishers will be in much of a position to negotiate.

Amazon knows this. Its strategy is simple: let everyone in and let readers sort ’em out. This will shake out many of the false negatives given by agents and publishers. Conventional publishers, who pay for expensive Manhattan office space and distribution warehouses and paper, can’t afford to do this using traditional methods.

Agents and publishers probably know this, but they know it in the same way Blockbuster knew it had to compete with Netflix.

The world is moving away from mystical hand-waving and towards data / experimentation. Businesses on the wrong side of that divide are going to suffer.

EDIT: I’m reading Culture: Leading Scientists Explore Societies, Art, Power, and Technology and, weirdly, came across this contribution, by Cory Doctorow, which discusses a different context that applies to my analysis of the literary marketplace:

We’re bad futurists, we humans. We’re bad at predicting what will be important and useful tomorrow. We think the telephone will be best used to bring opera to America’s living rooms. We set out nobly to make TV into an educational medium. We create functional hypertext to facilitate the sharing of draft physics papers.

If you need to convince a gatekeeper that your contribution is worthy before you’re allowed to make it, you’d better hope the gatekeeper has superhuman prescience. (Gatekeepers don’t have superhuman prescience.) Historically, the best way to keep the important things rolling off the lines is to reduce the barriers to entry. Important things are a fraction of all things, and therefore, the more things you have, the more important things you’ll have.

The worst judges of tomorrow’s important things is today’s incumbents. If you’re about to creatively destroy some incumbent’s business-model, that incumbent will be able to tell you all kinds of reasons why you should cut it out. Travel agents had lots of soothing platitudes about why Expedia would never fly. Remember travel agents? Wonder how that worked out for them.