the bot takeover of the web is faster than ever
Bots have either just overtaken humans as the majority users of the web, or are just about to. And it's only going to get worse.
A little over a year ago, I made a thorough video in which I explained how the web as we know it is being taken over by bots at great cost to us, both economic and social. While the web was originally built as a way for humans to share information, it’s now becoming a playground for AI, bots, crawlers, and scrapers.
From search engines on the prowl for new webpages to scrape and content to steal, fake accounts to boost propaganda or monetize political discord, to click fraud and spam, to AI-driven malware campaigns, machine traffic is now growing eight times faster than human activity thanks to a massive assist from large language models running wild. All this sounds pretty scary, but what exactly does that mean?
Well, the problem is somewhat nuanced and before we dive into the ramifications, I want to make sure we establish the difference between the web and the internet, as we use this interchangeably, but with this bot-pocalypse unfolding, really should not be anymore because it makes ascertaining what’s happening and why confusing at best, and too abstract to think about at worst.
The internet is a series of tubes... err... I mean peer to peer protocols called TCP/IP, created and unveiled around the same time as the Moon landing. Dig into these protocols, and you’ll have to navigate what looks like an explosion at the Scrabble tile factory, but fundamentally, these are all different formats and rules to communicate with electronic devices in a predetermined way that expects to route requests through a shifting, rapidly expanding ocean of nodes.
The web, on the other hand, is the human-friendly wrapper over this, presenting data as hypertext websites, pioneered in 1991 to help scientists share data. It’s the facade over the messy, ever-changing plumbing connecting billions of devices to each other and executing commands from humans and other machines, and it’s what lets us talk to each other and tell computers halfway around the world what to do.
So, in short, the internet is made for machines and the web is made for us. There was always going to be exponentially more internet traffic because machines talk to each other on timespans measured in milliseconds while humans take thousands of times, if not millions of times longer to react by comparison. If you let an update run for two hours while you grabbed lunch and attended a meeting, you may as well have been gone for a week in the computer realm.
Web traffic taken over by machines, on the other hand, is an aberration. I would even frame it as machines invading the human space. Part of this invasion is necessary. If you want to search the web, you need to have crawlers exploring what’s out there in the first place and extract key words, summaries, and metadata. But bots viewing or clicking on ads, posing as humans, and spying on your every move? That’s not what they’re supposed to do on Al Gore’s internet, dammit!
And that’s really the crux of the dilemma, isn’t it? We had an amazing ecosystem in which to escape the real world, or learn more about, to share with each other and be creative. Then, the very companies claiming to be the custodians of the web locked up all the biggest spaces into walled gardens they don’t want us to leave, and flooded it by letting the digital sewers overflow.
Again, all of this is a direct consequence of the commercial web being built around ads, then engaging in a race to the bottom to entice anyone with some dusty couch cushion change to place an ad or two. There are over a billion websites out there, a fifth of them are only serving ads for bots to view, all of them being scraped since AI and search crawlers don't know if they're worth indexing or not, and as a result, depending on who you ask, bots account for either 51% of all web traffic or 49% and are guaranteed to steamroll us meatbags in months.
Solving this problem isn't easy, however. The reason why the web turned to the TV-like model of advertiser support is because if people had to pay for every click and every link, micro-transaction style, the technology would've never taken off in the first place. It's exactly what killed the precursor to the web designed almost 20 years before the first HTTP web page went online, and no one wanted to repeat the same mistake.
Even if they went ahead with that plan, there was no way to process billions and billions of payments every day securely and without a ton of physical paperwork in the 1990s. If we get in our time machine and deliver the schematics for the global electronic payments network we have now, removing ad support meant you'd need a subscription for every website, and we know how well people took to the idea of paying an ever-rising subscription for anything and everything.
We can see today how media outlets erecting paywalls and demanding to disable ad blockers so they can absolutely slather your screen with ads, is driving users up a wall because their wallets are bleeding to death from a thousand monthly cuts. If you make every website yet another store or another paid service, trillions of dollars in market value evaporate overnight.
Social media implodes. Google is now a back office company. Stock markets are in chaos. News is either expensive and at least mostly factual, or free, and nothing but propaganda for sale to the highest bidder. We'd be looking at Great Recession 2.0 except we wouldn't because neither advertisers or tech companies relying on them would ever take the financial equivalent of a running kick to their genitals over and over and over again willingly.
On the other hand, if we end up with an internet build by bots, for bots, to watch and charge for ads people barely ever see while the human web is nothing but ads with a content break so it's now impossible to use the free apps, and its supposed custodians are busy maliciously destroying every incentive to contribute to it, or trying to lock us away in their own version of it, the system will implode anyway.
Far too many people today are scrolling the web and social apps to kill time or just out of habit. So, what happens when the bot takeover is finished and the human experience is just too miserable to suffer through, and the advertising lifeblood of the free and open web is nothing more than an array of bytes passed between a few million Python, JavaScript, or PHP scripts with nary an actual customer in sight?
