A Surfer's Experience

Bots visit websites so that you don't have to.


2023-09-19

The web is an excellent place to shout messages that may or may not be of general interest. It didn’t take long after the web came to figure that out and it hasn’t changed. But what has changed is how we hear the shouting: It is shifting from search engines that guide users through an endless number of websites, to a few websites that users don’t leave.

You might, as you wake up on some day of your surfing vacation, wonder what the waves are like today. Normally you would use a search engine to guide you to some website you never heard of and get an update from there. But to your pleasant surprise, the search engine anticipates your question and presents the answer directly, to the right of the links. “Wave height 0.6 meters”, the right-hand side says. Life is good. You roll around for a nap.

Everyone wins except for the small website that you would have visited if the search engine had not already pulled the useful information from it. That website did most of the work but got nothing. Although websites contain useful information, users prefer not to visit. Consider all the pain points that are a common part of visiting a website today: Verify that you are human. Consent to cookies. Decide if you want to view the site through an app instead. Download or update the app. Create an account or log in. Reset the password if you forgot it. When you arrive, the information you want is behind a paywall or drowning in ads. Websites optimize for other things than UX, like maintaining a working business model, fending off bot traffic and following regulations. Each change taxes the UX slightly, so humans send bots in their stead.

So we do more with fewer websites. Those websites build trust that users leverage as a guide in the ocean of web information. They can also leverage social aspects to improve their product, like Reddit does with upvotes or Google does more subtly by checking the duration of website visits. Algorithms that extract information from the web are only becoming better and come packaged in new and better interfaces. The next wave is certainly different.