Blog
We’ve spent six years sifting through billions of links and found that the web is both expanding and shrinking
Jul
The online world is constantly expanding—always accumulating more services, more users, and more activity. Last year, the number of sites registered under the “.com” domain exceeded 150,000,000.
However, more than a quarter of a century has passed since the first commercial utilize of the Internet, and its development in some key categories is beginning to ponderous.
We conducted a multi-year research project analyzing global trends in diversity and dominance on the Internet. Our researchpublished today in the Public Library of Science, is the first to reveal some long-term trends in the way businesses compete in the Internet age.
We have seen a dramatic consolidation of attention toward a shrinking (but increasingly dominant) group of Internet organizations. So while the functions, features, and applications offered on the Web continue to grow, the number of entities providing those functions is shrinking.
Network diversity is failing rapidly
We analyzed more than six billion comments from Reddit users dating back to 2006, as well as 11.8 billion Twitter posts dating back to 2011. In total, our research leveraged a massive 5.6 TB of data spanning more than a decade of global activity.
This dataset was more than four times larger than the original Hubble Space Telescope data that helped Brian Schmidt and his colleagues do their Nobel Prize-winning work in 1998 to prove the expansion of the universe is accelerating.
We analyzed all links to other sites and online services from Reddit posts—over a billion in total—to understand the dynamics of link growth, dominance, and diversity over the decade.
We used a measure of link “uniqueness.” On this scale, 1 means maximum diversity (all links have their own domain), and 0 means minimal diversity (all links are on a single domain, e.g. “youtube.com”).
A decade ago, there was a much greater variety of domains in Reddit user-posted links, with over 20 different domains for every 100 random links posted by users. Now, there are only about five different domains for every 100 links posted.
In fact, 60-70% of all attention on top social media platforms is focused on just ten popular domains.
In addition to social media platforms, we also studied linking patterns across the web, looking at nearly 20 billion links over three years. These findings reinforce the notion that “the opulent get richer” on the Internet.
The authority, influence, and visibility of the top 1,000 global websites (measured by network centrality or PageRank) are growing month over month at the expense of every other site.
Read more: The founder of the Internet now wants to ‘fix the web’, but his proposal falls brief of expectations
The variety of applications is growing
The web began as a source of innovation, novel ideas, and inspiration—a technology that opened up the playing field. Now it is also becoming a medium that actually stifles competition and promotes monopolies and the dominance of a few players.
Our findings resolve a long-standing paradox about the nature of the web: Does it aid businesses, jobs, and investment grow? Or does it make it harder to get ahead by allowing anyone to join the game? The answer, it turns out, is both.
While the diversity of sources is failing, functionality is constantly increasing thanks to novel services, products, and apps — such as music streaming services (Spotify), file-sharing programs (Dropbox), and messaging platforms (Messenger, WhatsApp, and Snapchat).
Infant Mortality Website
Another critical finding was a dramatic augment in the “infant mortality” rate on websites — with the “gigantic players” guarding their territory more fiercely than ever.
We analyzed novel domains that were consistently referenced or linked to on social media after they first appeared. We found that while almost 40% of domains created in 2006 were lively five years later, only a little over 3% of domains created in 2015 remain lively today.
The dynamics of online competition are becoming increasingly clear. And the loss of diversity is worrying. Unlike the natural world, there are no sanctuaries; competition is part of both nature and business.
Our research has profound implications for business leaders, investors, and governments around the world. It shows that network effects are not confined to internet companies. They have permeated the entire economy and are rewriting many previously accepted principles of economics.
For example, the idea that companies can maintain a competitive advantage based on where they are physically located is increasingly feeble. In the meantime, there are novel opportunities for companies to set up shop from anywhere in the world and serve a global customer base that is both mainstream and niche.
The best way to promote diversity is to augment the number of global internet companies that focus on providing differentiated services that meet increasingly niche consumer needs.
In Australia we are starting to see this through domestic companies such as Canvas, SecurityCulture AND I’m wonderingHopefully, there will be many more in the coming decade.
Read more: If something is free on the Internet, you are the product
