Dirk van der Woulde, Dutch telecom guru, sent notice this morning of a new paper out of Japan with the off-putting title: "The Impact and Implications of the Growth in Residential User-to-User Traffic." If that sounds dense and hard to read, rest assured that it is. MuniWireless posted a pdf of a slideshow by the paper's lead author that re-presents the major points in classic PowerPoint style that is a bit more digestible.
The earlier post, "Japan, Soon to Be Fiber Broadband Nation" noted the way that the Japanese have used their dense DSL broadband infrastructure (cable is a minor player by US standards) to leverage a rapid shift to fiber to the home (FTTH). The US Telcos haven't been able to manage the same shift--putting the US as a distinct competitive disadvantage as new forms of media and ways of communicating are developed elsewhere. Japan is well on its way, in both its rural and its urban areas, to being the world's first fully fibered nation.
The new "Residential User" study focuses on, well, the residential user. Commissioned by five major Japanese ISPs the study is motivated by an attempt to understand the enormous growth in backbone traffic that has occurred as the shift to fiber accelerated--symmetric residential traffic increased 45% in 2005. What caused that jump? Would it continue? The fear was very explicit:
There is a strong concern that if this trend continues Internet backbone technologies will not be able to keep up with the rapidly-growing residential traffic. More-over, commercial ISPs will not be able to invest in back-bone networks to support this traditionally low-profit customer segment.In other words they fear that they'll get caught in an American-style inability to support a major network expansion.
The anticipated culprit was the fabled 'power user' using peer to peer (P2P) networking to transfer large music and video files. Network types across the world have been fretting over the threat of these few evil users to "hog" network bandwidth. It seemed obvious that these guys were a big problem that had to be stopped. But a funny thing happened on the way to the obvious conclusion: the data stepped in and confused the matter.
While a few users--on the order of 4%--did use most of the residential bandwidth consumed--about 75%. Now note that this isn't an unusual situation. Consider good ole phones. My guess is that the top 4% of phone users initiates 75% of all calls (or some similar large percentage.) And I think we all know some of those people. The point is that having heavy users is neither unusual or a problem in itself. The concern that the Japanese ISPs appeared to have was that a single application--P2P file exchange was an illegal cancer that threatened to destroy the system. What actually appeared in the data was evidence that while a few users did account for a lot of traffic, P2P networking appeared to be only one reason more bandwidth was used and, much more surprising, these "heavy hitters" were not a population distinct from all the (good) customers. Instead they were normal users, a sort of leading edge that simply did more of what everyone was doing. An analysis of the traces that P2P traffic should leave failed to find the expected pattern and led to the following conclusion:
This suggests that high-volume traffic is generated not only by peer-to-peer file-sharing but also by other applications such as content-downloading from a single server. A plausible explanation for the large variance is that the majority of users use both file-sharing and downloading with different ratios.The "evil few" hypothesis was not tenable. It appeared, instead that what was going on was a version of "if you build it they will come." They offered the Japanese people large, cheap, symmetrical bandwidth over rock-solid fiber and the Japanese people quickly and apparently easily found a ways to use it. Those that were using the most bandwidth were not qualitatively different from the normal user; they were not doing something different from the normal user except for the quantity of bandwidth the used. They were simply a little ahead of the curve. P2P filesharing is not threatening the Japanese backbone. A new version of normal use, evoked by the availability of big broadband is. The Japanese ISPs/Telcos are simply going to have to bit the bullet and build more capacity. Fast.
There are a few points to ponder for Lafayette's anticipated build:
- "If you build it they will come." There is no need to be concerned that people won't find uses for the bandwidth made available; apparently it's not difficult to figure out. Make sure our local backbone is plenty beefy.
- "Symmetry is good." One of the surprises for the researchers was that upload and download amounts were roughly equal. Much of the American model is built around the presumption that we only need to "consume" download bandwidth, so its more efficient to limit upload speeds and use the scarce capacity for download bandwidth. The evidence here is that this is an artificial constraint. Give people symmetrical bandwidth and they'll find a way to use it. Gaming, serving, telephony, and video telephony all come to mind.
- "P2P networking is not the source of all Evil." Fear of P2P networking is widespread among system administrators. But the evidence is that it is only one element in an exploding demand for bandwidth. A demand that will have to be met in any case.