Tuesday, September 29, 2009

Naïveté and How It Can Break Your Internet

As of the end of last week, I was all set to put together a new post tonight about the latest load of BS from Nominum, but now that will have to wait a few days. There's something much more pressing (and probably relevant) to the five or six people reading this on the day that I post it that I think I should talk about. This morning, the new issue of a weekly podcast that I listen to was posted.

Jesse Brown is the host of Search Engine, a weekly technology news podcast that I have been listening to for about six months, every week, no exceptions (well... actually... excepting weeks when there's no show). Jesse – I hope he doesn't mind if I call him Jesse – seems to me to concentrate on the social aspects and effects of technology (and he may have even stated this himself, on his show), and by and large I think he does it well. This week's issue, titled Are You Gay? (The Internet Wants To Know), was posted this morning, and the second interview (starting at about seven and a half minutes in) was about two subjects I know very well: the DNS, and CIRA. Unfortunately, what I heard on Jesse's show this morning was both shocking and disappointing. And I told him so. In retrospect I was perhaps overly harsh in some of my criticism, but only insofar as twitter is a terrible medium for conveying nuance or detail.

My problem with the interview is that both interviewer and interviewee were wholly unprepared. I'll get to the interviewee in a moment, because what he had to say is my main source of outrage. For the next paragraph or two I'll quickly sum up what I thought was wrong with the way Jesse approached the whole thing, and then get on with the meat of the matter.

The interviewee in this case clearly came to the table with an agenda, and its this agenda that the whole interview was really about: the desire to make sweeping changes to the status quo. Jesse not only did not challenge the position that change is needed, but Jesse admits that he arrived at the interview without knowing anything about the subject at hand. He did no background research and didn't even look for dissenting opinions. This approach is full of fail. If the interviewer doesn't challenge the agenda, and isn't informed enough for critical analysis of the interviewee's answers then it becomes the interviewer's equivalent of publishing a press release as a news item. Of course the person with the agenda is going to present the facts in such a way as to support their argument, and not a balanced view. Worse in this case, the "facts" weren't even really facts for the most part. One would like to believe that a director of an organization would be able to coherently discuss what that organization does, and why it does it that way, but one might be wrong. This whole thing will cause me to reassess Search Engine as a source of information. When covering those subjects that I really know nothing about, will I be able to trust that the expert on the air is really an expert? Certainly not as I have trusted in the past.

I first became aware that Jesse was working on something like this interview last Saturday when he asked a question on twitter that was right up my alley (I missed an earlier, more direct reference). I responded there and in email, offering to help fill in the blanks. I also happen to know that several other people made similar offers. Jesse didn't take me up on my offer, or anyone else's that I'm aware of, and unfortunately it's clear now the reason is that the interview had already been completed at that point, and was simply waiting to be aired. That is far too late to look for supporting information.

So what was this whole thing about?

Barry Shell would like you to elect him to the board of directors for the Canadian Internet Registration Authority (CIRA), the organization that manages the .ca Internet domain. This in itself is not news. What's newsworthy is why he would like you to elect him. He claims that CIRA is too big and expensive and that the organization should be run more like free online services like craigslist.org, or perhaps like the .ca domain was run back in the late 90's. The problem is that Barry's views are hopelessly naïve, are based on a simplistic understanding of what CIRA actually does and, if implemented, would not only threaten the stability of your Internet (well... the two or three Canadians reading this) but would also threaten the stability of your economy, and possibly even your life.

A bold claim, I know. I plan to back it up. But first, some background on me and where I'm coming from so that you can judge my agenda.

As anyone who has read the sidebar will know, until fairly recently I was the DNS Operations Manager for CIRA. I no longer work there, and I am not a part of, nor a candidate for the board of directors being elected right now. I am still a DNS specialist, and I work for a different (some would say competing) domain registry. So my only association with CIRA at this point is that I am a .ca domain-holder concerned about how the organization is run because it directly impacts the Internet that I use every day. Really, the only difference between me and most of the people likely reading this is that I happen to possess some very detailed information about how CIRA is run currently, about the environment in which it operates, and about the possible side effects of changing either of those things. My agenda is to try to share as much of that information as I can, and hopefully to convince you that the way in which CIRA is run is far more important than Barry Shell would have you believe.

Before I get into the finances, which believe me will be brief, I think it's important to correct the collection of misinformation, bad assumptions, and vague statements that permeated the interview, and which would lead the uninformed to incorrect assumptions.

To begin with, CIRA is not "a server." CIRA isn't even an organization that just runs "a server." CIRA is what is known as a domain name registry, but what is that? To explain, I'll step back a bit from CIRA and start with two other related groups.

First, there's the people who register .ca domains: the registrants. These registrants go to a web hosting company, or their ISP, or some other company to pay to register a new domain. This company, known as a registrar, will charge anywhere from $10 to around $50 to take care of the process, possibly also setting up some email or a web site or some other service to go along with the newly registered domain. In the background, this registrar submits the newly registered domain to CIRA, and pays them $8.50. What does CIRA do for that $8.50? Well, there are two core services that CIRA provides.

As a domain registry, CIRA is responsible for ensuring uniqueness. Just like the land registry, CIRA ensures that no two people or organizations think they've registered the right to use the same space; the difference is that CIRA deals in domains rather than plots of land. The second core service that CIRA provides is to include that registration in a global directory known as the Domain Name System, the DNS. Note that is Domain Name System, not Domain Name Server, as Barry says. The key here is that the DNS is a large interconnected database made up of at least hundreds of thousands, more likely millions of servers. The directory is structured like a tree, with the root branching out to the top level domains, or TLDs, like .ca, .com, .net, .org, .info, and others. The TLDs branch out to registrants' domains like cbc.ca or craigslist.org.. and so on.

What this directory system does is convert those memorable host names you type into a web browser (like www.tvo.ca) or into your mail client as part of an email address, into numeric addresses and other information that the computers of the Internet actually use to talk to each other. This is no simple task, but Ben Lucier has a great little layman's explanation of how part of it works.

CIRA's position in this directory is at the top of the .ca branch of this tree. It is responsible for making sure that any computer that looks up a .ca domain gets to the right place. Due to some shortcuts built into this system, the DNS servers at the top of the tree only see a tiny fraction of the total lookups that occur, but even that tiny fraction means that the servers responsible for the .ca domain answer about 13,500 of these lookups every second.

Every. Second.

Now, that's actually pretty easy work for a bunch of DNS servers, but the statistic starts to underline the importance of CIRA's position in making sure that all of those .ca domains continue to function. And that number doubles approximately every 18 months. When you take into account that most Internet businesses keep equipment in service for three to five years, that means that equipment CIRA is putting into service today to handle 13,500 DNS lookups every second must be able to handle over 100,000 per second by the time it is replaced. It's important that this DNS service that CIRA provides never be unavailable, or those lookups go unanswered.

But what happens if CIRA's DNS servers are unable to answer those queries for a few seconds... or a few minutes, or hours, or days? Does it really matter that much?

Perhaps not, if you're just talking about someone's personal web site, as Barry seems to mostly be concerned with, or a blog, or the place you download a weekly podcast. And in 1998, before CIRA existed, when the .ca domain was run by a bunch of volunteers led by John Demco (not just the two or three people Barry says it was), perhaps it wouldn't have been important if these sites failed to work for some period of time. But of course, we don't live in 1997 anymore. And the model of running a domain registry with a handful of volunteers and some donated servers was replaced with CIRA precisely because the old way of doing things was no longer working.

Today these uses of the Internet are all important, and they're part of what has made the Internet such a fundamental part of our daily lives. But, one must also remember that because the Internet has become a fundamental part of our daily lives, it has also become a major engine in the world economy. According to a study commissioned by the Interactive Advertising Bureau, the Internet is responsible for $300 billion in economic activity in the US every year. I'm certainly no economist, but if one were to very simplistically scale that back to the size of the Canadian economy, that would mean the Internet injects $27 billion into our economy every year. That's not chump change. If the Internet is unreliable, what happens to that money?

But let's move beyond corporate uses of the 'net and the economy. What's this poppycock about a broken Internet threatening my life?

Well, the Internet is now a part of daily business. What most people forget, is that doing business online doesn't just mean shopping for gifts, or playing online games. People forget that most organizations now use the Internet for internal communication. Organizations like online stores and social media, sure, but also organizations like our governments, critical infrastructure like our water, power and gas distribution... and our emergency services.

You may hear claims from your ISP that they guarantee "five nines" of availability. It's a fairly common service guarantee on the Internet, and it means that they are up and running 99.999% of the time. Put another way, it means that they permit themselves about five minutes of down time per year. Domain registries like CIRA don't do "five nines". They can't afford five minutes of outage every year. The DNS at that level must be a 100% uptime proposition, or Bad Things happen.

When this is taken into account, a pretty high level of redundancy to ensure availability seems warranted. Not only does CIRA need to ensure that there's enough capacity to handle all of the DNS queries their servers receive, without the servers becoming overloaded, they must also ensure that servers can be taken offline for regular maintenance, and that unexpected failures like power loss, crashed computers, network failures at an ISP, or other breakages don't take down the whole system. And that doesn't even address the threat of deliberate vandalism.

You may have heard of a style of attack against Internet services known as "denial of service", or DoS. One form of this type of attack involves sending extremely large volumes of requests to a service in order to tie it up, and reduce the resources it has available for legitimate requests. It's becoming increasingly popular to direct these attacks at DNS services. Today, these attacks are carried out using what's known as a "botnet" which is tens of thousands to hundreds of thousands of computers on the Internet which have been taken over to be used for often illegal purposes. Remember how I mentioned that CIRA's servers have to be ready to handle 13,500 queries per second today, and over 100,000 in five years? Well, as it turns out, it's quite simple for a small botnet to dwarf those numbers.

If CIRA simply built to the expected normal load, and added a bit to handle broken servers, they would still be vulnerable to being taken out by a bored high school student. This is nothing compared to the resources available to organized crime, or other nation states. And if you think nations attacking each other over the Internet sounds like a bad spy flick then get out your popcorn, because it's already happening. In order to prepare for these potential attacks, some registries build out their DNS infrastructure to support well over 100 times the expected load.

Given all of this, the money Barry Shell seems to think that CIRA is wasting seems pretty well spent, to me. And I've really only scratched the surface of one part of CIRA's budget, which is available online as part of the annual report, by the way. The side of the registry which is responsible for actually taking registrations may not be quite as essential a service as the DNS is, but many Canadian businesses, the registrars I mentioned earlier, depend on that registry being available to take registrations or they start to lose money, so those systems need to be well built to a different level of tolerance to failure or attack. Then there's CIRA's customer service department, programmers to write the software, systems people like me to make it run, the back-office functions, required by any business, like finance and administration staff... it adds up pretty quickly. CIRA actually operates pretty modestly compared to most domain registries.

Now getting to those finances..

It's been suggested that CIRA should reduce the wholesale cost of registering a domain from $8.50 per year to something smaller. However, it's been demonstrated in past reductions in price that wholesale price reductions don't get passed on to the general public as you might expect. $10 or $15 a year to register a domain isn't really an onerous sum for the average Internet user who wants their own domain, and a few cents to a dollar reduction in that cost really doesn't benefit the average Canadian all that much. The only people it does benefit are organizations that buy domains in very large numbers, like the domain registrars that sell to the general public, and another class of Internet user known as a "domainer". Domainers are those people who own literally thousands to millions of domains each, and frequently use them to put up those web pages that are nothing but advertizing, hoping that when you mistype amazon.com and accidentally go to azamon.com that you'll click on one of their ads and make them a few cents. CIRA takes its stewardship of the .ca domain – a national public resource – very seriously, and has no interest in supporting the interests of domainers over the interests of average Canadians who may want to register those domains that are just being used for ads.

It's true that most years CIRA operates with a budget surplus, in recent years as much as $2M. Where does that money go? Not-for-profit organizations are required by law to not have a profit. It's in the name. If a not-for-profit organization does find itself actually making profit, then the Canada Revenue Agency steps in for its cut. There are some pretty specific rules about when a not-for-profit is permitted to have a surplus, and what it can do with those surplus funds. What CIRA has done with its surpluses so far is to pay off a debt owed to UBC in exchange for all of the years UBC volunteers managed the service before CIRA existed, and to pay into a fund which is meant to support CIRA through any lean or financially disastrous years that may be yet to come. This is standard operating procedure for many companies, and is an especially important layer of insurance for an organization that operates a piece of critical national infrastructure.

Barry suggests that instead of these things, CIRA should be supporting research and other concerns of benefit to Canadians' use of the Internet, as if this is his own idea. In actual fact, CIRA staff have been lobbying the board to do just that for several years, and in the last year or two CIRA has already engaged in operational support and direct funding for several programmes, to the extent that it has been able to do that without stressing its rainy day fund or regular budget.

For having been on CIRA's board for a year, Barry shows pretty intense ignorance of CIRA's business and the environment it operates in. It's one thing for a new candidate, without any prior experience on a board, or without experience in the domain sector of the Internet industry to arrive fresh-faced with misconceptions about what CIRA does. It is essential for anyone who wishes to serve on a board of directors to inform him or herself to the best of their ability about the business they're operating, and the industry in which it operates. For someone who has been doing the job for a year to be so uninformed as Barry Shell requires almost willful ignorance. It's actually a shame that this interview aired when it did, because the election in which Barry is running ends tomorrow at noon, and I'd like everyone voting to listen to it; this week's Search Engine the best argument to not vote for Barry Shell that there is.

Friday, September 4, 2009

The End of the Printed Word

For years now I've been watching interest in words printed on paper steadily decline among many of the people that I deal with on a day to day basis. Being in high tech, and the Internet in particular, the people around me are on the leading edge of this decline. It freaks me out, partly because I can't completely comprehend it, but mostly because I think there is a lot to be lost if the same disinterest permeates average folks to the same degree.

A couple of months ago I moved from Ottawa back to Toronto. For various reasons, in this move I chose to hire a professional moving company rather than just rent a truck and move everything myself. The cost of the move has come up in conversation a few times, and since the cost was based entirely the weight of the stuff I was moving, every conversation eventually leads to the same question: "How could you possibly have so much stuff?!" The reason for the surprise should be self-evident when you hear that I had nearly 4,500 pounds of possessions packed into a one bedroom apartment. The answer to the question lies partly in my upbringing as a pack rat, but mostly in the size of my library; nearly half of the boxes (and therefore well over half the weight) were books.

Some people react to this news in the way I originally expected, with a look that says, "oooooh, that explains it!" There are a significant number of people in my circle of friends (who are mostly geeks) and in the group of people I work with (virtually all geeks) who react in a completely different way.

"Haven't you ever heard of a PDF?"
"You know about the Gutenberg Project, right?"
"Why don't you just get a Kindle or something?"
"Dude, sell that shit. You need to do a purge."

Every one of these people, at some point, reference the same argument in some way. Sooner or later they all get around to saying that paper is obsolete, and that I should get with the times and move it all to digital formats. I can't express strongly enough how much I disagree with this view without sounding ridiculous, even to myself. My reasons are many.

On the practical side, there are all the usual arguments about the stability of the two technologies: paper doesn't crash, get corrupted, or become unreadable when the power is off. Sure, there are counter arguments to these, but none that I take very seriously. Someone once tried to counter the "books don't crash" argument by saying, "yeah, but they burn real nice." I pointed out that drive crashes that result in a total loss of all data have been far more frequent than fires that gut my apartment (so far, five to nil). Besides, any fire that's likely to take out my library is going to take out any hard drives in my computer at the same time.

I have more than purely practical reasons for preferring paper, though. There's a comfort with paper that simply hasn't been reproduced with any electronic medium so far, and I dare to predict won't be even when we have paper-thin computer displays. I mentioned some of this back in January. Electronic books don't let me flip quite as easily between pages. They don't take pencil marks in the margins all that well, and even when that's possible it's never quite as simple or convenient as with a book. They don't balance quite so comfortably over my head when I'm laying back on my couch engrossed in that pulpy novel. And, browsing a list of book titles on a computer is nothing like reading the spines along a shelf.

Incidentally, I'm the same with my music. I have encoded my entire CD collection into digital formats for ease of listening, but I still have all 600 or so discs on display in shelves because, unless I'm searching for a specific song, or specific artist, it's way easier to flip through a stack of CDs and find something I want to listen to than to scan through a cold list of 7,000 individual tracks.

This sensual aspect to the printed word – the tactile experience and several thousand years of ergonomic refinement – can't be replaced by any combination of technology we have today. Books have a smell, and a weight, and a unique feel that we connect to as much as we connect to the information they contain. And anyway, let's face it: there's something awe inspiring about the visible mass of knowledge in a library, or in the care and craft put into many books. This is something you just can't get from standing in front of computer no matter how many electronic books it contains.

To cement my reputation as a complete geek, I'm going to quote an old episode of Buffy The Vampire Slayer, because all the truth you need is in fiction. In the first season, the episode I Robot... You Jane introduced Jenny Calendar, the school computer science teacher. In a conversation at the end of the episode, Rupert Giles, the librarian and Buffy's handler and advisor in all things ancient and supernatural, explains to Calendar why books are so important:
"Honestly, what is it about them that bothers you so much?," Jenny asks, referring to computers.

"The smell."

"Computers don't smell, Rupert," she protests.

"I know. Smell is the most powerful trigger to the memory there is. A certain flower, or a whiff of smoke can bring up experiences long forgotten. Books smell: musty and rich. The knowledge gained from a computer is – it has no texture, no context. It's there and then it's gone. If it's to last, then the getting of knowledge should be tangible, it should be... um... smelly."
It's because of all of this that I reacted with a particularly strong and unpleasant combination of confusion, astonishment, and disgust when I heard that Cushing Academy, a prep school in Ashburnham, MA, had gotten rid of virtually its entire library, to be replaced with a coffee shop, study space, a handfull of Kindles, and a subscription to an online library. Yes, you read that right: according to The Boston Globe, aside from a small collection of rare volumes Cushing has either sold or donated its entire library to other organizations and individuals.

It's one thing for someone to convert their, relatively speaking, small personal library into electronic formats. It's quite another for a school, of all places, to eliminate all of its books and hope that an electronic equivalent will fill the void. I believe it's foolish to think it could be a substitute even in the best of circumstances, and utter folly to hope that students who are still learning to learn will have any hope of getting the same education sitting in front of a computer, with its myriad distractions in email, instant messaging, and other in-your-face social media, as they would sitting at a desk with a textbook and some note paper. And that's just textbooks I'm thinking off. I can't help but think literature is entirely doomed among the students of this particular school.

And I know I'm not the only one who has this sort of reaction. Earlier this afternoon I was witness to a short exchange (online, no less) between the friend who pointed this story out to me, and a friend of hers.
Plastikgyrl: I'm currently reading a book on my computer. It's reinforcing my bookless library horror reaction. As Giles said, computers don't smell. :(

Refashionista: totally -- I swear the vanilla / cigar smoke smell of old paper gets me hawt ;)
Refasionista may have thought she was being glib, but she reinforces the point about the visceral connection people have with knowledge gained through books. This is something that just can't be replaced by any other technology we have today, and may never be replaced.

As I've thought about this more today, my disgust at James Tracy, the headmaster at Cushing, has turned slowly to fear.

I'm behind by a few years, but I've just recently finished watching The Wire, an astonishingly good HBO crime series that aired from 2002 to 2006. One of the major themes of the fifth and final season was an examination of how print news is reacting to the pressures of an increasingly digital world. The move to an online format, where news is given away for free, is setting the entire industry up for an epic fail, and I fear that a new, functional business model won't be found in time to save print news from disappearing in a puff of blogger commentary.

Distribution of the traditional printed newspaper is dropping like the proverbial stone, and online advertizing based on page views and click-throughs is unpredictable, and a slim income at best. The financial foundation of the print media is a sandy beach, and the tide is coming in. And I'm part of the problem. Practically my entire generation has turned away from print media for our news. I don't have a good explanation for this, except perhaps for our desire for less time consuming pursuits, or the simple fact that most of the print news is available online for free anyway.

If this important pillar of the fourth estate were to completely collapse, I don't see how it could ever be recovered, or how the void it would leave could ever be filled. It could spell the doom of current events knowledge among the general population. TV news doesn't have the same ability to surround a story, and examine it in any sort of depth, and bloggers by and large don't do news. To use myself as an example, other than linking to a few outside sources, I'm not reporting any facts here; this is all opinion. Somebody who links to a news story and writes a few pages about how that news affects people isn't doing news, they're doing commentary. Real news takes time and dedication. It takes full time professionals with a access to resources, a beat, contacts, and a certain set of ethics. The few bloggers out there who are trying to do news are lacking those things to varying degrees. Without some sort of in-depth reporting going on, people's knowledge of the world at large is at risk.

I fear that print news is on its way out, and I worry that it may be the toad in the environment of print media, whose death is an early warning that the books I love so much aren't long for this world.