Please ensure Javascript is enabled for purposes of website accessibility
Recent News
Home / News / Epic recount website fail: One Dot One Dot One Dot One

Epic recount website fail: One Dot One Dot One Dot One

[Update 3/11/09: Crashgate made national news today as Wikileaks releases the database (and this story was cited by the whistleblower). Latest here.] [Correction: This PIM writer owes MDE blogger Michael Brodkorb a late correction for tonight’s story. He says that Ryan Flynn just posted the weird Coleman press release email’s odd administrative link as it arrived, embedded in an email, and thus, he stresses, he and MDE have no network connection with Coleman for Senate.com or FLS Connect! Sorry for speculating about that. –Dan]

Norm Coleman‘s web site crashed several times this week. The Coleman campaign says the crashes were due to high volume of hits (the site was linked from the Drudge Report); Coleman detractors maintain the Coleman campaign faked the crashes. All in all, the spin and nerdy noodlings around Norm Coleman’s website going down this week is, for the tech-lovers on PIM staff, easily the most entertaining recount development since “Lizard People,” and it’s even educated the public about proper Web security and the hitherto-politically obscure machinations of the domain name server (DNS) system itself. First, let’s take a trip down memory lane…

[Corrected, sorry, Noah]: Those of us in the gearhead set recall the electoral hacker kerfluffle launched with great aplomb in 2006, when then-Amy Klobuchar supporter (and current Uptake guy-but never a DFL staffer) Noah Kunin stumbled into an unsecured section of former U.S. Rep. Mark Kennedy‘s Senate campaign’s PR agency website that was supposed to remain secret. [He was “researching an article on Scott Howell‘s previous political ad campaigns when I found the ad in question,” he says.] (This practice of leaving sensitive files on publicly available directories has been accurately dubbed “security through obscurity”.) The usual suspects accused Kunin of nefarious sneaking around, and everyone learned that even the most straightforward probing of political websites could provide an avenue for the loosely-secured people to launch a political attack. [Kunin adds that Kennedy also used a DNS redirect to ‘fake’ taking down the website for ‘security’, but still left it running at the same IP address.]

Fast-forward to 2009: Quasi-U.S. Sen. Norm Coleman‘s team developed a reference for ballots they say are wrongly getting rejected. The site hit the Drudge Report and then promptly went offline on Wednesday. The Coleman team rapidly belted out a press release saying that massive interest had conked out colemanforsenate.com. Wily liberal web-heads Aaron Landry and Tony Webster decided to check things out, and discovered that the domain name colemanforsenate.com had been “pointed” to the very first IP address on the Internet, 1.1.1.1, thus ensuring anyone who attempted to visit it would land at a blank page. This led to much amusement and fruitful further diggings at MnPublius.com, which got dubbed “Crashgate”: “In short, they have configured their website to intentionally point at nothing.”

This, perhaps, could have been a legitimate mistake, but the inquisitors pointed out that the “time-to-live” (TTL) was much lower than usual. (TTL is the time interval at which a DNS lookup server is supposed to re-check its IP number references, to ensure that IP address updates get picked up around the world). Usually the TTL value is many minutes or hours, but it can be manually set to a few minutes, which lets the IP address changes propagate a lot more quickly. Thus, they could toggle between pointing to 1.1.1.1 and the real server (now 64.203.96.224) and have it take effect across the Internet quite rapidly. It seemed really unlikely someone would lower the TTL unless Coleman’s campaign was being disingenuous.

An esoteric event, perhaps, too technical to get written about. However, the hit parade continued as savvy Web types poked around. Compounding matters, Ryan Flynn, the new guy at Minnesota Democrats Exposed, posted the press release pointing to the Coleman for Senate website, originally in this form:

http://65.121.0.101/exchweb/bin/redir.asp?URL=http://www.colemanforsenate.com/

…and that IP address is owned by FLS Connect. This URL is in an odd form which seems like the kind of link you’d get from an internal network, which in turn suggests that MDE is somehow internally connected to the Coleman IT system, evidently provided by FLS Connect. Intrigue!
[Correction: no intrigue there, Brodkorb says. They just pasted the weird internal link as it came in the email press release – and there’s no network connection].

Further examinations revealed a multitude of Web no-nos, potentially breaching the privacy of Coleman’s big donor database: local “Organic Technology Consultant” Adria Richards at ButYoureAGirl.com publicized that the previous web server, located at 208.42.168.251, had left both a 205 MB database file located at /db/database.tar.gz available for all to obtain, and even better, an “Create Admin User” administrative page open to all. [The rumor is that the database file also contained credit card information!] Other faux pas like directory indexing and potentially harmful verbose error messages were also deduced by Richards.

Ever the cynics, PIM staff suspected the whole thing might have been an attempt to gin up more interest; if so, it backfired about as badly as theoretically possible, by revealing such a deficient level of protection for things that ought to be quite confidential. There’s concern that the Coleman people might try the “evil DFL hacker” message again, but it’s important to realize that no one did anything sneaky to get in: it’s their own fault for failing to secure their own website, and exposing their own very confidential database, possibly sharing thousands of donors’ credit card numbers to the whole Internet!

So finally, our conclusion: On the Internet, it is possible to fail at failing, as this joke pic from the epic FailBlog.org clearly conveys. Who knew?


Leave a Reply