I’ve written an introduction to SPBS, and the reasons I’m writing it here, at http://code.google.com/p/spbs/wiki/SPBSIntroduction

NOTE: I have only experienced this on the Vancouver Sun section of Canada.com. I’m currently out of different IP addresses to try. I’ll try again from home to see if the rest of Canada.com is infected.

How I found The Problem: I followed a link from Bynkii.com to a Vancouver Sun story that Bynkii discussed. My browser was immediately hijacked. I visited the site again, and found no errors. I then tried from a laptop connected to our company’s wireless network (completely separate from our internal network, and through a separate ISP) and saw the hijack again. That time I got screencaps. Visiting the site through the wireless again showed no problems, leading me to believe that this displays one time per IP address. Trying multiple browsers did not result in another browser hijack, and neither did clearing cookies, making me think it’s recording IP address and attacking once per IP address.

Details:

The first time you visit Canada.com, your browser will flash through a few redirects, and then the following popup will appear:

Note that hitting either Cancel or OK appears to have the same result — you can’t get out of there. Your browser will then appear to scan your hard drive for viruses. It’s all theater; it’s not actually doing anything at this point:

It will then pop up a message (in an Windows-style message box — not sure how it did that):

If you’re on a Macintosh, it will then offer to download an EXE file. I’m not sure if it will automatically download and run the file on a Windows machine, because I’m not about to try.

I uploaded the downloaded file to virustotal.com, and it found a variety of badness there, general consensus seems to be that this file isn’t so bad by itself, but will once installed download and install an additional slew of malware and trojans of unknown potency. (I was unable to find really definitive information about this file; I’m open to suggestions for better places to look.)

File Install-MnBhY2lmaWM-a2V5aW4-a2V5a received on 11.08.2007 03:28:07 (CET)

Antivirus	Version	Last Update	Result
AhnLab-V3	2007.11.8.0	2007.11.08	-
Authentium	4.93.8	2007.11.07	could be infected with an unknown virus
AVG	7.5.0.503	2007.11.08	Generic9.HLR
CAT-QuickHeal	9.00	2007.11.07	-
DrWeb	4.44.0.09170	2007.11.07	-
eTrust-Vet	31.2.5278	2007.11.07	-
FileAdvisor	1	2007.11.08	-
F-Prot	4.4.2.54	2007.11.07	W32/Heuristic-119!Eldorado
Ikarus	T3.1.1.12	2007.11.08	Virus.Win32.Renos.AE
McAfee	5158	2007.11.07	BraveSentry
NOD32v2	2645	2007.11.08	Win32/Hoax.Renos.PY
Panda	9.0.0.4	2007.11.07	Suspicious file
Rising	20.17.22.00	2007.11.07	-
Sunbelt	2.2.907.0	2007.11.07	-
TheHacker	6.2.9.119	2007.11.07	-
VirusBuster	4.3.26:9	2007.11.07	Trojan.Renos.Gen.2

Additional information
File size: 31288 bytes
SHA1: ef6fce7ad9a01d6cabb84bffc3bddee6f43bfe4e

I submitted this to the Internet Storm Center, along with a malware sample, and e-mailed webmaster@canada.com with a warning.

Update November 15, 2007: Wired just wrote about the issue; apparently it affects a number of different sites, as the hijack script was distributed via the Doubleclick ad network.

Update November 16, 2007: Wired wrote about the issue again, quoting me this time.

There’s very little payoff in stealing 20 servers (the amount stolen in the recent robbery) for the hardware value — this is most likely about stealing the DATA on the servers. Most people in the market for rackmount servers wouldn’t buy them off trucks, and so you’ve got a crime with serious time and very little payoff if they were stealing hardware. I wonder how much valuable data (including credit card numbers) was stored on those boxes?

Eckels took umbrage at reports that their facility had been robbed four times in the last two years:

One of the biggest mistakes is that people are talking about four robberies. A robbery means than property has been seized through violence or intimidation. C I Host has technically only been robbed twice in two years. The other two were break-ins where things were stolen, but not robberies.”

Umm. That doesn’t exactly make me feel better. Although the facility’s staff probably prefer the burglaries — their night manager was repeatedly tasered and “struck with a blunt instrument” during the most recent robbery.

I’ll be continuing this at the URL they suggested, and I’ll update this post.

Wednesday night my site went down. Consider it a comment on the level of service site5.com provides that this didn’t worry me too much. I knew it would be back up shortly, like it always is when Site5.com goes down. (Several times a week.) I had it on my list to move to a different hosting provider; I just hadn’t done it yet.

Then Thursday morning I get this e-mail:

From: devnull@site5.com
Subject: Subject: Site5 Incident Notification: oracle.site5.com
Date: May 24, 2007 8:53:45 AM PDT
To: [My E-mail Address]

Dear John Schofield,

My name is Todd Mitchell, I am Chief Operating Officer at Site5. I am writing you this morning in regards to the outage with oracle which began to affect you at approx. 18:00 EST, May 23rd, 2007.

At approx. 18:00 EST on May 23rd, 2007 oracle became unresponsive and as a result one of our system administrators requested a reboot for the server. A short time later, less than 10 minutes, our system administrator contacted our data center and hardware provider (Net Access Corporation / NAC) as the machine did not return to service from the unscheduled reboot. Upon further investigation by our data center, their initial determination was that the Operating System became corrupt and couldn’t initialize our disk array on boot.

At this point one of our lead system administrators decided in the best interest of our clients to begin a restoration from backup due to the inherent difficulties of restoring both a corrupted root and data filesystem. Upon making this decision, they proceeded to verify that our backups are intact and would at that point initiate our server restoration procedures. Unfortunately, after several rounds of integrity checking it was discovered that the backups for oracle are corrupt and unusable.

I am terribly saddened by this discovery. One of the core values of hosting is to first provide a quality hosting environment and secondly, ensure that data is available if a server should fail. Unfortunately due to dire circumstance, all data for this server is corrupt and completely unusable. We are in the process of returning our server to its original configuration and returning your account to its default state–your username and password remain the same. Please use them to FTP into your account and upload your local backups.

We realize how frustrating this is for you so we’re offering a 12 month service credit. This service credit is automatic (no need to request it and it will appear on your account within 24 hours) and it will be automatically applied to future invoices. Please note that this service credit has no cash value and cannot be requested in the form of cash, check and/or refund to a credit card. This service credit is, however, transferable between Site5 customer accounts. Alternatively, if you’d like to terminate your account, I have authorized our billing department to refund, back to your credit card on file, up to 12 months of previously paid for service.

This incident has been extremely trying for us. Site5 has been in business since 1999 and we’ve never experienced an issue like this–our clients have always received the best possible service from us. I simply cannot express in words how awful we feel and how heartbroken our system administrators are. We will, of course, be looking at how this happened and what we can do in the future to avoid this at all costs. I hope that we can regain your trust and patronage in some capacity.

Sincerely,

Todd Mitchell on behalf of the entire Site5 management team.

OK. So that’s completely screwed. I mean, for a professional hosting company (or any decent-sized company) not testing restores is unacceptable. It’s one of the cardinal rules of backups — make sure your backups are actually good. Making this mistake worse, it looks from the e-mail like they didn’t begin the restore process and discover their backups were corrupt — this is the classic form of this mistake. Instead, they did a backup validity check and determined (without having to perform a restore) that the backup was corrupt. That means they could have done this backup validity check at any time, without having to do a full restore — meaning there is even LESS excuse for this failure.

(In their defense, I have to say I like Site5’s handling of the issue, in terms of issuing credits or refunds. But that’s ALL that I like about this.)

Since I had already decided to leave Site5 at some point, this was an easy choice: Leave Site5 and get my money back!

From: John Schofield
To: Billing@site5.com
Date: May 24, 2007 9:23 AM
Subject: Re: Subject: Site5 Incident Notification: oracle.site5.com

We will be canceling our account with Site5. We request, per your e-
mail below, our previous 12 months of charges refunded. I expect to
see no further charges from Site5 on my credit card.

There is nothing I can say about how inexcusable a failure this is on
the part of Site 5. Words fail me.

Sincerely,
John Schofield

Then Site5, clearly determined to add insult to injury, sent this:

To: [My E-mail Address]
Subject: [Site5 #QZK-378129]: Re: Subject: Site5 Incident Notification: oracle.site5.com
Date: Thu, 24 May 2007 12:25:29 -0400
Reply-To: billing@site5.com

Hello John,

Thank you for writing. Though I am sorry to hear that you wish to cancel your account, we would be glad to assist you with any request that you may have.

As account cancellation will result in irreversible file loss, can you please reply with the following:

1. The full domain name of any account(s) you wish to cancel.

2. For security purposes, please verify either the last four digits of
the credit card on file in your account details or the physical
address that is on file in your account’s contact information.

3. Verification of a recent backup of the files. Have you been able to
download an archive of the account or any necessary files?

If you have any additional questions or concerns, feel free to contact us at anytime. Thanks again.

—
Jessica Noling
Customer Service
Site5 Internet Solutions, Inc.

http://www.site5.com

My reply:

From: John Schofield
To: Billing@site5.com
Date: May 24, 2007 9:30 AM
Subject: Re: [Site5 #QZK-378129]: Re: Subject: Site5 Incident Notification: oracle.site5.com

1. [My Domain Name]

2. [My address]

3. We are canceling our service because Site 5’s server went down,
and Site 5 did not have adequate backups in place. Our data has
already been irreversibly lost, by Site5’s incompetence. Thanks for
the thought, though.

Sincerely,

John Schofield

Site5 did refund all my hosting fees for the past 12 months, or I’d be foaming at the mouth far more than I am now. But despite them handling their failure (AFTER the fact) in a professional and responsible manner, I’m still thoroughly pissed I ended up in the situation in the first place.

I was able to restore almost all my content from backups, but some changes had been made in a DB on the site that had not been backed up recently — so I did lose some work because of this.

I’ve since set this site up (not sudosu.net; a different site) with dual hosting with two separate hosting companies, and my DNS provider doing failover between them, and automatic backup. That’s a future article.

I have no experience with Media Temple. But Grid-Server seems like one of the brilliant ideas that are obvious once someone else invents them. Oh sure, other people have done redundant web hosting before. Thousands of little guys plus Google, Amazon, Yahoo, MySpace, etc. are dividing web-hosting duties between many computers such that if one goes, down, the others can keep the website running.

But no commercial webhosting company I’m aware of currently offers grid-based hosting. This leads to outages and slowness for clients. DreamHost has had a number of recent problems. To their credit, they are quite open and honest about what went wrong. But there’s also a constant parade of announcements of servers going down for 10 minutes here, 10 minutes there for RAM upgrades, hardware replacements, etc.

There is NO REASON I can think of why my website, even at DreamHost’s low prices, should be dependent on ONE database server, ONE web server, and ONE file server. Any one of them goes down; my website goes down.

Creating a grid of dozens of computers, each of which can shuffle around websites to balance load or handle failures is not an easy task. It would take me loads of research before I was even ready to begin that sort of project, and I’d have to hire a team of people to do it. It’s a big deal.

On the other hand, it’s been proven that this sort of thing is possible. It’s not like we need basic research or feasibility studies. You can hire people who know how to do this if you’re willing to pay for it.

It’s not like this should be a tough decision for DreamHost. One of the primary benefits of this technology is to eliminate problems with over- and under-selling. The host has a pool of resources that can be grown incrementally, as resource use grows, without worrying about the growth rates of individual sites.

Sites have fewer worries about going down, both from the Slashdot / Digg effect, and from hardware failure or misconfiguration at the hosting company.

So why isn’t DreamHost doing this? My hope is that they’re already working on it — but I wish they’d let us know.

UPDATE June 9, 2007: This site is now hosted at NearlyFreeSpeech.net. So far, I love them. (An unfortunate truism of web hosting is that all hosts are great until they start to suck.) I’ve eventually been disappointed with every host I’ve tried. I host both personal and business websites; so far, pair.com and nearlyfreespeech.net are the only hosts on my “not sucking yet” list.