I’ve written an introduction to SPBS, and the reasons I’m writing it here, at http://code.google.com/p/spbs/wiki/SPBSIntroduction

http://biowiki.sudosu.net

(I’m keeping the shitty first draft to myself, at least for now.)

python `python -c “import pimp; print pimp.__file__”` -i readline

Moral? If at first you don’t succeed, Google. If Google fails, try, try again.

Update: And to make forward-delete work, try this.

NOTE: I have only experienced this on the Vancouver Sun section of Canada.com. I’m currently out of different IP addresses to try. I’ll try again from home to see if the rest of Canada.com is infected.

How I found The Problem: I followed a link from Bynkii.com to a Vancouver Sun story that Bynkii discussed. My browser was immediately hijacked. I visited the site again, and found no errors. I then tried from a laptop connected to our company’s wireless network (completely separate from our internal network, and through a separate ISP) and saw the hijack again. That time I got screencaps. Visiting the site through the wireless again showed no problems, leading me to believe that this displays one time per IP address. Trying multiple browsers did not result in another browser hijack, and neither did clearing cookies, making me think it’s recording IP address and attacking once per IP address.

Details:

The first time you visit Canada.com, your browser will flash through a few redirects, and then the following popup will appear:

Note that hitting either Cancel or OK appears to have the same result — you can’t get out of there. Your browser will then appear to scan your hard drive for viruses. It’s all theater; it’s not actually doing anything at this point:

It will then pop up a message (in an Windows-style message box — not sure how it did that):

If you’re on a Macintosh, it will then offer to download an EXE file. I’m not sure if it will automatically download and run the file on a Windows machine, because I’m not about to try.

I uploaded the downloaded file to virustotal.com, and it found a variety of badness there, general consensus seems to be that this file isn’t so bad by itself, but will once installed download and install an additional slew of malware and trojans of unknown potency. (I was unable to find really definitive information about this file; I’m open to suggestions for better places to look.)

File Install-MnBhY2lmaWM-a2V5aW4-a2V5a received on 11.08.2007 03:28:07 (CET)

Antivirus	Version	Last Update	Result
AhnLab-V3	2007.11.8.0	2007.11.08	-
Authentium	4.93.8	2007.11.07	could be infected with an unknown virus
AVG	7.5.0.503	2007.11.08	Generic9.HLR
CAT-QuickHeal	9.00	2007.11.07	-
DrWeb	4.44.0.09170	2007.11.07	-
eTrust-Vet	31.2.5278	2007.11.07	-
FileAdvisor	1	2007.11.08	-
F-Prot	4.4.2.54	2007.11.07	W32/Heuristic-119!Eldorado
Ikarus	T3.1.1.12	2007.11.08	Virus.Win32.Renos.AE
McAfee	5158	2007.11.07	BraveSentry
NOD32v2	2645	2007.11.08	Win32/Hoax.Renos.PY
Panda	9.0.0.4	2007.11.07	Suspicious file
Rising	20.17.22.00	2007.11.07	-
Sunbelt	2.2.907.0	2007.11.07	-
TheHacker	6.2.9.119	2007.11.07	-
VirusBuster	4.3.26:9	2007.11.07	Trojan.Renos.Gen.2

Additional information
File size: 31288 bytes
SHA1: ef6fce7ad9a01d6cabb84bffc3bddee6f43bfe4e

I submitted this to the Internet Storm Center, along with a malware sample, and e-mailed webmaster@canada.com with a warning.

Update November 15, 2007: Wired just wrote about the issue; apparently it affects a number of different sites, as the hijack script was distributed via the Doubleclick ad network.

Update November 16, 2007: Wired wrote about the issue again, quoting me this time.

There’s very little payoff in stealing 20 servers (the amount stolen in the recent robbery) for the hardware value — this is most likely about stealing the DATA on the servers. Most people in the market for rackmount servers wouldn’t buy them off trucks, and so you’ve got a crime with serious time and very little payoff if they were stealing hardware. I wonder how much valuable data (including credit card numbers) was stored on those boxes?

Eckels took umbrage at reports that their facility had been robbed four times in the last two years:

One of the biggest mistakes is that people are talking about four robberies. A robbery means than property has been seized through violence or intimidation. C I Host has technically only been robbed twice in two years. The other two were break-ins where things were stolen, but not robberies.”

Umm. That doesn’t exactly make me feel better. Although the facility’s staff probably prefer the burglaries — their night manager was repeatedly tasered and “struck with a blunt instrument” during the most recent robbery.

• Consumating.com: Free site. Lots of hipsters in their 20’s. Not a lot of women in their 30’s. Has a fun, quirky vibe that makes it a kind of fun site to hang out on, but not terribly useful as a dating site. Everyone gets a numerical rank based on their points. (I have 29 points, which puts me in 4348th place.) Relive the unpopularity of high school, electronically. (For some reason they didn’t use that as their slogan.)
• OKCupid.com: Free site. Another fun site with lots of games, quizzes, and etc., both created by editors and by other users. Slightly more oriented towards dating than Consumating, but just as filled with 20-somethings. Less of a “cooler-than-thou” vibe than Consumating. The site “gets to know you” by asking you hundreds of multiple choice questions submitted by users, and attempts to match you with people who have similar answers. Not very successfully, in my experience.
• HotOrNot.com: Pay site. Triumph of the shallow. It’s all about the pictures. You get less than a paragraph to describe yourself, and the user base does not seem terribly literate. Again, mostly people in their teens and early 20’s. HotOrNot started as a site where you rate people’s “hotness” on a scale of 1 to 10, 10 being highest. I signed up for HotOrNot in order to find out which of my photos were considered “hot” (or “least ugly”) so that I could use the higher rated pics on other dating sites. For that purpose, I highly recommend HotOrNot (and that part is free), but as a dating service, HotOrNot was an “eh” for me.
• Impersonals.com: Free site. Honestly, by far my favorite of all the dating sites, at least as far as user interface and functionality. And the questions they ask produced my favorite of all the profiles I’ve created. A really cool site, without a lot of BS, and with a good feel to it. The downside: At least in my area and demographic, very few people. And very few additions — it seems to be foundering. I’d LIKE to like Impersonals — but there’s just very few women (of any age) in my area, and most of them are in their early 20’s.
• PlentyOfFish.com: Free site. The MySpace of dating. Meaning it’s phenomenally popular, and phenomenally ugly. Good Lord, it’s ugly. If you care about user interface, web page aesthetics, or design of any kind, this site will make your eyes bleed. (As an example, they don’t crop pictures to squeeze them into a box, they just squeeze the picture into a box — so everyone looks either anorexic or like a sumo wrestler, depending on the aspect ratio of the original picture.) Still, it has the largest user pool of any site I’ve tried yet, and the users seem to include all ages. (I’m perfectly willing to date a brilliant and mature 18-year-old — but I think the odds of finding someone I’m compatible with are much better for women less than five years older or younger than I am.)
• Gk2Gk.com (Geek 2 Geek): Pay site. Decent enough dating site — not as gimmicky or Ajaxy as OKCupid or Consumating — fairly standard, in fact, with the exception that the people on it are explicitly looking for geeks. Since I am one, this site goes high up on my list. Niche dating sites are apparently a big draw these days. The matching is no more effective than on any other site — perhaps less so for me, because although I’m a computer geek, I’m not most other types of geek — and almost all of the questions are related to zeroing in on what geek hobbies and/or obsessions you have. Geek 2 Geek has one annoying quirk: If you edit your profile (even changing one character in the title), then people are not able to view your profile and you’re not able to send a message to anyone until your profile has been approved by Geek 2 Geek staff. Geek 2 Geek does generally approve your changes within 24 hours (even on weekends) but it’s still an annoying bit of suckage. (Unlike every other pay site listed here, Geek 2 Geek will NOT automatically renew your subscription — which is a nice bit of class.)
• Chemistry.com: Pay site. A service of Match.com, Chemistry attempts to do a “scientific” analysis of your personality, and match you up with compatible people. Their advertising targets eHarmony.com, a similar site that has some pretty distasteful limits. (For instance, eHarmony won’t serve gay people.) I have my doubts as to how effective the matching is, but the user-contact system they put in place seems to work very well. You get up to 10 “matches” a day, assigned to you by the system. If the system doesn’t assign you a match, you can’t look at them. (This is why I haven’t posted a link to my profile — as far as I know, I can’t.) You tell Chemistry if you’re attracted to each of the matches. If you say you are, Chemistry lets them know and lets them state whether they’re attracted to you. If they’re not interested in you, their name silently falls off your list of potential mates. Because of this, this is the first site I’ve seen where women will initiate contact with men in significant numbers. (On most other sites, the guy has to send the first e-mail — the sites let women initiate contact, it’s just that societal norms of men initiating contact in meatspace exist on dating sites as well.) Once you both agree that you find the other interesting, you answer some fairly meaningless questions about the importance of various personality traits by moving sliders to indicate importance. The other person does the same. Then, if both parties decide that the sliders match up enough, you take it to the next level. (I say the sliders are meaningless because — I mean really, if I rate “sense of humor” as important and she doesn’t, I’m not going to reject her. I’m going to try and find out if she laughs at my jokes and can make me laugh.) At the next stage, each party chooses two questions (from a list of a few dozen, or write your own) for the other party to answer in paragraph form. Each party reviews the other’s answers, and decides whether to take it to the next level, which is e-mail. After e-mailing back and forth, you can decide whether or not to meet in person. As I write this, it sounds HORRID — but perhaps because I’m a socially awkward geek, this computer-mediated dating really seems to work very well. It takes some of the initial awkwardness out of the introduction — and dating, online or in person, is awkward enough as it is.

Here’s my bug report: (with e-mail addresses slightly mangled for publication)

WHAT I DID:
1) On Edit Profile -> Contact page, selected “Add / Remove E-mails.
2) Hit “Change” under “Contact E-mail.”
3) Select the “New Address” box and enter “root at sudosu dot net”
4) Click the “Change Contact E-Mail” Button.
5) Enter my password and click “Confirm.”

WHAT I EXPECTED TO HAPPEN:
1) I expected Facebook to e-mail a confirmation message to
root at sudosu dot net.

WHAT ACTUALLY HAPPENED:
1) Facebook responded with “Please enter a valid email address.”

WHY THIS IS A PROBLEM:
1) root at sudosu dot net IS a valid e-mail address, and is my primary
personal address. People searching for me will use that address, and
will not be able to find me.

And their response:

Hi John,

Unfortunately we do not support email addresses with generic
prefixes like the email you have listed (i.e. info@, webmaster@, root@
etc.). You will need to use a personal email address that does not
contain this type of prefix. Since email addresses of this nature are
typically used for organizations and businesses, we do not allow them
to be used for personal Facebook accounts. There are no exceptions
to this rule. We sincerely apologize for any inconvenience this policy
may have caused you.

Thanks for contacting Facebook,

Landon
Customer Support Representative
Facebook

So Facebook knows me by my jschofield at gMail address. I hope this doesn’t prevent people from finding me on Facebook.

If you see “https://” in the address bar, that connection is generally secure. In many cases, you can make an insecure connection secure by adding an “s” in the address bar and hitting “enter.” (Whether or not it works depends on the site’s web server configuration. Some aren’t set up to support secure connections.)

Google redirects you to a secure https connection while you’re logging in, but sends you back to http for everything else. So if you use Google Mail (aka gMail) without doing anything to secure it, any eavesdropper can read all your mail. This is not a huge problem on a wired connection, but if you’re using any kind of wireless connection, you should be concerned — and if you use an open wireless connection, you should be alarmed.

The “add an ’s’” trick doesn’t always work with gMail, as I’ve noticed it switching back to http seemingly at random.

You can get around this by bookmarking https://mail.google.com/mail — if you start there, Google will leave the entire session protected.

Another solution is Mark Pilgrim’s Grease Monkey extension to Firefox, GMailSecure. It automatically redirects from http://mail.google.com to https://mail.google.com — nicely solving this problem. Typing just “gmail.com” in the address bar first redirects to “http://mail.google.com/mail” (because of Google) and then redirects from there to “https://mail.google.com/mail” (because of GMailSecure).

Actually, no. When you Google for “John Schofield,” I’m WAY down on the list — not even on the first page. (Some Civil War General gets all the link love.) So now I’m John Mark Schofield, which gets a LOT more Google juice. (At this time, I’m #1 on the results.)

I’ll be continuing this at the URL they suggested, and I’ll update this post.

Wednesday night my site went down. Consider it a comment on the level of service site5.com provides that this didn’t worry me too much. I knew it would be back up shortly, like it always is when Site5.com goes down. (Several times a week.) I had it on my list to move to a different hosting provider; I just hadn’t done it yet.

Then Thursday morning I get this e-mail:

From: devnull@site5.com
Subject: Subject: Site5 Incident Notification: oracle.site5.com
Date: May 24, 2007 8:53:45 AM PDT
To: [My E-mail Address]

Dear John Schofield,

My name is Todd Mitchell, I am Chief Operating Officer at Site5. I am writing you this morning in regards to the outage with oracle which began to affect you at approx. 18:00 EST, May 23rd, 2007.

At approx. 18:00 EST on May 23rd, 2007 oracle became unresponsive and as a result one of our system administrators requested a reboot for the server. A short time later, less than 10 minutes, our system administrator contacted our data center and hardware provider (Net Access Corporation / NAC) as the machine did not return to service from the unscheduled reboot. Upon further investigation by our data center, their initial determination was that the Operating System became corrupt and couldn’t initialize our disk array on boot.

At this point one of our lead system administrators decided in the best interest of our clients to begin a restoration from backup due to the inherent difficulties of restoring both a corrupted root and data filesystem. Upon making this decision, they proceeded to verify that our backups are intact and would at that point initiate our server restoration procedures. Unfortunately, after several rounds of integrity checking it was discovered that the backups for oracle are corrupt and unusable.

I am terribly saddened by this discovery. One of the core values of hosting is to first provide a quality hosting environment and secondly, ensure that data is available if a server should fail. Unfortunately due to dire circumstance, all data for this server is corrupt and completely unusable. We are in the process of returning our server to its original configuration and returning your account to its default state–your username and password remain the same. Please use them to FTP into your account and upload your local backups.

We realize how frustrating this is for you so we’re offering a 12 month service credit. This service credit is automatic (no need to request it and it will appear on your account within 24 hours) and it will be automatically applied to future invoices. Please note that this service credit has no cash value and cannot be requested in the form of cash, check and/or refund to a credit card. This service credit is, however, transferable between Site5 customer accounts. Alternatively, if you’d like to terminate your account, I have authorized our billing department to refund, back to your credit card on file, up to 12 months of previously paid for service.

This incident has been extremely trying for us. Site5 has been in business since 1999 and we’ve never experienced an issue like this–our clients have always received the best possible service from us. I simply cannot express in words how awful we feel and how heartbroken our system administrators are. We will, of course, be looking at how this happened and what we can do in the future to avoid this at all costs. I hope that we can regain your trust and patronage in some capacity.

Sincerely,

Todd Mitchell on behalf of the entire Site5 management team.

OK. So that’s completely screwed. I mean, for a professional hosting company (or any decent-sized company) not testing restores is unacceptable. It’s one of the cardinal rules of backups — make sure your backups are actually good. Making this mistake worse, it looks from the e-mail like they didn’t begin the restore process and discover their backups were corrupt — this is the classic form of this mistake. Instead, they did a backup validity check and determined (without having to perform a restore) that the backup was corrupt. That means they could have done this backup validity check at any time, without having to do a full restore — meaning there is even LESS excuse for this failure.

(In their defense, I have to say I like Site5’s handling of the issue, in terms of issuing credits or refunds. But that’s ALL that I like about this.)

Since I had already decided to leave Site5 at some point, this was an easy choice: Leave Site5 and get my money back!

From: John Schofield
To: Billing@site5.com
Date: May 24, 2007 9:23 AM
Subject: Re: Subject: Site5 Incident Notification: oracle.site5.com

We will be canceling our account with Site5. We request, per your e-
mail below, our previous 12 months of charges refunded. I expect to
see no further charges from Site5 on my credit card.

There is nothing I can say about how inexcusable a failure this is on
the part of Site 5. Words fail me.

Sincerely,
John Schofield

Then Site5, clearly determined to add insult to injury, sent this:

To: [My E-mail Address]
Subject: [Site5 #QZK-378129]: Re: Subject: Site5 Incident Notification: oracle.site5.com
Date: Thu, 24 May 2007 12:25:29 -0400
Reply-To: billing@site5.com

Hello John,

Thank you for writing. Though I am sorry to hear that you wish to cancel your account, we would be glad to assist you with any request that you may have.

As account cancellation will result in irreversible file loss, can you please reply with the following:

1. The full domain name of any account(s) you wish to cancel.

2. For security purposes, please verify either the last four digits of
the credit card on file in your account details or the physical
address that is on file in your account’s contact information.

3. Verification of a recent backup of the files. Have you been able to
download an archive of the account or any necessary files?

If you have any additional questions or concerns, feel free to contact us at anytime. Thanks again.

—
Jessica Noling
Customer Service
Site5 Internet Solutions, Inc.

http://www.site5.com

My reply:

From: John Schofield
To: Billing@site5.com
Date: May 24, 2007 9:30 AM
Subject: Re: [Site5 #QZK-378129]: Re: Subject: Site5 Incident Notification: oracle.site5.com

1. [My Domain Name]

2. [My address]

3. We are canceling our service because Site 5’s server went down,
and Site 5 did not have adequate backups in place. Our data has
already been irreversibly lost, by Site5’s incompetence. Thanks for
the thought, though.

Sincerely,

John Schofield

Site5 did refund all my hosting fees for the past 12 months, or I’d be foaming at the mouth far more than I am now. But despite them handling their failure (AFTER the fact) in a professional and responsible manner, I’m still thoroughly pissed I ended up in the situation in the first place.

I was able to restore almost all my content from backups, but some changes had been made in a DB on the site that had not been backed up recently — so I did lose some work because of this.

I’ve since set this site up (not sudosu.net; a different site) with dual hosting with two separate hosting companies, and my DNS provider doing failover between them, and automatic backup. That’s a future article.

Things I use every day: (Just about)

• twitter.com/schof
• flickr.com/photos/schof/

Things I’m not sure about yet:

• www.friendster.com/schof
• www.linkedin.com/in/schof
• www.myspace.com/johnmarkschofield
• www.socializr.com/user/schof
• http://www.facebook.com/profile.php?id=684838741
  • Note that Facebook knows me as jschofield at gmail dot com instead of my sudosu.net address. Here’s the explanation.

I have no experience with Media Temple. But Grid-Server seems like one of the brilliant ideas that are obvious once someone else invents them. Oh sure, other people have done redundant web hosting before. Thousands of little guys plus Google, Amazon, Yahoo, MySpace, etc. are dividing web-hosting duties between many computers such that if one goes, down, the others can keep the website running.

But no commercial webhosting company I’m aware of currently offers grid-based hosting. This leads to outages and slowness for clients. DreamHost has had a number of recent problems. To their credit, they are quite open and honest about what went wrong. But there’s also a constant parade of announcements of servers going down for 10 minutes here, 10 minutes there for RAM upgrades, hardware replacements, etc.

There is NO REASON I can think of why my website, even at DreamHost’s low prices, should be dependent on ONE database server, ONE web server, and ONE file server. Any one of them goes down; my website goes down.

Creating a grid of dozens of computers, each of which can shuffle around websites to balance load or handle failures is not an easy task. It would take me loads of research before I was even ready to begin that sort of project, and I’d have to hire a team of people to do it. It’s a big deal.

On the other hand, it’s been proven that this sort of thing is possible. It’s not like we need basic research or feasibility studies. You can hire people who know how to do this if you’re willing to pay for it.

It’s not like this should be a tough decision for DreamHost. One of the primary benefits of this technology is to eliminate problems with over- and under-selling. The host has a pool of resources that can be grown incrementally, as resource use grows, without worrying about the growth rates of individual sites.

Sites have fewer worries about going down, both from the Slashdot / Digg effect, and from hardware failure or misconfiguration at the hosting company.

So why isn’t DreamHost doing this? My hope is that they’re already working on it — but I wish they’d let us know.

UPDATE June 9, 2007: This site is now hosted at NearlyFreeSpeech.net. So far, I love them. (An unfortunate truism of web hosting is that all hosts are great until they start to suck.) I’ve eventually been disappointed with every host I’ve tried. I host both personal and business websites; so far, pair.com and nearlyfreespeech.net are the only hosts on my “not sucking yet” list.