Despite the over-the-top nature of the letter, posting it here and picking on it started to seem more and more like kicking someone when he’s down. There was no larger message, no lesson to learn from the letter — just me making fun of someone who made an asshole of himself.

I deleted it. I don’t want to be that guy.

My house flooded on Friday, Saturday, and Sunday. I’ve already cleaned up twice; there’s mud on my floor right now, on Monday morning. In addition, we’re late on a big project at the office. I worked 11 hours on Saturday and 9 hours on Sunday.

And we didn’t accomplish the thing that we spent all weekend working to accomplish.

If it looks like it won’t rain for a while, I may mop my floor tonight and clean up the mud for the third time.

(Oh, and my up-hill neighbor built a sandbag dike that sent a WALL of water down past (and into) my house, knocking down a fence, covering my brick patio and a small grass lawn with a layer of mud, and killing the fish in the fishpond.)

There’s very little payoff in stealing 20 servers (the amount stolen in the recent robbery) for the hardware value — this is most likely about stealing the DATA on the servers. Most people in the market for rackmount servers wouldn’t buy them off trucks, and so you’ve got a crime with serious time and very little payoff if they were stealing hardware. I wonder how much valuable data (including credit card numbers) was stored on those boxes?

Eckels took umbrage at reports that their facility had been robbed four times in the last two years:

One of the biggest mistakes is that people are talking about four robberies. A robbery means than property has been seized through violence or intimidation. C I Host has technically only been robbed twice in two years. The other two were break-ins where things were stolen, but not robberies.”

Umm. That doesn’t exactly make me feel better. Although the facility’s staff probably prefer the burglaries — their night manager was repeatedly tasered and “struck with a blunt instrument” during the most recent robbery.

Well, there’s a standard way of doing that in Linux/Unix, called setuid. You set the “setuid bit” to on, and the program will run as the owner of the program. So, since it’s owned by root, a regular user can run a particular program, and have that program run with the permissions that root has. Very handy.

But I couldn’t figure out why the program was not being installed setuid root. I could see in the postinst script that the command was valid. I could cut-and-paste the chmod line (that set the setuid bit) from the script to the command-line, run it, and it worked perfectly. And there weren’t any other commands in the postinst script that affected permissions (the setuid bit is a permission bit) for that file.

However, there WAS a chown command later in the script. (It started in a parent directory and recursed into the directory with the C program I was dealing with.) Eventually, I narrowed it down to that chown line, and once I saw chown was the cause, I guess it sort of made sense. I was able to reproduce the problem by chowning a file after setting the setuid bit.

I guess from a security standpoint, you could do a lot of stupid things by setting the setuid bit (making a program operate as the owner of the program) and then changing the owner. So, to prevent you from shooting yourself in the foot, changing ownership of a file unsets the setuid bit.

Still, this seems somewhat counter to the philosophy of Unix — first, to not do unexpected things, and second, to give users approximately an order of magnitude more rope than they’d need to hang themselves. I can’t think of many other commands that silently prevent you from doing something that MAY be stupid. Unix usually assumes that you know what you’re doing — even to the point that you can enter a command to delete every file on your hard drive, and Unix will happily delete all your files without asking for confirmation. The fact that chown has this behavior was NOT obvious to me.

(This is a highly technical rant, and I’m writing it late at night after coding for a while — if it doesn’t make any sense, it’s probably me, not you.)

I don’t understand this continuing obsession with buying things that you need to break before they do what you want. It’s not just the iPhone; people did the exact same thing with the AppleTV too… I thought the big draw for Apple hardware was that “It Just Works.” By breaking it, you must know you’re giving up the “Just Works” factor, so what’s left? Rounded corners?

My current theory is that it’s some twisted form of wish fulfillment. “I wish this company understood the value of openness, but they don’t, so I’m going to keep buying their closed, crippled shit until they get it.” Yeah, let me know how that works out for you.

Hello, Dear Hiring Manager,
My Name Is REDACTED. I Am Looking For A Stable Employment Please, I Am Currently Working Temp Jobs And I Am Kinda Tired Of It Now.
I Need Something That Will Work With Me, I Am In College And I Take Most Classes On-Line, But I Still Need To Take Some Classes / Labs On-Campus, So I Need Something That Will Work With This. Also I Am Starting Out On My Own So I Am Looking At Something Around $12-$15+ An Hour.
I Am A Former UNITED STATES MARINE And I Am Looking At Reenlisting In The ARMY NATIONAL GUARD Soon As Well!!
I Would Like Something In The Redondo Beach/Gardena/Torrance/Manhattan Beach Areas, But I Can Work With Locations. I Will Also Accept Locations In Culver City, Santa Monica, LA, And West LA Areas As Well. I Am Looking For Something Permanent Not Temporary But I Can Work With This As Well. I Have Extensive Experience In A Couple Lines Of Work. IE Customer Service, Sales, Retail, Wholesale, Food (restaurant, and Fast Food), Office, Clerical, Admin, Government, Legal, Financial, Baning, Construction, And Promotion.
I Have Attached A Sample Resume For Some Of My Previous Jobs. If You Would Like More Let Me Know.

I Am Available To Discuss Options And Set Up Interviews All Day.
Please Contact Me As I Am Immediately Available To Work!

Other than removing his name, I changed nothing. He sent the same thing to us three times over several days. None of them had a resume attached.

He didn’t get a phone interview.

There are two amazing things about this — first, that Apple’s support tech was incompetent enough to generate a label like this, and that DHL was smart enough to actually deliver it on time to the correct place.

(I won’t include his real name, but Jeff’s last name is NOT “Jones” and doesn’t sound anything like that. It’s pretty obvious the support rep just made up “Jones.”)

The actual address:

2121 Cloverfield Blvd.
Suite 205
Santa Monica, CA 90404

The address printed on the label:

2121, clover field , blvd
sweet 205 centre monca
COLORADO CA 90404

Umm. What?

Wednesday night my site went down. Consider it a comment on the level of service site5.com provides that this didn’t worry me too much. I knew it would be back up shortly, like it always is when Site5.com goes down. (Several times a week.) I had it on my list to move to a different hosting provider; I just hadn’t done it yet.

Then Thursday morning I get this e-mail:

From: devnull@site5.com
Subject: Subject: Site5 Incident Notification: oracle.site5.com
Date: May 24, 2007 8:53:45 AM PDT
To: [My E-mail Address]

Dear John Schofield,

My name is Todd Mitchell, I am Chief Operating Officer at Site5. I am writing you this morning in regards to the outage with oracle which began to affect you at approx. 18:00 EST, May 23rd, 2007.

At approx. 18:00 EST on May 23rd, 2007 oracle became unresponsive and as a result one of our system administrators requested a reboot for the server. A short time later, less than 10 minutes, our system administrator contacted our data center and hardware provider (Net Access Corporation / NAC) as the machine did not return to service from the unscheduled reboot. Upon further investigation by our data center, their initial determination was that the Operating System became corrupt and couldn’t initialize our disk array on boot.

At this point one of our lead system administrators decided in the best interest of our clients to begin a restoration from backup due to the inherent difficulties of restoring both a corrupted root and data filesystem. Upon making this decision, they proceeded to verify that our backups are intact and would at that point initiate our server restoration procedures. Unfortunately, after several rounds of integrity checking it was discovered that the backups for oracle are corrupt and unusable.

I am terribly saddened by this discovery. One of the core values of hosting is to first provide a quality hosting environment and secondly, ensure that data is available if a server should fail. Unfortunately due to dire circumstance, all data for this server is corrupt and completely unusable. We are in the process of returning our server to its original configuration and returning your account to its default state–your username and password remain the same. Please use them to FTP into your account and upload your local backups.

We realize how frustrating this is for you so we’re offering a 12 month service credit. This service credit is automatic (no need to request it and it will appear on your account within 24 hours) and it will be automatically applied to future invoices. Please note that this service credit has no cash value and cannot be requested in the form of cash, check and/or refund to a credit card. This service credit is, however, transferable between Site5 customer accounts. Alternatively, if you’d like to terminate your account, I have authorized our billing department to refund, back to your credit card on file, up to 12 months of previously paid for service.

This incident has been extremely trying for us. Site5 has been in business since 1999 and we’ve never experienced an issue like this–our clients have always received the best possible service from us. I simply cannot express in words how awful we feel and how heartbroken our system administrators are. We will, of course, be looking at how this happened and what we can do in the future to avoid this at all costs. I hope that we can regain your trust and patronage in some capacity.

Sincerely,

Todd Mitchell on behalf of the entire Site5 management team.

OK. So that’s completely screwed. I mean, for a professional hosting company (or any decent-sized company) not testing restores is unacceptable. It’s one of the cardinal rules of backups — make sure your backups are actually good. Making this mistake worse, it looks from the e-mail like they didn’t begin the restore process and discover their backups were corrupt — this is the classic form of this mistake. Instead, they did a backup validity check and determined (without having to perform a restore) that the backup was corrupt. That means they could have done this backup validity check at any time, without having to do a full restore — meaning there is even LESS excuse for this failure.

(In their defense, I have to say I like Site5’s handling of the issue, in terms of issuing credits or refunds. But that’s ALL that I like about this.)

Since I had already decided to leave Site5 at some point, this was an easy choice: Leave Site5 and get my money back!

From: John Schofield
To: Billing@site5.com
Date: May 24, 2007 9:23 AM
Subject: Re: Subject: Site5 Incident Notification: oracle.site5.com

We will be canceling our account with Site5. We request, per your e-
mail below, our previous 12 months of charges refunded. I expect to
see no further charges from Site5 on my credit card.

There is nothing I can say about how inexcusable a failure this is on
the part of Site 5. Words fail me.

Sincerely,
John Schofield

Then Site5, clearly determined to add insult to injury, sent this:

To: [My E-mail Address]
Subject: [Site5 #QZK-378129]: Re: Subject: Site5 Incident Notification: oracle.site5.com
Date: Thu, 24 May 2007 12:25:29 -0400
Reply-To: billing@site5.com

Hello John,

Thank you for writing. Though I am sorry to hear that you wish to cancel your account, we would be glad to assist you with any request that you may have.

As account cancellation will result in irreversible file loss, can you please reply with the following:

1. The full domain name of any account(s) you wish to cancel.

2. For security purposes, please verify either the last four digits of
the credit card on file in your account details or the physical
address that is on file in your account’s contact information.

3. Verification of a recent backup of the files. Have you been able to
download an archive of the account or any necessary files?

If you have any additional questions or concerns, feel free to contact us at anytime. Thanks again.

—
Jessica Noling
Customer Service
Site5 Internet Solutions, Inc.

http://www.site5.com

My reply:

From: John Schofield
To: Billing@site5.com
Date: May 24, 2007 9:30 AM
Subject: Re: [Site5 #QZK-378129]: Re: Subject: Site5 Incident Notification: oracle.site5.com

1. [My Domain Name]

2. [My address]

3. We are canceling our service because Site 5’s server went down,
and Site 5 did not have adequate backups in place. Our data has
already been irreversibly lost, by Site5’s incompetence. Thanks for
the thought, though.

Sincerely,

John Schofield

Site5 did refund all my hosting fees for the past 12 months, or I’d be foaming at the mouth far more than I am now. But despite them handling their failure (AFTER the fact) in a professional and responsible manner, I’m still thoroughly pissed I ended up in the situation in the first place.

I was able to restore almost all my content from backups, but some changes had been made in a DB on the site that had not been backed up recently — so I did lose some work because of this.

I’ve since set this site up (not sudosu.net; a different site) with dual hosting with two separate hosting companies, and my DNS provider doing failover between them, and automatic backup. That’s a future article.

This is, quite frankly, a little terrifying.

U.S. citizens suspected of terror ties might be detained indefinitely and barred from access to civilian courts under legislation proposed by the Bush administration, say legal experts reviewing an early version of the bill.

…

According to the draft, the military would be allowed to detain all “enemy combatants” until hostilities cease. The bill defines enemy combatants as anyone “engaged in hostilities against the United States or its coalition partners who has committed an act that violates the law of war and this statute.”

Legal experts said Friday that such language is dangerously broad and could authorize the military to detain indefinitely U.S. citizens who had only tenuous ties to terror networks like al Qaeda.

…

Scott L. Silliman, a retired Air Force Judge Advocate, said the broad definition of enemy combatants is alarming because a U.S. citizen loosely suspected of terror ties would lose access to a civilian court — and all the rights that come with it. Administration officials have said they want to establish a secret court to try enemy combatants that factor in realities of the battlefield and would protect classified information.

The administration’s proposal, as considered at one point during discussions, would toss out several legal rights common in civilian and military courts, including barring hearsay evidence, guaranteeing “speedy trials” and granting a defendant access to evidence. The proposal also would allow defendants to be barred from their own trial and likely allow the submission of coerced testimony.