I hike and do martial arts training regularly, and consider myself in reasonably good shape. However, on Monday I started an exercise program with a local Crossfit affiliate, Team Crossfit.

This was an intense workout, but I’ve done intense (worse) workouts in the past, with no ill effects other than soreness. We did jumping pull-ups (where you jump up to a “completed” pull up position, then lower yourself as slowly as you can. This did a fair bit to blow out my arms and upper back, as those muscles were not used to that kind of exertion. After about 20 minutes doing jumping pull-ups, they switched from the skills portion to the “timed workout” portion, and added wall-balls (I then threw a 14-pound medicine ball 10 feet up a wall) and staying in the plank position for a while.

(The routine was supposed to be 20 jumping pull-ups, 20 wall-balls, and 1 minute in the plank position — with the whole sequence repeated five times.)

My trainer saw that I was straining, and cut the routine in half for me. (Ten pull-ups, ten wall-balls, and 30 seconds in plank position.) I ended up stopping at the end of round 4.

I want to emphasize that this did NOT seem like a dangerously high amount of exercise to either me or the Crossfit trainer. I lay no responsibility for this on Team Crossfit. This is actually the second time I’ve taken Crossfit (I went for a month or two over the summer) and I had harder workouts then that left me more sore than this one, with no ill effects. Nevertheless, it’s obvious to me now that I should have started up this exercise regime more gradually. That’s mistake number 1.

Mistake number 2 was insufficient hydration. I drank what I thought was enough after my workout on Monday, and then stopped — after about 25 ounces of water. After that I just drank normally. On Tuesday, I didn’t work out, and I had a cup of coffee, and about five cups of tea — that’s it. In hindsight (hell, even at the time had I bothered to think about it) this was vastly insufficient hydration.

Mistake number 3 was taking Advil for my muscle pain. This would normally be a good thing, but I learned that Advil has an inhibiting effect on the kidneys — it closes down the tubes in the kidneys through which liquid flows. This is perfectly fine if you have a headache, but if the kidneys are trying to remove the poisonous effects of excess exercise at the same time you’re taking Advil, it compounds the problem.

The result of those three mistakes was at least one night in the hospital for me, as the doctors struggle to treat my Rhabdomyolysis.

My WHAT?

Tuesday night, after work, I urinated when I got home and saw that my urine was very dark in color, like a flat can of cola.

The materials that CrossFit gives out when you start includes a warning that, if your urine looks like Coca Cola, you should go immediately to the emergency room. I called @dakimjerry, hoping that he would talk me out of it, and he told me to go to the ER. Shit. I called Kate S., hoping the same, and she told me to go to the ER. Shit. I went to the Providence-Tarzana ER.

There they found blood in my urine (not actually blood, but myoglobin, a result of muscle cells dying and breaking up). Microscopic examination of my urine showed no blood cells, so it wasn’t blood, but muscles breaking down. They ran my levels of Creatine kinase (a muscle enzyme you would only see in the blood if muscles were breaking down) and they came back abnormally high. (The test didn’t specify the level.)

The ER doctor ended up giving me two liters of intravenous (IV) saline, and sending me home with instructions to see my own doctor.

Just when I got home at 2 AM, the ER Dr. called, and said he’d changed his mind and would like to see me at UCLA Medical Center ER the next morning. He’d run some tests, and if they showed my levels getting better, I’d be fine. Otherwise I’d be admitted to the hospital.

This seemed like a good plan, and I showed up at the ER the next morning.

There I was immediately plugged into another IV saline drip, and they got blood and urine for testing. The results of these tests were … interesting.

My kidney function was fine — A-OK — but my Creatine kinase level, which should be between 60 and 400, was at 136,000.

Holy fucking shit. Really?

A doctor told me that normally this level of Creatine would mean a transfer to intensive care, but because I seemed to be suffering no discomfort, pain, or ill-effects, they would simply put me under observation.

They admitted me to the hospital, and began pumping a liter of IV through me an hour. The idea is to wash the myoglobin and Creatine kinase out of my system before they can damage my kidneys.

As of about 8 PM, my Creatine kinase level was down to about 122,000 — still insanely high, but better — and I’m peeing like crazy (and collecting my pee so they can measure it).

That’s the news so far — I still feel fine — other than muscle soreness have felt fine throughout this whole thing — and am staying in a nice hospital with decent food and free wifi.

You Learn Who Your Friends Are:

I have to say I have been incredibly touched by the outpouring of support from friends and family and coworkers. I’ve been entirely clear-headed and rational about this the whole time I’ve been writing this, but now that I’m talking about the people who’ve reached out to me, I’m getting teary-eyed and emotional.

Kate brought my Mom (who can’t drive) down to the hospital, and came with far more water and juice and cliff bars and good chocolate than I need; Jon made me the best geek care-package ever, including a laptop and an iPhone charging cable; Tegan delivered it and stayed to talk and tell me funny (and sometimes slightly disturbing) medical stories.

People have been calling, texting, and emailing to wish me well; the outpouring of support from old friends and new has been far more than I expected; I’m incredibly lucky in far more than a medical sense.

My phone doesn’t work (I may be in a basement) but the regular phone by my bed is 310-267-9533 310-267-9423. Feel free to call or stop by.

Right now I’m just a machine for turning IV fluids into urine, but I’m a damn lucky urine-producing machine, and I’m honored to be friends and loved ones with all of you.

Update Thursday, December 17, 6:15 AM:
In the middle of the night I started having shortness of breath, and we figured it was because we had put so much liquid in my body so quickly it was entering my lungs. From 11 AM to midnight I was on 1 liter an hour; at midnight they switched me to 500 milliliters an hour. After my shortness of breath, we slowed the IV down to 0.2 liter an hour, and put me on oxygen and a oxygen sensor (on my finger at the moment) and I’m now feeling better and my oxygen level is back up in the high 90’s. (It dropped as low as the high 80’s.)

Update Thursday, December 17, 8:10 AM:

Saw my doctor (one of them) just now; my Creatine is down to 90,000, which is good, but they can’t run the IV through me as quickly as they’d like because of the fluid in my lungs. (Fluid is MUCH better since they slowed the IV, and I now have very minimal shortness of breath. Biggest problem is the damn O2 sensor on my finger keeps me from touch typing.

I asked if I’d be out today; his answer was that if it was today, it would be late in the evening. Sounds like a “no” to me. Crap.

Update Thursday, December 17, 9:10 AM:

Now I have a target — they’ll release me when my Creatine level is below 10,000. Also, my lungs are clear, and I’ve ditched the oxygen and the O2 sensor. Sweet! I can touch-type!

Update Thursday, December 17, 8:15 PM:

OK, we’ve got a correction, there was some confusion with what the earlier doctor told me. The original ER Dr. called to see how I was doing (I think he’s a genuinely nice guy, but I also think I’m an interesting case to him; most people with my CK levels present much worse.) and mentioned it was too bad my CK levels had gone back up. Say what?

I later confirmed what he told me with a nurse in the observation ward. At 1 AM this morning, I had a 90,000 CK level. (Again, under 400 is normal, and under 10,000 lets me go home.) However, about that time, I started getting fluid in my lungs, and we reduced me from 500 milliliters per hour to 200. That allowed my body to move the water out of my lungs, and made me breathe better. But it also allowed my CK levels to creep up. They were measured at 103,000 at 5 AM this morning.

I’ve been at 350 milliliters per hour of saline IV drip most of the day, and my 5:30 PM CK reading (I’m now getting blood drawn twice a day instead of every four hours) was 89,000.

I’m definitely not going home tonight. I’m kind of doubtful I’ll get home tomorrow, and this may mean we’re moving to something more aggressive than IV fluids. (This is me saying this; no doctor has said so.)

Good news is I’m moving to a private room, still in Observation, but I expect I’ll sleep better without people being moved into and out of beds right next to mine all night long. (Not that I’m eager to move, but I’m all packed up – all belongings in bags and etc. except this laptop.)

Update Thursday, December 17, 10:50 PM:

I’m in the new private room. This is excellent. (New phone is 310-267-9423.)

My left arm is swelling up like Hellboy (maybe there’s something to this super-hero theory after all) so they switched the IV to my right arm. This is not excellent. I used to be a phlebotomist’s dream; now I’m apparently a nightmare. Took two RNs and 40 minutes to get an IV in. My whole body is swelling up with fluids. I’m the fucking Pillsbury doughboy. (OK. Not quite that bad, But they estimated this morning that they’ve given me 14 liters of IV fluid and I’ve urinated 7.) They figured that I’d put on about 10 pounds of water weight. (Leaving the “period” jokes to those more qualified.)

I’m going to CLOSE THE DOOR and go to sleep — what a wonderful concept. Goodnight, all.

Update Friday, December 18, 8:17 AM:

Just spoke with my doctors and got my CK results. CK is 68,118; plan is the same — flood me with IV fluids until it goes down. Doctor said I may be here a few more days.

Update Friday, December 18, 9:12 PM:

Lovely day in the Observation suite — got to walk outside twice (got my car from temporary parking and drove it to the hospital parking, and did not have a ticket!) and took a shower.

You would not believe, by the way, the looks you get while walking down a public street in a hospital gown. I think people thought I was an escaped mental patient — at least one person asked me if I was one, and lots of others stared.

Vanessa stopped by with snickerdoodles, Heather L. stopped by with a get-well card from everyone at work, and more Gatorade and candy than any sane man needs, and then Kate S. stopped by with my Mom to visit for a while. All of that was wonderful; a great way to spend a Friday evening (if your Friday evening had to be spent in a hospital). (And I gave Kate my house key; she’ll be coming back tomorrow with movies and clothes.)

All that’s the good news. The CK results from my 6 PM blood draw were 54,574. Yesterday I went down 20,000; today I went down 13,500. It’s good that I’m going down, but I’m frustrated that it went down less today than it did yesterday. We’ll see what tomorrow morning’s reading is. I was thinking I’d be out of here by Sunday or Monday; now I’m not so sure.

Update Saturday, December 19, 8:19 AM:

Great news! CK results from 4:42 AM this morning were 32,147. That’s a loss of 35,971 in the last 24 hours, and means that I’ll almost certainly be going home tomorrow.

Haven’t spoken with doctors yet; I’ll update when I do.

Update Saturday, December 19, 3:26 PM:

Went for a walk this morning; a one-hour walk around the UCLA campus. It was beautiful. I did get stared at a lot and asked if I needed help. What? You’ve never seen a man in sneakers and a hospital gown with an IV in his arm going for a brisk walk?

@dakimjerry came by and visited for a while; it was great hearing about his recent adventures on the road.

Kate S. is coming by later today after stopping of at my house; she’s going to bring CLOTHES. And SOCKS. And DVDs. And my TOOTHBRUSH. (I’ve been using a flimsy one from the hospital that bends; it’s bristles on the end of a drinking straw.)

Because I’m swelling up with liquids the doctors have given me a diuretic; my “output” was about 1.8 liters in an hour. Think about the size of a 2-liter soda bottle for a second…

No idea when I’m leaving. I’m getting closer to the magic 10,000 number, but the doctors say this won’t be linear — some days I’ll drop a lot, and some days I won’t. I handled my parents’ Medicare Part D drug plan selection over the Internet, and ordered some groceries to be delivered for them. (They can’t drive, or lately, use a computer.)

Update Saturday, December 19, 6:15 PM:

My CK level from 4:30 PM today was 27,221! Getting down there!

Update Sunday, December 20, 10:20 AM:

Just got back from a one-hour walk to the UCLA sculpture garden, and then a shower and fresh clothes. Heaven!

Even better, I spoke with a doctor, and my CK levels are down to 14,000! Wohoo! They’re taking me off the IV and testing to see whether it continues to come down by oral hydration. (They’re not, unfortunately, taking the IV out of my arm yet — they’re just not hooking it up to the bag.)

The doctor thinks it’s likely I’ll go home tomorrow!

Update Sunday, December 20, 5:22 PM:

My 4 PM blood test came back with a CK level of 11,096! I should be out of here tomorrow for sure! I’ll probably be under 10,000 by my 5 AM blood draw!

Update Sunday, December 20, 7:14 PM:

Just spoke with my night nurse; he said all the nurses are really excited about my case — they’ve never seen a rhabdomyolysis case with numbers this high. Normally there’s serious kidney damage at around 50,000 — my kidneys are fine, and my highest level measured was 136,000. Still crossing my fingers I’ll go home tomorrow.

Update Monday, December 21, 10:30 AM:

I was just told that my CK level from this morning was 6000. I’ll be released as soon as the attending physician sees me and signs off on it, and I’ll need to drink a liter of water a day.

WHOOOOOOOOOOOOOOOO!

The modification date the script reported was correct, but the creation date the script reported was different from the creation date that doing “Get Info” in the OS X Finder reported. Hmm. Not good. I did some investigating.

Let’s look at a test file. The Finder reports the following creation and modification dates:
Created: Monday, April 10, 2006 17:04
Modified: Thursday, April 13, 2006 15:54

Python reports a different creation date and the same modification date:

>>> import os.path
>>> import time
>>> stamp = os.path.getctime('datetest.txt')
>>> time.ctime(stamp)
'Fri Apr 10 15:51:10 2009'
>>> stamp = os.path.getmtime('datetest.txt')
>>> time.ctime(stamp)
'Thu Apr 13 15:54:43 2006'

Hmmm. What does “ls” in the Terminal report? The standard “ls -l” reports creation date. Adding “-T” makes it report the full date in all cases. And “-c” counter-intuitively means “display modification date.”

$ ls -lT datetest.txt
-rwxr-xr-x@ 1 schof  schof  2448091 Apr 13 15:54:43 2006 datetest.txt
$ ls -lcT datetest.txt
-rwxr-xr-x@ 1 schof  schof  2448091 Apr 10 15:51:10 2009 datetest.txt

The plot thickens. The modification date matches the Finder and Python, but both Python and ls are reporting an incorrect (according to the Finder) creation date.

Quite a bit of Googling showed me the “mdls” tool — or “metadata ls.” Very useful. This shows the complete set of metadata for a file. Including the OS X creation date. Actually, two creation dates, one for the file, and one for the file content. (I’m not sure what circumstances would make those creation dates differ. The documentation I’ve been able to find has been unclear and contradictory.)


$ mdls datetest.txt
kMDItemContentCreationDate     = 2006-04-10 17:04:34 -0700
kMDItemContentModificationDate = 2006-04-13 15:54:43 -0700
...
kMDItemFSContentChangeDate     = 2006-04-13 15:54:43 -0700
kMDItemFSCreationDate          = 2006-04-10 17:04:34 -0700
...
kMDItemLastUsedDate            = 2009-04-11 11:48:03 -0700
kMDItemUsedDates               = (
2009-04-11 00:00:00 -0700
)


Now that we’ve got all that information, what does it tell us? The Unix/Linux standard for “creation date” is to show you the date on which a particular file was created. If you copy file “a” to file “b,” those are two different files, and the “creation date” for file “b” will be the date you made the copy.

OS X metadata travels with the file, so if you copy file “a” to file “b” using ditto on the command-line or using the Finder, the Unix creation date will be the date the copy was done, but the OS X creation date of file “b” will be the same as file “a.”

There’s good arguments for handling “creation date” the Unix way, and there are good arguments for doing “creation date” the OS X way, but mixing them as OS X does is kind of frustrating.

I’ve written a quick-and-dirty Python example script that reports the Unix creation date and the OS X creation date for any particular file. Since it’s released under the open-source MIT license, feel free to use it in your own programs. You can download it here: http://www.sudosu.net/getcreationdate.safe.

Google took me to an Apple Knowledge Base article that is out of date: http://support.apple.com/kb/HT2319

This article states that you should, from the terminal, run “/etc/daily” to run the daily maintenance tasks. That may have been true before LaunchD took over from cron, but it’s not true anymore. There is no /etc/daily file on my 10.5 system.

The other option they suggest, “sudo periodic weekly” did work, and generated the apropos index files.

The problem: I have a process that runs on a remote server for quite a while. I wish to be alerted when it completes. As long as it’s running, it creates a lockfile in /var/lock. So when the lockfile goes away, I want to be alerted.

The solution:

Part 1: In my .bashrc file, I’ve aliased the command “alert” to play a sound:

alias alert=’aplay /usr/share/sounds/k3b_success1.wav’

But of course, I can only play that on my local machine — does me no good to sound an alarm on a computer in the server room, even if it had speakers.

Part 2: I can create a “while” one-liner that waits until the file is deleted, then executes the next line:

while [ -e ~/lockfile ]; do sleep 10; done; alert

However, this can only test against files on my local system. This brings me to…

Part 3: You can use ssh to run a command on a remote system, instead of just logging into the remote system:

ssh user@example.com “while [ -e /var/lock/program.lock ]; do sleep 10; done” ; alert

It runs the while loop on the remote system, not giving back control to the local system until the while loop exits (which happens when the lock file no longer exists). Once the while loop exits, ssh exits, and we run the “alert” command, which plays a sound alerting me to the change.

Took me about 10 times longer to write up than it did to implement. But it’s the sort of thing you can just throw together quickly on a *nix system to make your life easier.

If you read the other books in the field, will you derive most of these ideas yourself? Probably. But I still got a few good techniques from this book — among them, a new approach to what BCD calls my “Project Portfolio” — my lists of projects, tasks, and who they’re assigned to.

Worth reading. Not worth reading first, but worth the time and money spent on it.

The first is “Getting Real” by 37signals. If you’re doing web-application development, I’d say this is a book you should read. I’m not doing that, and thus, didn’t see a whole lot of value here. I’m making a mental note to reread this if I ever do get into a web development project, and it did make me think about ways I could make our current development procedures more agile — but if the question is “Was this worth the time it took for me to read it?” then the answer is “No.”

The second is “The Mythical Man Month” by Frederick Brooks. It came highly-recommended in every software management book I’ve read, so I figured it had to be on my list. While there is much still of value in this book, easily a third to a half seems completely out-of-date — advice on whether to use printed manuals or microfiche that seems as applicable as a discussion of which tools are best for shoeing horses.

While the concepts discussed in TMMM were clearly revolutionary, I’m not sure it makes the first cut for a software management reading list. Most of the ideas it contains are captured better in other, more current books, particularly “Peoplware,” “Joel On Software,” and “Managing Humans.”

First, and best is Peopleware, by Tom DeMarco and Timothy Lister. This is another one of those must-read books. I’ve been telling everyone that I can’t believe I’ve been allowed to have the job I have without reading it. If you are in the business of software (or in the business of managing any type of “knowledge worker,”) you must read this. Really. I’m buying copies for all the managers and execs at work.

Second is the second “Joel On Software” book, “More Joel On Software,” by Joel Spolsky. I wasn’t as wowed by this one as I was by the first, but it’s still very much a worthy read. Most of the articles in the first one were new to me, while I had already read most of the 2nd one’s content on Joel’s website. Still definitely worth reading.

(Shortest review I’ve ever written.)

While I don’t agree with everything Joel has written, I find most of what he writes to be a clear exposition of common-sense approaches to managing software development. No BS, just simple stuff that works. A small percentage of “Joel On Software” is mind-changing, opening your eyes to new ways of doing things or new ways of thinking about old software.

Highly recommended.

I of course already have passwordless SSH set up to connect to that server, and use ssh-agent to store the passphrase for my SSH key, as described in another entry. However, I could not get passwordless SSH to work from a cron job. If anyone has tips on how to do that, I’d love to hear it.

So I created a second SSH public/private key pair with no passphrase, (in ~/.ssh/backup and ~/.ssh/backup.pub) and figured out how to have Duplicity call ssh, scp, and sftp with the correct parameters to specify the new key pair. (There’s about a zillion different ways of specifying that, and only ONE that works across all three programs.)

The passphrase mentioned here is the one used to encrypt the duplicity backups.

PASSPHRASE=’YourPassphraseGoesHere’ duplicity \
–no-print-statistics \
–ssh-options “-oIdentityFile=/home/schof/.ssh/backup” \
/home/schof \
scp://johnmarkschofield@example.com/duplicity

On the other hand, I’ve met exactly ONE of the type he’s talking about in my career. Not sure where that leaves me as a managing geek who needs to hire brilliant geeks.

Building the VMware VmPerl Scripting API.

Using compiler “/usr/bin/gcc”. Use environment variable CC to override.

Unable to compile the VMware VmPerl Scripting API.

********
The VMware VmPerl Scripting API was not installed.  Errors encountered during
compilation and installation of the module can be found here:
/tmp/vmware-config4

You will not be able to use the “vmware-cmd” program.

Errors can be found in the log file:
‘/tmp/vmware-config4/control-only/make.log’

When you look at make.log, you see a series of errors like this:

In file included from VmPerl.xs:6:
/usr/lib/perl/5.8/CORE/perl.h:420:24: error: sys/types.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:451:19: error: ctype.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:463:23: error: locale.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:480:20: error: setjmp.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:486:26: error: sys/param.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:491:23: error: stdlib.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:496:23: error: unistd.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:776:23: error: string.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:925:27: error: netinet/in.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:929:26: error: arpa/inet.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:939:25: error: sys/stat.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:961:21: error: time.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:968:25: error: sys/time.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:975:27: error: sys/times.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:982:19: error: errno.h: No such file or directory

The answer is simple:

sudo apt-get install libc6-dev

Now Google has the answer.

Ssh-agent is a great program that lets you add the password to your SSH private key to memory, and then you don’t need to type in the ssh key passphrase every time. The basic usage is that you start BASH as a child of ssh-agent, and then use a program called ssh-add to prompt you for the password and store it in memory.

On OS X, there’s a GREAT program called SSHKeychain that handles this, storing the password in your OS X keychain, so it’s really seemless.

On Linux, you need to type in “ssh-add” manually every time you want to store the key, and after that your SSH sessions will be seamless.

However, I’m always forgetting to do that, and thus getting prompted for the password. Too many seams. I added the following code snippet to the end of my .bashrc file, and thus, every time I open a bash shell, it checks whether ssh-agent has any keys in memory. If it does, the shell starts as normal. If ssh-agent doesn’t have any keys in memory, it prompts you for the password. Simple, and as seamless as I can make it.

## Add key to ssh-add if it has not been added.

ssh-add -l &> /dev/null
SSHADDRESULT=$?
if [ "$SSHADDRESULT" -ne "0" ]; then
ssh-add
fi

UPDATE 2008-07-02: Here’s a much more succinct way of writing that:

ssh-add -l &>/dev/null || ssh-add

However, there’s some talk around Apple’s projections of selling 10 million iPhones in a year that are starting to bother me.

Regardless, Macworld’s Jason Snell has already detailed how Apple will likely beat the 10 million mark for calendar 2008. So the question really is just by how much.

Frankly I don’t care if Apple beats their projections or not. This won’t affect the quality of the iPhone, the quality of other Apple products, or anything else I care about. I’m not a cheerleader, I’m someone who wants to use products that don’t suck. And as long as Apple’s products suck less than anyone else’s, I’ll use Apple.

If you’re doing massive web stores in Java — well, you’ve probably already read this. If you haven’t I’d say this book is a must-read. Of course, my projects aren’t web-scale, aren’t e-commerce, and don’t use Java, and to this point, haven’t — so do take my comments with whatever amount of salt you feel is appropriate.

Even if you’re doing shrink-wrap software, I’d say “Release It!” has something of value for you — it appears to be a collection of hard-won, common-sense techniques for creating what Nygard calls “cynical software” — software that assumes that itself, and the other software it links to will all fail at one time or another, and is prepared for that failure. A number of failure modes spring to mind, but Nygard enumerates them, the anti-patterns that create them, and the patterns you can use to respond to them with exceptional clarity and obvious expertise.

In addition to being chock-full of good ideas and interesting concepts, it’s also filled with entertaining war stories about various failures and how Nygard and his team responded to them. And for a systems geek, there’s few more satisfying ways to spend an hour than reading about the nightmare series of cascading failures that ruined someone else’s weekend.

Highly-recommended.

The success of our company depends on my performance. If I do my job poorly, our company could go broke, and all our employees could be out of work. (Lest you think I sound grandiose, there’s a number of other people at our company who can make the same statements. I’m not unique; our company depends on the efforts of a number of very talented people.)

Our products improve the quality of life for thousands of seniors across the country, and there’s other groups of people I can’t talk about yet who may also start being helped by my company. If I do my job well, their lives get better, and if I don’t, their lives don’t improve as much.

Now, this is not life-or-death. I’m not a doctor or a  pilot, and nobody’s going to die if I mess up. But quality of life is important — in the best case, our products could give a grandmother a few more months or years of recognizing her grandchildren. In the worst case, our work brings a smile and a slight lift to a senior’s day.

So those are the stakes, and I have a challenging job I care deeply about. Where’s the problem?

My crisis of confidence is that I’ve realized recently that I’m just not smart enough, and I know there’s nothing I can do to get smarter. I reached the limits of what I could do with BASH programming — so I taught myself Python, and began using that instead. Almost everything that I do involves editing text of one kind or another, on multiple platforms. After working with Vim and TextMate, I’ve settled on Emacs, and I’m working hard to master it. It’s available on every platform I use, and becoming an expert will reward the effort I put in. I spend my free time reading technical or management books, trying to improve my skills. I can make myself both more efficient and more effective. But I can’t make myself smarter.

The answer to all this is to simply to do the best that I can, accept that I’m doing the best that I can, and move on. Usually I tell myself that it’s enough to make our procedures and our product a little better every day. And usually that helps. Not tonight.

If I were to suddenly have a 100-fold decrease in my job performance, could that torpedo the company? Yes. (Again, there’s other people at work who could say the same thing.) What effect could a 100-fold increase have, both for the company and the users of our products?  Hard to imagine, but it would be significant. But I don’t see how a 100 times increase is possible for me — I’m already busting my ass to get to where I am.

I’m not beating myself up because someone else attacking the same problem could do it 10% better than I could. Someone else (I don’t know who, but let’s imagine) might do it ten-thousand times better. There’s no fixed endpoint, no race I have to finish. Part of this job is defining the endpoint, and a flash of inspiration could put it somewhere many orders of magnitude better than anything I can do.

I want to improve by orders of magnitude, and the best I can come up with is tacking on a few percentage points of improvement here and there.

Now realistically, I may not be the best, but I’m the best Dakim has, at least for my position. And Dakim would have a hard time replacing me if I left. But if there were some real rockstar to take my place (and people significantly better than me do exist), that rockstar could make my contributions look puny, and could take our product much farther than I’ve been able to.

Where does that leave me? It leaves me going to work in the morning, continuing to bust my ass and do the best I can.  And hopefully, able to forgive myself for my best being what it is.

for afile in /home/auser/adirectory/*; do echo $afile; done

Useful for all kinds of things. And I can never remember how the syntax changes between doing a for statement in a bash script and doing it as a one-liner on the command line.

First, lose the headphones. That didn’t affect our decision at all, but I could imagine it affecting it at other companies.

Second, you’re obviously intelligent and capable. You messed up on the unit test, but I’m attributing that to interview nerves – I’m reasonably certain you could handle the technical demands of the job. But that’s only the first hurdle.

The next is personality and fit with the company — and this is a sticky one. In an interview, I’ve got about an hour to decide if you’re the kind of person I want to spend the next three or more years with, in relatively close quarters. Can you argue a position strongly, but lose gracefully? You may be right about the technical merits of something, but the business reality means we have to do something else — can you live with that? On a good day, we go two steps forward and one step back — bad days it’s one step forwards and two steps back. Can you deal with that frustration? I received certain cues from you — interrupting, hitting the table during the coding test, etc. — that you might have been hard to work with. Is that true? I can’t tell in an hour. But saying “no” to a candidate who would have been terrific is a smaller risk than saying “yes” to a candidate who turns out to be a bad hire.

I hope you appreciate my honesty in writing this, rather than brushing you off with “not a good fit.” I did so because I think your question about your interview performance is sincere, and I’m hoping my answer helps.

If it’s not obvious from reading the e-mail, Rands has been a big influence on my management style.

I did share my own story — the story that kept me up Sunday night. I told parts of it to other people on Monday, and I wrote the whole thing down tonight — bright light slays demons, and I don’t want this in my dreams anymore.

You can read it here: http://www.sudosu.net/essays/no-more-guns/ .

@ 43:48

“We learned early on that much of the focus in infection prevention was on how you place the catheter. What we learned through our observations (and borrowing the processes from Toyota) was it was in maintaining the catheter. Once it was in, how do you take care of it? And what we initially discovered was, there weren’t reliable processes in place by which nurses and others took care of catheters. So now it’s very unambiguous, very clear to everybody where the catheter is in the person’s body, what the condition of the catheter is every day. There’s a specific person responsible each day for looking at that catheter site and making sure it’s intact. And any time there’s a question, that’s immediately kicked up to a higher-level person who can make a decision about whether the catheter needs to come out or not.

“So identifying a highly-reliable, unambiguous process in which everyone’s responsibility is made clear creates a much more likely defect-free process during the course of that catheter being in place.”

It’s not that Shannon tells us anything really new here. It’s that he talks about one aspect of what I’m trying to do at Dakim, and comes up with something better than a trunk of Strunks:

“A highly-reliable, unambiguous process in which everyone’s responsibility is made clear.” Damn right.

Despite the over-the-top nature of the letter, posting it here and picking on it started to seem more and more like kicking someone when he’s down. There was no larger message, no lesson to learn from the letter — just me making fun of someone who made an asshole of himself.

I deleted it. I don’t want to be that guy.

(Let’s not forget, I started at 255.)

And I’m up to 8 miles of strenuous hiking — did my house to Parker Mesa last weekend; doing a 7-miler with 1400 feet of elevation gain next weekend.

And I think it’s time to get back into Krav Maga. I’m in good enough shape now that I could get through a session without dying.

Me: I’m sorry, Dakim has a policy of not hiring from recruiters. Have a nice day.

Him: John John John JOHN! This guy was a .NET star at Disney! You’ve got to talk to him!

Me: We don’t use .NET.

Him: We’ve got a GREAT Java programmer from BofA.

Me: We don’t use Java. Goodbye.

I’ve had recruiters try every scummy thing they can think of — a common tactic appears to be lying to the receptionist (telling her you’re a client) to try to get through to me.

We’re rethinking the policy of not hiring from recruiters (although we’re still in the air), so I thought up a way I can deal with them without wasting a whole lot of my time. I’ll update you later as to how well it actually works. (If this appears a little harsh, remember that this is NOT recruiters we’re hiring to help us find candidates — this is recruiters cold calling because they saw our ad on Craigslist. When they call, I’ll simply ask for their e-mail, and get off the phone and send them this.)

Dakim, Inc.
2121 Cloverfield, Suite 205
Santa Monica, CA 90404

Dear Recruiter:

Thank you for your interest in providing recruitment help to Dakim. We welcome your involvement in making Dakim a better, bigger company.

However, to avoid wasting either of our time, we’ve developed a few ground rules that make the recruiter/client relationship go more smoothly.

1) All initial communication will be via e-mail. I’ll call you when it’s time to get on the phone.

2) All e-mails providing information on a candidate MUST also provide the fee your company will take if we hire the individual.

3) Dakim has no problem finding decent candidates. We have a serious problem finding spectacular ones. You may send us up to three candidates, period. (If we make one of them an offer, the count resets to zero.) But if you’ve sent us three jerks, you’re obviously using the shotgun approach, and it’s a waste of your time, our time, and the candidates’ time.

The positions currently open are:
*
*
*

Thanks very much for your interest in recruiting for Dakim!

Sincerely,

John Schofield
Director, Information Technology

I need to start going to conventions, user group meetings, etc. — immediately I’m going to start attending the Los Angeles Linux User Group meetings and the local Python SIG. I’m not sure what else I’m going to do, but it’s becoming obvious that I don’t scale — that the long-term solution is NOT for me to increase my skills, as I’ve been doing, but to hire people who have the skills.

That said, in the short term, it’s clear that there’s three technical areas where I must improve my skills:

1. I need to be a world-class guru in apt-based package management.
2. I need to greatly improve my skills at low-level TCP/IP issues — packet captures, diagnosis of networking problems, etc.
3. I need to greatly improve my skills at dealing the Linux kernel, on all levels — compiling, kernel options, loadable modules & drivers, etc.

I guess it’s good to have a clear vision of where I need to go. Now I need to get there.

We’re now going to be able to focus on growth and moving forward; this will remove a huge number of the roadblocks that have been holding us back.  It’s big news.

(Unfortunately for my friends, it probably doesn’t mean I’ll be spending less time at work — we’re leaping into action like a sprinter leaving the blocks.)