File “Created Date” Under OS X — Harder Than You Think

A client had an OS X server with tens of thousands of files in a directory tree, and wanted to move some of them based on their creation date. I put together a Python script that worked perfectly on my test system, but failed in production. (I put in a “–dryrun” option, so no harm was done.)

The modification date the script reported was correct, but the creation date the script reported was different from the creation date that doing “Get Info” in the OS X Finder reported. Hmm. Not good. I did some investigating.

Let’s look at a test file. The Finder reports the following creation and modification dates:
Created: Monday, April 10, 2006 17:04
Modified: Thursday, April 13, 2006 15:54

Python reports a different creation date and the same modification date:

>>> import os.path
>>> import time
>>> stamp = os.path.getctime('datetest.txt')
>>> time.ctime(stamp)
'Fri Apr 10 15:51:10 2009'
>>> stamp = os.path.getmtime('datetest.txt')
>>> time.ctime(stamp)
'Thu Apr 13 15:54:43 2006'

Hmmm. What does “ls” in the Terminal report? The standard “ls -l” reports creation date. Adding “-T” makes it report the full date in all cases. And “-c” counter-intuitively means “display modification date.”

$ ls -lT datetest.txt
-rwxr-xr-x@ 1 schof  schof  2448091 Apr 13 15:54:43 2006 datetest.txt
$ ls -lcT datetest.txt
-rwxr-xr-x@ 1 schof  schof  2448091 Apr 10 15:51:10 2009 datetest.txt

The plot thickens. The modification date matches the Finder and Python, but both Python and ls are reporting an incorrect (according to the Finder) creation date.

Quite a bit of Googling showed me the “mdls” tool — or “metadata ls.” Very useful. This shows the complete set of metadata for a file. Including the OS X creation date. Actually, two creation dates, one for the file, and one for the file content. (I’m not sure what circumstances would make those creation dates differ. The documentation I’ve been able to find has been unclear and contradictory.)


$ mdls datetest.txt
kMDItemContentCreationDate     = 2006-04-10 17:04:34 -0700
kMDItemContentModificationDate = 2006-04-13 15:54:43 -0700
...
kMDItemFSContentChangeDate     = 2006-04-13 15:54:43 -0700
kMDItemFSCreationDate          = 2006-04-10 17:04:34 -0700
...
kMDItemLastUsedDate            = 2009-04-11 11:48:03 -0700
kMDItemUsedDates               = (
2009-04-11 00:00:00 -0700
)

Now that we’ve got all that information, what does it tell us? The Unix/Linux standard for “creation date” is to show you the date on which a particular file was created. If you copy file “a” to file “b,” those are two different files, and the “creation date” for file “b” will be the date you made the copy.

OS X metadata travels with the file, so if you copy file “a” to file “b” using ditto on the command-line or using the Finder, the Unix creation date will be the date the copy was done, but the OS X creation date of file “b” will be the same as file “a.”

There’s good arguments for handling “creation date” the Unix way, and there are good arguments for doing “creation date” the OS X way, but mixing them as OS X does is kind of frustrating.

I’ve written a quick-and-dirty Python example script that reports the Unix creation date and the OS X creation date for any particular file. Since it’s released under the open-source MIT license, feel free to use it in your own programs. You can download it here: http://www.sudosu.net/getcreationdate.safe.

OS X Docs are Wrong; How To Run Periodic Maintenance

I just reimaged my laptop, and discovered that the “apropos” command had nothing in its database. (You use “apropos” to look up man pages related to a topic. It looks through an index to find appropriate pages to list, and the periodic maintenance tasks had not created the index.) OK. No problem. Google is my friend.

Google took me to an Apple Knowledge Base article that is out of date: http://support.apple.com/kb/HT2319

This article states that you should, from the terminal, run “/etc/daily” to run the daily maintenance tasks. That may have been true before LaunchD took over from cron, but it’s not true anymore. There is no /etc/daily file on my 10.5 system.

The other option they suggest, “sudo periodic weekly” did work, and generated the apropos index files.

Another nice little one-liner; wait until a file is there (or not there)

This is the sort of thing that’s second nature to a command-line geek, but I thought it was cool enough (in its little way) to share.

The problem: I have a process that runs on a remote server for quite a while. I wish to be alerted when it completes. As long as it’s running, it creates a lockfile in /var/lock. So when the lockfile goes away, I want to be alerted.

The solution:

Part 1: In my .bashrc file, I’ve aliased the command “alert” to play a sound:

alias alert=’aplay /usr/share/sounds/k3b_success1.wav’

But of course, I can only play that on my local machine — does me no good to sound an alarm on a computer in the server room, even if it had speakers.

Part 2: I can create a “while” one-liner that waits until the file is deleted, then executes the next line:

while [ -e ~/lockfile ]; do sleep 10; done; alert

However, this can only test against files on my local system. This brings me to…

Part 3: You can use ssh to run a command on a remote system, instead of just logging into the remote system:

ssh user@example.com “while [ -e /var/lock/program.lock ]; do sleep 10; done” ; alert

It runs the while loop on the remote system, not giving back control to the local system until the while loop exits (which happens when the lock file no longer exists). Once the while loop exits, ssh exits, and we run the “alert” command, which plays a sound alerting me to the change.

Took me about 10 times longer to write up than it did to implement. But it’s the sort of thing you can just throw together quickly on a *nix system to make your life easier.

“Behind Closed Doors” by Rothman & Derby

“Behind Closed Doors: Secrets of Great Management” by Johanna Rothman and Esther Derby is a strange book; while not having any ideas that are really NEW to anyone who’s read the other management books I’ve recommended, BCD still manages to impress and be worth reading. It does this by having lots of small, practical advice for managers, in lots and lots of different areas.

If you read the other books in the field, will you derive most of these ideas yourself? Probably. But I still got a few good techniques from this book — among them, a new approach to what BCD calls my “Project Portfolio” — my lists of projects, tasks, and who they’re assigned to.

Worth reading. Not worth reading first, but worth the time and money spent on it.

Two Not-Highly-Recommended Books

This should, of course, be subtitled “Books Not To Read If You Are Me.” These aren’t necessarily bad books (in fact, both of them are good books) but they weren’t that useful to me or to people in exactly my situation. Take this with as much salt as you feel appropriate, depending on what you’re working on.

The first is “Getting Real” by 37signals. If you’re doing web-application development, I’d say this is a book you should read. I’m not doing that, and thus, didn’t see a whole lot of value here. I’m making a mental note to reread this if I ever do get into a web development project, and it did make me think about ways I could make our current development procedures more agile — but if the question is “Was this worth the time it took for me to read it?” then the answer is “No.”

The second is “The Mythical Man Month” by Frederick Brooks. It came highly-recommended in every software management book I’ve read, so I figured it had to be on my list. While there is much still of value in this book, easily a third to a half seems completely out-of-date — advice on whether to use printed manuals or microfiche that seems as applicable as a discussion of which tools are best for shoeing horses.

While the concepts discussed in TMMM were clearly revolutionary, I’m not sure it makes the first cut for a software management reading list. Most of the ideas it contains are captured better in other, more current books, particularly “Peoplware,” “Joel On Software,” and “Managing Humans.”

Two Good Software Books

I just finished reading two different books on software development and management; both are highly recommended.

First, and best is Peopleware, by Tom DeMarco and Timothy Lister. This is another one of those must-read books. I’ve been telling everyone that I can’t believe I’ve been allowed to have the job I have without reading it. If you are in the business of software (or in the business of managing any type of “knowledge worker,”) you must read this. Really. I’m buying copies for all the managers and execs at work.

Second is the second “Joel On Software” book, “More Joel On Software,” by Joel Spolsky. I wasn’t as wowed by this one as I was by the first, but it’s still very much a worthy read. Most of the articles in the first one were new to me, while I had already read most of the 2nd one’s content on Joel’s website. Still definitely worth reading.

“Joel On Software” By Joel Spolsky

If you’re in the business of developing software, you should read the “Joel On Software” book. Period.

(Shortest review I’ve ever written.)

While I don’t agree with everything Joel has written, I find most of what he writes to be a clear exposition of common-sense approaches to managing software development. No BS, just simple stuff that works. A small percentage of “Joel On Software” is mind-changing, opening your eyes to new ways of doing things or new ways of thinking about old software.

Highly recommended.

Backing up with Duplicity over SSH

I’m attempting to set up Duplicity as a backup tool, running from cron over ssh, to backup my home directory to another server.

I of course already have passwordless SSH set up to connect to that server, and use ssh-agent to store the passphrase for my SSH key, as described in another entry. However, I could not get passwordless SSH to work from a cron job. If anyone has tips on how to do that, I’d love to hear it.

So I created a second SSH public/private key pair with no passphrase, (in ~/.ssh/backup and ~/.ssh/backup.pub) and figured out how to have Duplicity call ssh, scp, and sftp with the correct parameters to specify the new key pair. (There’s about a zillion different ways of specifying that, and only ONE that works across all three programs.)

The passphrase mentioned here is the one used to encrypt the duplicity backups.

PASSPHRASE=’YourPassphraseGoesHere’ duplicity \
–no-print-statistics \
–ssh-options “-oIdentityFile=/home/schof/.ssh/backup” \
/home/schof \
scp://johnmarkschofield@example.com/duplicity

How To Hire The Right People

I must say, this article by Steve Yegge has done quite a bit to refine my thoughts about hiring geeks. (The beginning is a little long-winded. Push through. It’s worth it.)

On the other hand, I’ve met exactly ONE of the type he’s talking about in my career. Not sure where that leaves me as a managing geek who needs to hire brilliant geeks.

Installing VMware Server 1.0.6 on Ubuntu Hardy

It generally works pretty well, but I found the following problem, and Google was no help:

Building the VMware VmPerl Scripting API.

Using compiler “/usr/bin/gcc”. Use environment variable CC to override.

Unable to compile the VMware VmPerl Scripting API.

********
The VMware VmPerl Scripting API was not installed.  Errors encountered during
compilation and installation of the module can be found here:
/tmp/vmware-config4

You will not be able to use the “vmware-cmd” program.

Errors can be found in the log file:
‘/tmp/vmware-config4/control-only/make.log’

When you look at make.log, you see a series of errors like this:

In file included from VmPerl.xs:6:
/usr/lib/perl/5.8/CORE/perl.h:420:24: error: sys/types.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:451:19: error: ctype.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:463:23: error: locale.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:480:20: error: setjmp.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:486:26: error: sys/param.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:491:23: error: stdlib.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:496:23: error: unistd.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:776:23: error: string.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:925:27: error: netinet/in.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:929:26: error: arpa/inet.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:939:25: error: sys/stat.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:961:21: error: time.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:968:25: error: sys/time.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:975:27: error: sys/times.h: No such file or directory
/usr/lib/perl/5.8/CORE/perl.h:982:19: error: errno.h: No such file or directory

The answer is simple:

sudo apt-get install libc6-dev

Now Google has the answer.