Do what I mean, dammit. Or, why being silently “helpful” is evil. January 8, 2009
Posted by andrewgdotcom in Mac, Ranting.3 comments
For the last three years (i.e. before Time Machine), I have been using rsync to make incremental backups to an external FireWire disk from my trusty iBook. Now, rsync does this by backing up into a fresh location each time and referring to the previous backup to check if any data can be de-duplicated, which it does by creating hard links.
rsync -a --link-dest=$PREVIOUS_BACKUP $SOURCE $NEW_BACKUP
A wrapper script is needed because rsync doesn’t do any rotation of the backup paths (mine preserves six previous backups, which is quite enough), but overall the solution is elegant – restoration from any given backup is just a cp -pR away, due to the transparent nature of hard links. However (and this is the important bit) rsync only de-duplicates when the source file is identical to its previous backup in every way, including metadata.
As time went on and my laptop drive started getting full, I found that the backup window was getting suspiciously long for an 80GB disk. But it was the lack of space that finally drove me to buy a bigger external drive (admittedly, backups aren’t the only thing taking up GB on my FireWire farm).
Finally I ripped apart the wrapper script and dug through the previous backups. Turns out that rsync wasn’t preserving all the metadata, specifically file ownership. Google, as ever, gave me the answer:
Official Google Mac Blog: User 99, Unknown
By default, OS X silently maps all file ownership on external HFS+ disks to a special user “unknown”, while pretending to the user that he still owns the files. This is a “feature” to prevent weird permissions problems when swapping external drives between machines. It is so low-level that not even root can override it – if you try as root to chown a file on an external HFS+ filesystem, it will silently do nothing.
The upshot is that rsync didn’t believe that any backed-up files were identical to their originals, and so didn’t de-duplicate anything. Instead of six lean incrementals, I have six wasteful bulk backups, some of them incomplete because I ran out of backup window. And all the file ownerships have been lost, so a system restore would have failed spectacularly – if I had been foolish enough to try.
You can turn off this behaviour for any given disk (Finder>Drive>right-click>Get Info>uncheck “Ignore ownership…”), but it’s a bit late now.
The history meme April 16, 2008
Posted by andrewgdotcom in Linux, Mac, Software.add a comment
Spreading the meme…
serenity:~ andrewg$ history | awk ‘{a[$2]++} END {for(i in a)print a[i] ” ” i}’ | sort -rn | head -10
108 telnet
63 rscreen
43 ping
24 sudo
24 ssh
20 host
16 scp
16 more
14 ls
13 xdvi
Hm. I seem to be using the command line mainly as a gateway into remote systems – which reflects my average working day. The stray ‘xdvi’ is due to my recent heavy use of TextMate to write a paper in LaTeX. Not sure why I’ve been using sudo so much on my Mac though.
Similarly, on my work Linux laptop:
andgal@nbgal185:~$ history | awk ‘{a[$2]++} END {for(i in a)print a[i] ” ” i}’ | sort -rn | head -10
68 rscreen
52 ping
45 host
36 sudo
35 rdesktop
32 xrandr
21 startmenu
20 ssh
18 ifconfig
15 telnet
rscreen is merely a wrapper for ssh:
function rscreen() { /usr/bin/ssh -t $1 ’screen -dr || /usr/bin/screen || /bin/bash’; }
and startmenu is a cool but dodgy hack to get into my windows virtual machine:
alias startmenu=’nohup rdesktop -A -s “c:\program files\seamlessrdp\seamlessrdpshell.exe explorer.exe” 192.168.185.128 -u andgal -p xxxxxxxx&
xrandr reflects the fact that I have to configure my dual-screen setup by hand after each boot under Ubuntu 7.10, as the GUI configurator just Doesn’t Work. I had to test this many many times. Apparently the latest Ubuntu beta fixes most of these problems.
