Reassociating old Time Machine backups

In an attempt to get myself cheap remote backups over the internet, I bought a Raspberry Pi kit and set it up as a hackintosh Time Capsule by attaching my USB backup disk to the Pi. I however wanted to keep my existing backup history, so instead of using a fresh Linux-formatted partition (like a clever boy) I tried to get the Pi to use my existing HFS+ filesystem. Anyone interested in trying this should probably read about Linux’s flaky HFS+ user mapping and lack of journaling support first, and then back away very slowly. I blame this for all my subsequent problems.

After some effort I did get my aging Macbook to write a new backup on the Pi, but I couldn’t get it to see the existing backups on the drive. Apple uses hard links for deduplication of backups, and because remote filesystems can’t be guaranteed to support them it uses a trick. Remote backups are written not directly on the remote drive, but into a sparse disk image inside it. Thinking that it would be a relatively simple matter to move the old backups from the outer filesystem into the sparsebundle, I remounted the USB drive on the Mac (as Linux doesn’t understand sparsebundles, fair enough).

The Macbook first denied me the move, saying that the case sensitivity of the target filesystem was not correct for a backup – strange, because it had just created the sparsebundle itself moments before. Remembering the journaling hackĀ  I performed “repair disk” on both the sparsebundle and then the physical disk itself. At this point disk utility complained that the filesystem was unrecoverable (“invalid key length”) and the physical disk would no longer mount. In an attempt to get better debug information from the repair, I ran fsck_hfs -drfy on the filesystem in a terminal. This didn’t help much with the source of the error, but I did notice that at the end it said “filesystem modified =1”. Running it again produced slightly different output, but again “filesystem modified =1”. It was doing something, so I kept going.

In the meantime, I had been looking into ways of improving the backup transfer speed over the internet. I originally planned to use a tunnel over openvpn, but this would involve channeling all backup traffic through my rented virtual server, which might not be so good for my bank account. I did some research into NAT traversal, and although the technology exists to allow direct connections between two NATed clients (libnice), I would have to write my own application around it and at this point I was getting nervous about having no backups for an extended period. I had also been working from home and getting frustrated with the bulk transfer speed between home and work, and came to the conclusion that my domestic internet connection couldn’t satisfy Time Machine’s aggressive and inflexible hourly backup schedule.

Six iterations of fsck_hfs -drfy later, the disk repair finally succeeded and the backup disk mounted cleanly. At this point, I decided a strategic retreat was in order. I went to set up Time Machine on the old disk, but it insisted that there were no existing backups, saying “last backup: none”. Alt-clicking on the TM icon in the tray and choosing “Browse Other Backup Disks” showed however that the backups were intact. While I could make new backups and browse old ones, they would not deduplicate. As I have a large number of RAW photographs to back up, this was far from ideal. There is a way to get a Mac to recognise another computer’s backups as its own (after upgrading your hardware, for example) . However, it threw “unexpectedly found no machine directories” when attempting the first step. It appeared that not only did it not recognise its own backup, it didn’t recognise it as a backup at all.

After a lot of googling at 2am, it emerged that local Time Machine backups use extended attributes on the backup folders to store information relating to (amongst other things) the identity of the computer that had made the backup. In my earlier orgy of fscking, the extended attributes on my Mac’s top backup folder had been erased. Luckily, I still had the abandoned sparsebundle backup in the trash. Inside a sparsebundle backup, the equivalent metadata is stored not as extended attributes, but in a plist file. In my case, this was in /Volumes/Backups3TB/.Trashes/501/galactica.sparsebundle/com.apple.TimeMachine.MachineID.plist, and contained amongst other bits and bobs the following nuggets:

<key>com.apple.backupd.HostUUID</key>
<string>XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX</string>
<key>com.apple.backupd.ModelID</key>
<string>MacBookPro5,1</string>

These key names were a similar format to the extended attributes on the daily subdirectories in the backup, so I applied them directly to the containing folder:

$ sudo xattr -w com.apple.backupd.HostUUID XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX /Volumes/Backups3TB/Backups.backupdb/galactica
$ sudo xattr -w com.apple.backupd.ModelID MacBookPro5,1 /Volumes/Backups3TB/Backups.backupdb/galactica

After that was fixed, I could inherit the old backups and reassociate each of the backed up volumes to their master copies:

$ sudo tmutil inheritbackup /Volumes/Backups3TB/Backups.backupdb/galactica/
$ sudo tmutil associatedisk -a / /Volumes/Backups3TB/Backups.backupdb/galactica/Latest/Macintosh\ HD/
$ sudo tmutil associatedisk -a /Volumes/WD\ 1 /Volumes/Backups3TB/Backups.backupdb/galactica/Latest/WD\ 1/

The only problem arose when I tried to reassociate the volume containing my photographs. Turns out they had never been backed up at all. They bloody well are now.


 

So what happened to my plan to run offsite backups? I bought a second Time Machine drive and will keep one plugged in at home and one asleep in my drawer in work, swapping them once a week. This is known as the bandwidth of FedEx.

Text messages (iMessages) coming from email addresses when using iOS 7

If you have recently upgraded your iPhone (and you should, see https://gotofail.com), your iMessage settings may be messed up – this isn’t visible to you, but it is to your contacts. If someone is sending you text messages that appear to come from an email address rather than their phone number, then they have this problem – please tell them and send them here.

To check if you have the problem yourself:

  • Go to settings > Messages > Send and Receive
  • Under “You can be reached by iMessage at”, if your phone number is not ticked, you have this problem.

To fix (this worked for me):

  • Go to Settings > Messages
  • Turn off iMessage
  • Go to Settings > Cellular
  • Turn off Cellular Data
  • Reboot phone
  • Turn Cellular Data and iMessage back on
  • Go to Settings > Messages > Send and Receive
  • Under “Start new conversations from”, tick your phone number

If you want to fail back to SMS messaging when iMessage is unavailable, make sure to enable Settings > Messages > Send as SMS. This was disabled by default on my fresh iOS7, and can cause text messages to be mysteriously delayed.

For more detail on the problem, see http://simonhackett.com/2013/11/06/ios7-imessage-mobile-number-linking/

Jack vs USB headphones on Mac

A work colleague recently resigned and left his USB headset on his desk. Foolishly thinking that one plug is better than two, I swapped my old jack-connector skype headset for his. Unlike the headphone jack, USB-connected headphones do not automatically disable the internal speakers on my macbook when plugged in, leading to a blast of noise across the cube farm when I tried to test them.

The problem is twofold – firstly, Mac OS treats the headphone jack and internal speakers as the same device, whereas USB headphones are naturally treated as an extra device. There is therefore no configuration option in the software to alter the behaviour – it’s buried in the BIOS for all I can tell. I got around this by downloading SoundSource – it remembers speaker and microphone configurations and restores your earlier sound settings as you plug devices in and out.

But then I came across the second problem – it only worked 50% of the time. Turns out that a single physical device appears as different virtual devices depending on what USB port it’s plugged into. I had to configure the headset to be the default mic and speakers while it was plugged into one port, then plug it into the other and repeat the process.

It’s at times like this that I appreciate how my mother feels.

DNS cache locking on Server 2008

So I’ve been informed that there are some bizarre problems resolving a website that has recently changed providers from digiweb to novara (wasn’t my idea). From elsewhere the new site appears reliably, but from inside our network we are getting the following results:

andgal@nbgal185:~$ host -t any peracton.com
peracton.com has SOA record ns1.novara.ie. hostmaster.host.ie. 2011080416 10800 3600 604800 14400
peracton.com name server ns1.novara.ie.
peracton.com name server ns2.novara.ie.
peracton.com name server ns3.nameserver.ie.
peracton.com mail is handled by 10 smail3.host.ie.
peracton.com has address 80.93.17.28

andgal@nbgal185:~$ host peracton.com
peracton.com has address 78.137.166.130
peracton.com mail is handled by 10 mail2.hosting.digiweb.ie.

The first set of results is the “correct” one, so why is host (and nslookup, and dig, and firefox…) still going to the old address by default? I suspect it is something to do with cache locking on our Server 2008 DNS forwarder. It seems that even after I have forced a fresh request by using “-t any”, the stale cached A record is being returned for normal searches. This is apparently a security measure to protect against cache poisoning. It would appear that the TTL on the old A record was unusually long, which means that I had to flush the cache on the primary DNS forwarder (the backup DNS forwarder is fine, presumably because the old record was never in its cache).

Sure enough, running “dnscmd /clearcache” on the offending server fixed the problem.

 

 

 

 

 

 

 

 

Ubuntu upgrade hell

So I decided to upgrade my work ubuntu laptop. I had been putting it off for ages because of the hell I went through upgrading it from 6.06 to 8.10, but being stuck on old versions of pretty much everything (but especially openoffice) was becoming impossible. Strangely though, it was when I tried to install monkeysphere that I finally snapped. Time to do a dist-upgrade to 10.04 I thought.

I started using the desktop package manager – it had been prompting me with “a new version of Ubuntu is available” for quite some time, so I pushed the button and let it do its thing. After an initial false start (out of disk space) it downloaded the upgrades and went to work. About half an hour in, it decided to restart kdm, which is where the fun started.

Now I’m at a command login and the upgrade is in a bad state. A few apt-get -f installs later I find that there’s a file missing in the latex hyphenation config and one of the postinst scripts keeps failing. The postrm script fails too, so I can’t even remove the offending package. After sleeping on the problem, grep -r comes to my rescue and I find the reference to the missing file. Commenting that out just bumped me to the next missing file, so I rmed the lot. I know I now have a badly broken latex system but dammit, I just want this upgrade to finish.

A few more apt-get -f installs and apt-get dist-upgrades later, and the system is ready to reboot. But grub can’t find the root partition, and it drops me to an initramfs which can’t see any hardware. So I had to fire up a 10.04 install cd that I fortunately had to hand, and use it to chroot into the root partition and rebuild the initrd.

Finally, it boots. But I can’t login graphically because gnome-session can’t be found. Back into the command line and apt-get install ubuntu-desktop, which takes another half an hour because it’s all missing, the lot of it. At this point, I notice something odd – I can use my thinkpad fingerprint reader to log in on the command line, but not graphically – scanning the finger when in X gives nothing, not even an error.

Anyway, my xorg.conf file is apparently no longer valid, as it doesn’t recognise the dual screen setup. I rename it and let it run on auto-detect and the screen comes back, but the EmulateWheel on my trackball now doesn’t work. So I run X :2 -configure to get a skeleton xorg.conf file, save that in /etc/X11/xorg.conf and cut&paste my old mouse settings into it. This doesn’t work.

At this point, having used sudo several times, I accidentally discover how to make the fingerprint sensor work while inside X. When prompted, scan your finger. Wait two seconds, hit Ctrl-C and then enter. Don’t ask me why.

It turns out that in recent versions of xorg, you need to set the option “AllowEmptyInput off” in the ServerFlags section or else it ignores any mouse or keyboard configuration sections. Sure enough, this allows EmulateWheel to work again, but the mouse pointer also moves while the emulated scroll wheel is turning. I’m now running very late and this sorry saga will have to continue in the new year.

Merry Christmas, software developers.

How to create a DNAME record in Server 2008

I had to do a little searching on the internet to work out how to do this, so here it is in a single post.

DNAME records are supported in Server 2008’s DNS service, but you cannot add them (or edit them) in the graphical tool. You need to use the command line. Right-click “Command Prompt” and run as administrator. Then you can type in the following:

DNSCMD /RECORDADD myzone.example.com mynewrecord DNAME myoldrecord.example.com

If you now refresh the graphical DNS tool you will see a new record with a blank type and contents “Unknown – view properties for more info”. If you do this, you will see the raw hex data for the DNAME RR (type 0x27). The only thing you can do with it in the graphical tool is delete it.

More JIRA shenanigans

As part of my ongoing project to put every management process possible onto JIRA, I have been working on a way to make our internal account codes smarter. Previously, I was using a custom field with predefined values for each account code – this worked well until I wanted to automate certain tests and actions based on account code metadata such as project start/end dates and approval matrices (this issue was also a significant problem). Two methods sprung to mind: either I could define a custom field that could contain structured data, or I could implement the account codes as Jira issues and use issue linking. In my solution I ended up doing a bit of both.

The advantage of using JIRA issues to store account codes is twofold: firstly I can define whatever custom fields I require to hold structured data (e.g. budget holder userids, timestamps) in a format that JIRA can easily understand; and secondly the financial administrators and I can manage this data through a familiar interface. The big disadvantage is that issue linking in JIRA is done through a separate interface from issue creation/editing, meaning a two-stage process for my users. The ideal solution would be a drop-down menu in the issue creation/editing screen, but this means using a custom field – and one I would probably have to write myself.

Being a typical sysadmin, I decided to hack something together with sticky tape and string. I found a likely-looking custom field in the form of the JIRA Database Values Plugin. Using this to tap directly into JIRA’s back-end database I can populate a drop-down menu using some horrendous left joins:

sql.query=select i.id, cv1.stringvalue, cv2.stringvalue, cv3.stringvalue from (select id from jiraissue where project=10200 and issuetype=13) as i left join (select issue, stringvalue from customfieldvalue where customfield=10140) as cv1 on i.id=cv1.issue left join (select issue, stringvalue from customfieldvalue where customfield=10141) as cv2 on i.id=cv2.issue left join (select issue, stringvalue from customfieldvalue where customfield=10142) as cv3 on i.id=cv3.issue order by cv2.stringvalue, cv3.stringvalue;

primarykey.column.number=0
rendering.viewpattern={1,(none)} {2} {3}

The joins are necessary because JIRA uses a relational back end which assumes a fixed number of columns in each table. It handles issues with custom-defined fields by storing the custom field values in a separate table with back-references to the appropriate row in the main table.

The JDVP plugin provides a pretty front end to a custom field that stores the primary key of a database row, which in this case is jiraissue.id. I can then recall the account code issue in a JSS script by passing the primary key to getIssueObject() – the example below is a post-function that I use in an approval workflow to ensure that nobody can approve his own request:

from com.atlassian.jira import ComponentManager

# Only perform functions on Travel Request issues
if(issue.getIssueTypeObject().getId()=="8") :
  cm = ComponentManager.getInstance()
  cfm = cm.getCustomFieldManager()
  im = cm.getIssueManager()

  accountCodeID = issue.getCustomFieldValue(cfm.getCustomFieldObject('customfield_10134'))
  acIssue = im.getIssueObject(int(accountCodeID))
  acBudgetHolder = acIssue.getCustomFieldValue(cfm.getCustomFieldObject('customfield_10070'))
  acEmergencyBudgetHolder = acIssue.getCustomFieldValue(cfm.getCustomFieldObject('customfield_10151'))

  # Set the budget holder to the account code's budget holder, unless that is the reporter.
  # The Java API for custom fields is crap. I'm using a trick here.
  # http://confluence.atlassian.com/pages/viewpage.action?pageId=160835&focusedCommentId=224398519&#comment-224398519
  # DO NOT USE issue.setCustomFieldValue() - it does NOT persist. This is a feature (!)

  cf=cfm.getCustomFieldObject('customfield_10070')
  if issue.getReporter() == acBudgetHolder :
    cf.getCustomFieldType().updateValue(cf,issue,acEmergencyBudgetHolder)
  else :
    cf.getCustomFieldType().updateValue(cf,issue,acBudgetHolder)

I have two things to say about the above. Firstly, scripts in a language where leading whitespace is significant are very difficult to post to blogs. Secondly, I meant everything I said about the custom fields API. In my previous post, I complained about the Java standard library requiring double references – well here’s another example. Read the last ten lines and try to tell me that the developer in question was sober when he designed the interface.

This does exactly what I want (like all good sticky-tape solutions should). The only thing the JDVP did that annoyed me was to always render “Unknown” (or the localisation equivalent) in the case that one of my database fields contained NULL (some of our accounting codes have optional subcodes, for example). After discussions with the developer, I submitted a patch which made the NULL behaviour configurable, and he has included it in the latest release. Thanks, Wim.