Recently I had to track down a minor bug in the OpenBSD kernel. I tapped QEMU and GDB as debugging tools for the task, running on Ubuntu 12.04 as the host OS. This combination worked extremely well, so for the record here’s how I set it all up.

OpenBSD comes equipped with two kernel debugging mechanisms: ddb and kgdb. ddb(4) is an in-kernel debugger, enabled by default in the GENERIC kernel, and can be invoked either explicitly from the console or automatically in the event of a panic. It is analogous to the Linux debugger kdb in that it can be used to set breakpoints and examine the stack or register state, but (like kdb) it is not a source-level debugger.

For source debugging there is kgdb(7), which offers the ability to remotely debug the kernel by way of a GDB stub running over a serial port; this is similar to the Linux debugger kgdboc. However, kgdb it is not available in the GENERIC kernel, and it imposes an additional set of configurations and debugger latencies on the user. If your debugging task is amenable to running OpenBSD within a virtual machine, as mine was, then there is an easier and better way…read more

I’ve been using OpenBSD as a NAT router for a few years now. Here’s a guide showing how I got it up and running, in case anyone is interested in doing the same.

I’ve been banging my head against the wall on account of different music players which can’t play the subset of songs I have encoded as either Vorbis or AAC. So I’m slowly converting my entire music library over to MP3, which works everywhere, even if it’s less efficient.

But before I go digging through my old CDs I need to identify which albums I have to re-encode. My music is organized in folders by artist and then album, e.g.:

~\Music\Library\Pixies\Surfer Rosa\Where Is My Mind.mp3

So this means I effectively have to list the names of folders containing non-MP3 music files. Fortunately Windows PowerShell makes this a one-liner (though admittedly it’s a pretty long line; the backtick is PowerShell’s line continuation syntax):

ls -r Music\Library `
| ?{ $_.PSIsContainer -And ( $_.GetFiles() `
| ?{ $_.Name -Match "\.(m4[ap]|ogg|wma)$" } ) } `
| %{ New-Object PSObject -Property `
@{ Artist = (gi $_.PSParentPath).Name; Album = $_.Name } }

This will give you a nice list of non-MP3 albums, like:

Album              Artist
-----              ------
That's Your Fire   Aloha
Noble Beast        Andrew Bird
Believe It Mammals Bats & Mice
Charm School       Bishop Allen
[...]

And thanks to PowerShell’s object-oriented pipes, this is done without having to worry about any quoting or whitespace issues.

The more I learn PowerShell, the less I find myself relying on Cygwin for simple Windows administration tasks. I think Microsoft did a nice job balancing all the different requirements they had to meet with PowerShell. At the very least, it’s good to finally see a strong command shell built into Windows.

This is a quick guide to booting the OpenBSD installer on a PC Engines ALIX board with tinyBIOS (such as the ALIX 2d3) via PXE, using just the following:

• PC with two network interfaces. One of these needs to be Ethernet, and the other must connect to the Internet. For example, any standard PC laptop with both WiFi and Ethernet adapters will work if there’s a WiFi Internet connection available.
• Null modem cable
• Ethernet crossover cable
• USB-serial adapter (unless your PC has a built-in RS-232 port)
• Ubuntu Linux 10.10 desktop live CD

Thanks to the versatility of the Ubuntu live CD (specifically the use of AUFS to provide a writable root directory in RAM), you can set up the necessary PXE boot server without making any permanent changes to your PC.

$ sudo -s
# apt-get update
# apt-get install dhcp3-server tftpd xinetd cu

# ifconfig eth0 up 192.168.2.1 netmask 255.255.255.0

# echo 1 > /proc/sys/net/ipv4/ip_forward
# iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
# iptables -A FORWARD -i eth0 -j ACCEPT
# iptables -A FORWARD -i wlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT

authoritative;

shared-network LOCAL-NET {
    option domain-name-servers 8.8.8.8;

    subnet 192.168.2.0 netmask 255.255.255.0 {
        option routers 192.168.2.1;
    filename "pxeboot";
    range 192.168.2.100 192.168.2.200;
    default-lease-time 600;
    max-lease-time 7200;
    }
}

INTERFACES="eth0"

# /etc/init.d/dhcp3-server start

service tftp
{
    socket_type = dgram
    protocol = udp
    wait = yes
    user = root
    server = /usr/sbin/in.tftpd
    server_args = -s /tftpboot
}

# /etc/init.d/xinetd restart

# cu -e -o -s 38400 -l /dev/ttyUSB0

boot> stty com0 38400
boot> set tty com0
boot> bsd.rd

I just found a great web site providing a better Kindle content search than what’s baked into Amazon: eReaderIQ.com. You can search by price, publication date, reading level, and whether the book you want is in the public domain, among other things.

For example, this query lists only free Kindle books in the public domain. Very handy.

I had a funny experience yesterday.

I don’t typically use Microsoft Outlook with my home email account. But it was bundled with the copy of Office 2010 that I installed a few months ago, and I figured: heck, if I have it anyway I might as well hook it up to my personal IMAP account. So I configured Outlook, played with it a bit, then promptly forgot all about it.

Fast forward to yesterday evening. I launch my copy of Outlook for the first time in months in order to try something with the calendar, but then I get distracted and walk away from the computer. Five minutes later my cell phone gets a message from my FreeBSD server:

EMAIL SERVER HIJACKED - FIREWALLING POSTFIX

Oh. Fun.

A slight digression before I can get on with the story: I try my very best to be a good Internet citizen, and that of course means not allowing oneself to become a spam relay. Because this email server exists only for my personal use, it was simple to write a Perl script that monitors my Postfix logs and, if it sees anything grossly out of the ordinary sent out through the server, reconfigures PF to block outbound connections to TCP port 25. It seemed the responsible thing to do, especially since some PHP applications on my web server have permission to relay through this Postfix instance; and let’s face it, WordPress has a less than stellar security track record.

This script has never given me a false positive. So this was definitely cause for concern.

I shelled into my VPS and went straight for the mail logs. I was dismayed to find more than twenty entries like this:

Dec  8 23:35:38 frodo postfix/smtpd[13146]: 13DB43F4DC: \
    client=redacted[xxx.xxx.xxx.xxx], sasl_method=LOGIN, sasl_username=redacted
Dec  8 23:35:39 frodo postfix/cleanup[13148]: 13DB43F4DC: \
    replace: header Received: from noatun (redacted [xxx.xxx.xxx.xxx]) \
(Authenticated sender: redacted)
by frodo.paleogene.net (Postfix) with ESMTPSA id 13DB43F4DC
for <redacted@example from redacted[xxx.xxx.xxx.xxx]; \
    from=<redacted@example.com> to=<redacted@example.com> \
    proto=ESMTP helo=<noatun>: Received: from auth-client.paleogene.net \
    (auth-client.paleogene.net [206.125.175.178])
(Authenticated sender: hidden)
by frodo.paleogene.net (Postfix) with ESMTPSA id 13DB43F4DC
for <redacted@example.com>; Wed,  8 Dec 2010 23:35:37 -0500 (EST)

All of them sent within seconds of one another, and all of them to random Gmail accounts and other recipients that I absolutely did not recognize. Worse, the messages had been sent through a properly SASL authenticated connection from my laptop (HELO noatun). To quote that old horror flick: They’re coming from inside the house!

I’ve heard tell of Windows viruses that will quietly send spam through the user’s configured Outlook mail account, so that the messages originate from a legitimate mail server and have a better chance of skipping recipients’ spam folders. I have no idea whether this is actually a common occurrence in the real world, but it’s plausible enough, and I couldn’t think of any better explanation why 24 messages had been sent out from my computer in a matter of seconds, all to recipients I’d never heard of. I still could not fathom how a spam bot might have made its way onto my laptop despite my precautions, but better safe than sorry…

Process Explorer and TCPView failed to reveal anything suspicious, so I shut down the laptop, yanked out its hard drive, hooked that up to another machine via a USB-SATA adapter, and started a full offline virus scan.

Meanwhile I kept poking around on the mail server. I had a bit of luck when I checked /var/spool/postfix/deferred — a few messages had been caught in the Postfix outbound queue when my script added its firewall rule. I was morbidly curious what had been sent.

…it was a freaking Message Disposition Notification.

Now sure, I knew all about email read receipts, but I’d immediately ruled them out as the cause of this behavior because all of my email clients, Outlook included, are configured to at least ask my permission before sending one:

[caption id=”attachment_668” align=”aligncenter” width=”570” caption=”Outlook completely ignores this”][/caption]

And anyway, I hadn’t read or deleted any email messages in Outlook, so why would it want to send any MDNs in the first place?

I did some googling and the pieces slowly began falling into place. It turns out there’s a horrific bug in recent Outlook versions’ handling of read receipts: unread messages deleted from an IMAP folder can send a “not read” MDN, even if you’ve explicitly configured Outlook not to do so.

And as I had just discovered, it isn’t even necessary to delete the messages from within Outlook itself to trigger this behavior. Months ago when I set up Outlook for my IMAP account, I was subscribed to several Debian and other technical mailing lists that I’ve since left (these days I prefer to use the Gmane NNTP gateway to read such lists). Apparently when Outlook started up and noticed that these lists’ mail folders no longer existed on my server, it decided to send a “not read” notification for each unread message that requested one in Outlook’s old cached copy of the folders.

Some searching of the mailing list archives confirmed my hypothesis. Every recipient of one of these mystery messages had posted something to one of these lists, presumably (probably unknowingly) including a Disposition-Notification-To: header in their posts.

I can’t find the words to describe just how wrongheaded this behavior is. Not only is it semantically incorrect — how can Outlook assume that just because a message has been deleted from the IMAP server, it wasn’t first read in some other mail client before being purged? — but it’s a horrendous privacy risk too, and a boon to email address harvesters.

And in this case, it led me to waste two hours of my time as I had to diagnose this unsolicited, seemingly spam bot-like burst of messages that went out through my server.

As much as it has improved over the years, I think this proves Outlook 2010 is still entirely untrustworthy as an email client. Avoid using it if at all possible.