OpenBSD comes equipped with two kernel debugging mechanisms: ddb and kgdb. ddb(4) is an in-kernel debugger, enabled by default in the GENERIC kernel, and can be invoked either explicitly from the console or automatically in the event of a panic. It is analogous to the Linux debugger kdb in that it can be used to set breakpoints and examine the stack or register state, but (like kdb) it is not a source-level debugger.

For source debugging there is kgdb(7), which offers the ability to remotely debug the kernel by way of a GDB stub running over a serial port; this is similar to the Linux debugger kgdboc. However, kgdb it is not available in the GENERIC kernel, and it imposes an additional set of configurations and debugger latencies on the user. If your debugging task is amenable to running OpenBSD within a virtual machine, as mine was, then there is an easier and better way…

The VM monitor QEMU provides its own GDB stub useful for debugging virtualized systems. For source-level debugging this still requires obtaining a kernel compiled with the appropriate debug info, but the stub makes no additional requirements of the guest OS and performs very well. The necessary QEMU tools can be installed on Ubuntu 12.04 as follows:

$ sudo apt-get install qemu-kvm qemu-utils

To start with, you’ll need an OpenBSD installation in a QEMU virtual machine. Prepare your virtual disk image (20GB will be more than enough):

$ qemu-img create -f qcow2 ~/obsd52.qcow2 20G

Next, download the OpenBSD i386 install CD image from a mirror, such as:

ftp://ftp3.usa.openbsd.org/pub/OpenBSD/5.2/i386/install52.iso

Now boot a VM from the OpenBSD installation CD, with the QCOW2 image that you just created as your virtual disk image. Aside from setting a system RAM size of 256 MB, the options specified here will provide a userspace NAT virtual network interface for the VM:

$ qemu-system-i386 -m 256M -net nic -net user -cdrom ~/install52.iso \
      ~/obsd52.qcow2

Wait for the CD image to boot and then install OpenBSD in the usual manner. When prompted, specify that the SSH server should be started by default, as this will be needed shortly. Reboot into the new guest OS when the installer has finished.

OpenBSD isn’t yet ready for debugging, though – although the system boots, its kernel does not provide the debug symbols needed by GDB. So on the guest system now, download and install the system kernel source code from an FTP mirror:

obsd# cd /tmp
obsd# ftp ftp://ftp3.usa.openbsd.org/pub/OpenBSD/5.2/sys.tar.gz
obsd# cd /usr/src
obsd# tar xzf /tmp/sys.tar.gz

Make a new kernel configuration based on the i386 GENERIC config, then edit this config to turn on debug symbol generation by adding the line makeoptions DEBUG="-g" under the option lines:

obsd# cd /usr/src/sys/arch/i386/conf
obsd# cp GENERIC DEBUG
obsd# vi DEBUG

Now you can build and install the new kernel. Specifying the COPTS environment variable here overrides the default optimization level -O2, which will make control flow much easier to follow while single-stepping in the debugger:

obsd# config DEBUG
obsd# cd ../compile/DEBUG
obsd# make depend
obsd# COPTS="-O0" make
obsd# make install

Halt the OpenBSD guest and exit QEMU, and then restart the VM with additional options -s and -redir, to enable the GDB stub and access to the VM’s SSH service respectively:

$ qemu-system-i386 -m 256M -net nic -net user -s \
      -redir tcp:2022::22 ~/obsd52.qcow2

When the guest is booted, SCP from the host to copy the kernel source tree and build output.

$ scp -pr -P2022 localhost:/usr/src/sys ~/obsd52-sys

With debug output enabled the OpenBSD kernel build produces an unstripped ELF kernel image named bsd.gdb, which is suitable for use by GDB. Load this into GDB and then attach to QEMU’s debug stub (which defaults to TCP port 1234):

$ cd ~/obsd52-sys/arch/i386/compile/DEBUG
$ gdb bsd.gdb
(gdb) target remote :1234

The VM will halt when you connect the debugger, leaving you free to set system call breakpoints or inspect state before continuing.

This went from mildly annoying to a major headache when I needed to run the debugger in a remote Emacs session, over SSH from Mac OS X’s Terminal.app which does not support xterm mouse emulation. So I wrote this gdb-select-window function and corresponding key bindings to finally get the desired behavior…

Put this in your ~/.emacs.el or init.el and fire up gdb-many-windows, and then you’ll be able to quickly switch between the GDB windows with the following keyboard shortcuts:

• C-c C-g c — comint (GDB command) buffer
• C-c C-g l — locals buffer
• C-c C-g r — registers buffer
• C-c C-g u — source window
• C-c C-g s — stack buffer
• C-c C-g b — breakpoints buffer
• C-c C-g t — threads buffer

I ran into a really fun bug at work yesterday, where I discovered that my C program was branching down logically inconsistent code paths. After drinking another cup of coffee and firing up GDB I realized that somehow, a boolean variable in my code was simultaneously testing as both true and not true.

While I cannot reproduce the actual source code here, the effect was that code like

bool p;
 
/* ... */
 
if ( p )
    puts("p is true");
 
if ( ! p )
    puts("p is false");

would produce the output:

p is true
p is false

So what’s going on here?

Well it turns out that the authors of the C language specification (and the people who went on to implement compilers for it) were serious about the concept of undefined behavior. In particular, the result of attempting to use an uninitialized variable is undefined.

And in this case that’s exactly what happened: I had failed to properly initialize some memory. Easy, bug fixed. But what I think is interesting are the reasons this code failed in precisely the way that it did. In order to investigate that, we need to get specific.

On 64-bit Linux (Ubuntu 12.04), compiling the following program:

#include <stdio.h>
#include <stdbool.h>
 
int main(int argc, char *argv[])
{
    volatile bool p;
 
    if ( p )
        puts("p is true");
    else
        puts("p is not true");
 
    if ( ! p )
        puts("p is false");
    else
        puts("p is not false");
 
    return 0;
}

with GCC 4.6.3, using the command line:

$ gcc bool1.c -g0 -O0 -fno-dwarf2-cfi-asm -masm=intel -S -o bool1.asm

produces this (truncated) assembly language:

        .file   "bool1.c"
        .intel_syntax noprefix
        .section        .rodata
.LC0:
        .string "p is true"
.LC1:
        .string "p is not true"
.LC2:
        .string "p is false"
.LC3:
        .string "p is not false"
        .text
        .globl  main
        .type   main, @function
main:
.LFB0:
        push    rbp
.LCFI0:
        mov     rbp, rsp
.LCFI1:
        sub     rsp, 32
.LCFI2:
        mov     DWORD PTR [rbp-20], edi
        mov     QWORD PTR [rbp-32], rsi
        movzx   eax, BYTE PTR [rbp-1]
        test    al, al
        je      .L2
        mov     edi, OFFSET FLAT:.LC0
        call    puts
        jmp     .L3
.L2:
        mov     edi, OFFSET FLAT:.LC1
        call    puts
.L3:
        movzx   eax, BYTE PTR [rbp-1]
        xor     eax, 1
        test    al, al
        je      .L4
        mov     edi, OFFSET FLAT:.LC2
        call    puts
        jmp     .L5
.L4:
        mov     edi, OFFSET FLAT:.LC3
        call    puts
.L5:
        mov     eax, 0
        leave
.LCFI3:
        ret

To perform the test if ( p ) here, first the stack variable is loaded into a 32-bit register with movzx eax, BYTE PTR [rbp-1], and then we use the instruction test al, al which sets the zero flag (ZF) if the lower eight bits of this value are zero. Next we execute the conditional jump je .L2, which jumps to print “p is not true” if ZF was set; otherwise we don’t jump, and “p is true” gets printed instead.

Next let’s examine the second test, if ( ! p ), at label .L3. This starts out the same by loading the boolean variable into register eax, but notice how the negation is handled. Rather than reorder the jumps or use jne instead of je, the compiler explicitly negates the boolean by performing a bitwise exclusive-or: xor eax, 1.

Normally this would be fine — a bool variable is only supposed to contain a value of zero or one, in which case its value can be negated by XOR with 1. When you cast to a bool at runtime, the compiler generates code to ensure only one or zero gets stored. For instance, the cast in this program:

#include <stdbool.h>
 
volatile char c = 0xff;
volatile bool p;
 
int main(int argc, char* argv[])
{
    p = (bool)c;
    return 0;
}

is implemented as the following four instructions:

        movzx   eax, BYTE PTR c[rip]
        test    al, al
        setne   al
        mov     BYTE PTR p[rip], al

wherein setne sets the register al to exactly 1 if the char contained any nonzero value, before saving the register’s 8-bit value to the boolean variable.

But the compiler affords us no such protection if we accidentally use an uninitialized value as a boolean. It doesn’t have to, it’s not the compiler’s responsibility; the result of using an uninitialized stack variable is undefined. And so if we somehow wind up with a value of e.g. 0x60 stored at the address of a bool variable (as I saw during my troubleshooting yesterday), both the variable and its negation (via exclusive or with 1) will be nonzero, and therefore test as true.

Interestingly, enabling optimization (-O2) in GCC causes the compiler to factor out the XOR and instead reorder the jumps, meaning this program actually behaves more robustly under compiler optimization (for certain definitions of “robust” anyway):

        .file   "bool1.c"
        .intel_syntax noprefix
        .section        .rodata.str1.1,"aMS",@progbits,1
.LC0:
        .string "p is true"
.LC1:
        .string "p is not true"
.LC2:
        .string "p is false"
.LC3:
        .string "p is not false"
        .section        .text.startup,"ax",@progbits
        .p2align 4,,15
        .globl  main
        .type   main, @function
main:
.LFB22:
        sub     rsp, 24
.LCFI0:
        movzx   eax, BYTE PTR [rsp+15]
        test    al, al
        je      .L2
        mov     edi, OFFSET FLAT:.LC0
        call    puts
.L3:
        movzx   eax, BYTE PTR [rsp+15]
        test    al, al
        je      .L7
        mov     edi, OFFSET FLAT:.LC3
        call    puts
.L5:
        xor     eax, eax
        add     rsp, 24
.LCFI1:
        ret

And of course when we compare char, int, or other multiple-bit values for truthiness, the compiler makes no such assumption that the value can be logically negated by bitwise XOR; instead it uses jne in place of the je instruction. (Maybe someone with more knowledge of the compiler can say why GCC with -O0 uses an xor at all, when testing the negation of a bool.)

from pointfree import *
from operator import add

fn = pfmap(len) \
     >> pfmap(lambda n: n**2) \
     >> pfreduce(add, initial=0)

fn(["foo", "barr", "bazzz"]) # == 50

See the overview for a lengthier introduction. You can install the pointfree module from the Python Package Index:

$ pip install pointfree

Links:

• Module documentation
• Github project page
• Python Package Index page

Eventually I figured it out: basic accounting is just graph theory. The traditional ways of representing financial information hide that structure astonishingly well, but once I had figured out that it was just a graph, it suddenly all made sense.

He goes on to illustrate how a profit-and-loss statement and a balance sheet can be visualized on a simple DAG.  Good stuff…

But before I go digging through my old CDs I need to identify which albums I have to re-encode. My music is organized in folders by artist and then album, e.g.:

~\Music\Library\Pixies\Surfer Rosa\Where Is My Mind.mp3

So this means I effectively have to list the names of folders containing non-MP3 music files. Fortunately Windows PowerShell makes this a one-liner (though admittedly it’s a pretty long line; the backtick is PowerShell’s line continuation syntax):

ls -r Music\Library `
| ?{ $_.PSIsContainer -And ( $_.GetFiles() `
| ?{ $_.Name -Match "\.(m4[ap]|ogg|wma)$" } ) } `
| %{ New-Object PSObject -Property `
@{ Artist = (gi $_.PSParentPath).Name; Album = $_.Name } }

This will give you a nice list of non-MP3 albums, like:

Album              Artist
-----              ------
That's Your Fire   Aloha
Noble Beast        Andrew Bird
Believe It Mammals Bats & Mice
Charm School       Bishop Allen
[...]

And thanks to PowerShell’s object-oriented pipes, this is done without having to worry about any quoting or whitespace issues.

The more I learn PowerShell, the less I find myself relying on Cygwin for simple Windows administration tasks. I think Microsoft did a nice job balancing all the different requirements they had to meet with PowerShell. At the very least, it’s good to finally see a strong command shell built into Windows.

This is a quick guide to booting the OpenBSD installer on a PC Engines ALIX board with tinyBIOS (such as the ALIX 2d3) via PXE, using just the following:

• PC with two network interfaces. One of these needs to be Ethernet, and the other must connect to the Internet. For example, any standard PC laptop with both WiFi and Ethernet adapters will work if there’s a WiFi Internet connection available.
• Null modem cable
• Ethernet crossover cable
• USB-serial adapter (unless your PC has a built-in RS-232 port)
• Ubuntu Linux 10.10 desktop live CD

Thanks to the versatility of the Ubuntu live CD (specifically the use of AUFS to provide a writable root directory in RAM), you can set up the necessary PXE boot server without making any permanent changes to your PC.

Ubuntu packages

Boot the Ubuntu live CD and quit the installer. Ensure that Ubuntu has a working Internet connection, then enable the “universe” package repository by uncommenting the corresponding lines in /etc/apt/sources.list. Now open a terminal and run the following commands to install prerequisite packages:

$ sudo -s
# apt-get update
# apt-get install dhcp3-server tftpd xinetd cu

Network configuration & NAT

Run this command to configure a static address on the Ethernet interface:

# ifconfig eth0 up 192.168.2.1 netmask 255.255.255.0

I’ve found you may also need to configure the static address in the “Network Connections” dialog (under Preferences in the System menu) to prevent Network Manager from getting in the way. This is sort of hackish, but we only need it to work for the duration of the install.

Now enable routing and configure a simple NAT using iptables so that the ALIX board can access the internet through your PC’s wireless connection:

# echo 1 > /proc/sys/net/ipv4/ip_forward
# iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
# iptables -A FORWARD -i eth0 -j ACCEPT
# iptables -A FORWARD -i wlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT

Connect your PC’s Ethernet port to the first port on your ALIX board using the crossover cable. On the ALIX 2d3, the first port is the one adjacent to the USB ports.

DHCP server

Replace /etc/dhcp3/dhcpd.conf with the following contents:

authoritative;

shared-network LOCAL-NET {
    option domain-name-servers 8.8.8.8;

    subnet 192.168.2.0 netmask 255.255.255.0 {
        option routers 192.168.2.1;
	filename "pxeboot";
	range 192.168.2.100 192.168.2.200;
	default-lease-time 600;
	max-lease-time 7200;
    }
}

Also, edit the file /etc/default/dhcp3-server so that the last line reads:

INTERFACES="eth0"

Now you can start the DHCP server.

# /etc/init.d/dhcp3-server start

TFTP server

Create an xinetd file /etc/xinetd.d/tftp as:

service tftp
{
    socket_type = dgram
    protocol = udp
    wait = yes
    user = root
    server = /usr/sbin/in.tftpd
    server_args = -s /tftpboot
}

Next create the aforementioned directory /tftpboot. Download the files bsd.rd and pxeboot from the /4.8/i386/ directory on your favorite OpenBSD mirror and copy them into this directory.

Restart xinetd to load the new configuration.

# /etc/init.d/xinetd restart

Serial console

Connect your laptop’s serial port (or plugged-in USB-serial adapter) to the ALIX board’s serial port with your null modem cable, then use the cu command to connect to the serial console. For example, if you’re using a USB adapter and your ALIX’s BIOS has the default serial port settings:

# cu -e -o -s 38400 -l /dev/ttyUSB0

PXE boot

With your serial console ready, plug in the ALIX board’s power adapter, and you should see the board begin to boot. While the memory check is being performed, press the ‘s’ key to enter the tinyBIOS settings, and verify that PXE boot is enabled (if it isn’t, press ‘e’ to toggle it).

After exiting the BIOS settings menu, the board will reboot. It should find your PXE server and bring you to an OpenBSD boot menu. Enter the following at this menu:

boot> stty com0 38400
boot> set tty com0
boot> bsd.rd

The installer will boot from the bsd.rd image that you downloaded. Now perform the installation as normal, but remember to configure the serial port as your system console in the installer.

For example, this query lists only free Kindle books in the public domain. Very handy.

I don’t typically use Microsoft Outlook with my home email account. But it was bundled with the copy of Office 2010 that I installed a few months ago, and I figured: heck, if I have it anyway I might as well hook it up to my personal IMAP account. So I configured Outlook, played with it a bit, then promptly forgot all about it.

Fast forward to yesterday evening. I launch my copy of Outlook for the first time in months in order to try something with the calendar, but then I get distracted and walk away from the computer. Five minutes later my cell phone gets a message from my FreeBSD server:

EMAIL SERVER HIJACKED – FIREWALLING POSTFIX

Oh. Fun.

A slight digression before I can get on with the story: I try my very best to be a good Internet citizen, and that of course means not allowing oneself to become a spam relay. Because this email server exists only for my personal use, it was simple to write a Perl script that monitors my Postfix logs and, if it sees anything grossly out of the ordinary sent out through the server, reconfigures PF to block outbound connections to TCP port 25. It seemed the responsible thing to do, especially since some PHP applications on my web server have permission to relay through this Postfix instance; and let’s face it, WordPress has a less than stellar security track record.

This script has never given me a false positive. So this was definitely cause for concern.

I shelled into my VPS and went straight for the mail logs. I was dismayed to find more than twenty entries like this:

Dec  8 23:35:38 frodo postfix/smtpd[13146]: 13DB43F4DC: \
    client=redacted[xxx.xxx.xxx.xxx], sasl_method=LOGIN, sasl_username=redacted
Dec  8 23:35:39 frodo postfix/cleanup[13148]: 13DB43F4DC: \
    replace: header Received: from noatun (redacted [xxx.xxx.xxx.xxx]) \
(Authenticated sender: redacted)
by frodo.paleogene.net (Postfix) with ESMTPSA id 13DB43F4DC
for <redacted@example from redacted[xxx.xxx.xxx.xxx]; \
    from=<redacted@example.com> to=<redacted@example.com> \
    proto=ESMTP helo=<noatun>: Received: from auth-client.paleogene.net \
    (auth-client.paleogene.net [206.125.175.178])
(Authenticated sender: hidden)
by frodo.paleogene.net (Postfix) with ESMTPSA id 13DB43F4DC
for <redacted@example.com>; Wed,  8 Dec 2010 23:35:37 -0500 (EST)

All of them sent within seconds of one another, and all of them to random Gmail accounts and other recipients that I absolutely did not recognize. Worse, the messages had been sent through a properly SASL authenticated connection from my laptop (HELO noatun). To quote that old horror flick: They’re coming from inside the house!

I’ve heard tell of Windows viruses that will quietly send spam through the user’s configured Outlook mail account, so that the messages originate from a legitimate mail server and have a better chance of skipping recipients’ spam folders. I have no idea whether this is actually a common occurrence in the real world, but it’s plausible enough, and I couldn’t think of any better explanation why 24 messages had been sent out from my computer in a matter of seconds, all to recipients I’d never heard of. I still could not fathom how a spam bot might have made its way onto my laptop despite my precautions, but better safe than sorry…

Process Explorer and TCPView failed to reveal anything suspicious, so I shut down the laptop, yanked out its hard drive, hooked that up to another machine via a USB-SATA adapter, and started a full offline virus scan.

Meanwhile I kept poking around on the mail server. I had a bit of luck when I checked /var/spool/postfix/deferred — a few messages had been caught in the Postfix outbound queue when my script added its firewall rule. I was morbidly curious what had been sent.

…it was a freaking Message Disposition Notification.

Now sure, I knew all about email read receipts, but I’d immediately ruled them out as the cause of this behavior because all of my email clients, Outlook included, are configured to at least ask my permission before sending one:

Outlook completely ignores this

And anyway, I hadn’t read or deleted any email messages in Outlook, so why would it want to send any MDNs in the first place?

I did some googling and the pieces slowly began falling into place. It turns out there’s a horrific bug in recent Outlook versions’ handling of read receipts: unread messages deleted from an IMAP folder can send a “not read” MDN, even if you’ve explicitly configured Outlook not to do so.

And as I had just discovered, it isn’t even necessary to delete the messages from within Outlook itself to trigger this behavior. Months ago when I set up Outlook for my IMAP account, I was subscribed to several Debian and other technical mailing lists that I’ve since left (these days I prefer to use the Gmane NNTP gateway to read such lists). Apparently when Outlook started up and noticed that these lists’ mail folders no longer existed on my server, it decided to send a “not read” notification for each unread message that requested one in Outlook’s old cached copy of the folders.

Some searching of the mailing list archives confirmed my hypothesis. Every recipient of one of these mystery messages had posted something to one of these lists, presumably (probably unknowingly) including a Disposition-Notification-To: header in their posts.

I can’t find the words to describe just how wrongheaded this behavior is. Not only is it semantically incorrect — how can Outlook assume that just because a message has been deleted from the IMAP server, it wasn’t first read in some other mail client before being purged? — but it’s a horrendous privacy risk too, and a boon to email address harvesters.

And in this case, it led me to waste two hours of my time as I had to diagnose this unsolicited, seemingly spam bot-like burst of messages that went out through my server.

As much as it has improved over the years, I think this proves Outlook 2010 is still entirely untrustworthy as an email client. Avoid using it if at all possible.

In some cases, there is a way to tell good arguments from bad using what is called informal logic. This name distinguishes it from formal logic, which is used in mathematics; natural language is less precise than mathematics, and does not always follow the same rules. Perhaps more surprisingly, the name also reflects the fact that there is a lot of disagreement over what it means. Informal logic is actually a fairly young discipline, developed in the 1960s, and intended to apply new techniques from formal logic to argument and critical thinking. Philosophers are still wrestling with this application, and there are several competing schools of thought.

For all the differences, there are some core elements upon which everyone agrees. This article is a tutorial introducing these basic ideas of informal logic. In particular, we focus on deductive reasoning, which is one of the cornerstones of analytical thought. Hopefully you will come away from this article with the tools to distinguish a productive debate from an acrimonious flame war.

This should be required reading for anyone who’s engaged in debate on a mailing list or web forum.

How am I just finding this now? He’s made major progress since filming that video; check out his YouTube channel for more. Seriously man, thumbs up…