Wednesday, July 12, 2006

Linux kernel debugging

So I bought a PCI 802.11g wireless card for my desktop computer a few weeks ago. Specifically went with a particular model (Zyxel) because it advertised Linux support. I'm tired of the 25 foot piece of Cat 5 cable stretched across my living room floor.

Well, it intermittently hangs the PC. And I mean "have to reach down and hit the power switch" hang.

Since I have a bit of experience with Linux kernel debugging, I figured I would take a look. It's surprising though how much I have taken for granted with my development environment at Netilla. What I thought would be a simple operation was, well, not so much.

First off, there's X11. Having a system with X makes it much more difficult to debug since when it locks up you can't get access to the console. So you need a serial console, which means you need another computer. Having the logs go to a second computer is a good idea anyway, since you can save them to a file (something which you typically can't do on a system that has panic'd)

For me, that meant I needed to:
  • Get a USB to serial adapter for my iBook
  • Download and install OpenDarwin
  • Install minicom
  • Find my serial cable
  • Find my null modem adapter
  • Reconfigure inittab to spawn a console on /dev/ttyS0
  • Modify grub to support console on both tty0 and ttyS0 (setting the baud rate appropriately)
  • Increasing the kernel log level
  • Enable the magic SysRq key
  • Download the Suse kernel source
  • Figure out where the hell Suse keeps third-party kernel module source code

I am familar with each of these steps since I was responsible for kernel debugging at Netilla, but I guess I didn't appreciate what a pain in the ass it would be to reproduce such an environment in my apartment.

And so I do all this and I am now getting kernel debug output to dump to minicom and save to a file on my iBook. And of course, I have been successfully connected through my wireless card for the last hour and now I can't reproduce the problem. That's just my luck....