Tuesday, February 26, 2008

Long day

Spent the entire day doing support. Not quite what I had planned when I woke up this morning.

Ever wonder what the decision making is behind the "Pending Reboot" precheck done by the SQL Server 2005 installer? In SQL Server 2000, it just checks for the "PendingFileRenameOperation" registry key and if it's present then it says a reboot is required. Because people were annoyed by all the reboots, Microsoft changed the algorithm to be smart enough to tell if the stuff requiring a reboot had anything to do with SQL Server. Here's how they describe it:

In addition, the Microsoft Installer technology also handles the existing PendingFileRenameOperations (PFRs) much better. You may recall that SQL Server 2000 setup would block if any files at all appeared on the registry key we discussed earlier. SQL Server 2005 setup is smart enough to block only if the files belong to SQL Server (meaning there could be a conflict), and even in those cases can often reach into the PFR key and directly update it without needing a reboot. This technology is included at some level in both MSI and update.exe, which are the current standard Microsoft installers that are used to perform all updates.

From http://www.microsoft.com/technet/technetmag/issues/2006/12/SQLQA/default.aspx

Want to know what the actual algorithm is? Well, either buy my company's product or disassemble sqlspars.dll at text:498C7F40 and find out for yourself...



Sunday, February 24, 2008

New theme

So I changed the theme on my LiveJournal. Big deal. I really don't put much time into the selection process, but in this case I needed to change themes because I was tired of images and code samples more than 400 pixels wide being truncated.

Now I can put images such as the fun comic below and you can see more than the first three frames:


Finished watching the Sopranos series that I had gotten on Netflix. WTF?

I found this comic called xkcd. It's awesome! I spent an hour today going through the back comics.

Here's a small sampling:

Note to self: change my Livejournal style sheet so the layout can be wider wider than 400 pixels. I found better comics, but couldn't post them because the images get clipped.

Spent the weekend in Philadelphia with Vikki. Got some work done. Kind of a lazy weekend. We did have falafel for lunch, which was quite yummy.

We saw There Will be Blood. Very strange, but I liked it.

Friday, February 22, 2008


It's snowing here pretty good now in New York City.

Had a flashback to my teenage years, trying to remember the issues involved in walking in the snow. Don't stand too close to the road. Avoid the unexpectedly deep puddle. Walk a few feet behind someone so he/she can discover the unexpectedly deep puddle for you.

I think perhaps it's time for me to get some new shoes. The tread is pretty worn out on these and I was slipping and sliding my entire walk to work. The small hole in the toe of my left shoe is kind of annoying too. If only someone would invent some sort of thing you can put over your shoes to keep the rain/snow out. Perhaps you could make it out of rubber....

Tuesday, February 19, 2008

"Encrypted" hard drives

Somebody took the time to reverse engineer one of those new USB enclosures that claims to encrypt the hard drive with 128 bit AES, and found out it is just doing XOR encryption:


Pretty disgraceful. It turns up they encrypted the key with AES, but the actual data with a "proprietary algorithm" that in fact is trivial to break. Here's the chip the product uses. Notice that there's no mention in their specifications of a proprietary encryption algorithm or that the chip only encrypts the keying material with AES.


You just can't trust those fuckers in the computer security business. Yes, I realize my last job was as a software developer for a company that made security products. That just makes me all the more aware. I'm not saying the company I worked for made bad products but rather that the job exposed me to alot of other companies' security products (including source code in many cases), and I can say that they were all complete crap. And we're not talking about security companies that nobody has ever heard of. We're talking about top security vendors that make the firewalls and authentication tokens your company is probably using to protect it's network.

Unfortunately we're not talking about security bugs. Every product will have bugs in it and some of those are exploitable (although there are techniques that can be employed to compartmentalize the risk). Just about every vendor I worked with had fundamental design flaws in their product that were just ridiculous.

Do you really think the chip vendor above didn't realize that the design was fatally insecure? They're taking the gamble that you don't find out. You could argue that they were incompetent, but bear in mind that doing it right would have cost alot more money to design and the unit cost on the chip would be much higher. Why on Earth would they spend the money if you're too stupid to be able to tell the difference? It's pure economics.

Ever wonder why encryption products certified for government use are so damn expensive? It's because proper design, implementation, and validation is such a costly process. Programs like FIPS 140-1 in the U.S. or CESG in the U.K. are specifically setup to not allow certification of poorly designed products.

The company you work for probably paid tens of thousands of dollars for these products that are riddled with security holes that they pray you never look close enough to find out about.

Monday, February 18, 2008

Presidents Day code-a-thon

Had the day off from work for President's Day.

I've spent 29 out of the last 34 hours working on getting ATSC working under Linux for my HVR-950 (went to bed at 4am this morning). I got the lgdt3303 enabled and processing, but the xc3028 tuner is not getting a signal lock on anything in the ATSC frequency range. It's a completely digital tuner, so getting it to work is entirely driven by the microcode loaded into the device at the given time and the tuning commands sent over the i2c bus. I don't have any documentation on the registers, so I'm completely working off of usb wire dumps and comparison to the mrec driver. I miss the days at 3Com where I just had to go to the hardware library to grab the data book for whatever chip I was interested in.

Learned about all sorts of new stuff though in the last two days: 8VSB, AGC, Reed-Solomon error correction, demodulation, frequency locking, PSIP, usbmon.

Also, wasted a good portion of the day not realizing that although ATSC uses the same 6MHz bandwidth as NTSC that the channel locations do not directly correlate to the NTSC spectrum. It wasn't until I ran atsc-scan against the mrec driver that I realized that ATSC channel 25.1 locks on at 53.302 MHz. Channel 4.1 locks on at 55.702 MHz and not at 67.25. I was testing by just looking to see if I got a signal lock on the first few entries in the scan file, which it turns out weren't valid in my area even if the tuner worked perfectly.

Would be nice to have Paul around, since he actually knows something about how video works. The "Digital Video Electronics" book I bought a few weeks ago doesn't really compare to his PhD in EE.

Sunday, February 17, 2008

Belkin Flip now working with my Dell 530n

When I bought my Dell 530n with Linux a few months ago, I also went out and picked up a Belkin Flip that allows me toggle the keyboard/mouse/video between the Dell and one of my other PCs. Been generally happy with it except for one problem - the Dell hangs on boot if plugged into the Flip. Every time I rebooted, I would unplug the flip and plug in the keyboard directly. Kind of annoying since in the last few weeks I have been doing Linux kernel development which requires frequent reboots.

Finally got around to making a bootable USB thumb drive (since the PC doesn't have a floppy drive) and updated to the latest BIOS (1.0.10). Problem all fixed. PC now boots as expected when using the Flip.

On a sidenote, Linux makes it WAY too hard to make a DOS boot disk, even though FreeDOS is readily available under the GPL. I don't need DOS very often (pretty much just for BIOS upgrading), but still with Windows it's as simple as checking the "make DOS boot disk" box on the format dialog.


Had a nice weekend. Saturday went and saw Cloverfield. Didn't particularly like it. Lots of annoying unanswered questions like, "Where the hell did the monster come from?". Broke out the wok and Vikki and I made tofu and vegetable stir fry. Quite yummy. Watched some Blue Planet.

Today went to Veselka for lunch before Vikki went back to Philly. Mmmm... pierogies.

Friday, February 15, 2008

HP-UX aCC compiler bug

Been working on a crash on a program for running on hpux on and off for the last couple of days. Basically, here is where the code was crashing (simplified version of the code in question):

void PollData::toBinary(BinaryPollData *bpd) {
memset(bpd, 0, sizeof(BinaryPollData));
bpd->bpd_version = 4;
bpd->cpu_pct = 5; // <--- CRASHES HERE

Now this is a bit strange, since I am basically just setting a couple of fields in a struct that was initialized to zero). Also, why does it crash when setting the "cpu_pct" field? Why not crash on the line before when setting the "bpd_version" field?

It turns out I uncovered a compiler bug.

First, let's look at the struct definition itself:

#pragma pack (1)
struct BinaryPollData {
short bpd_version;
int cpu_pct;
#pragma pack ()

Why is the struct packed? Well, because it's used in an wire protocol that supports multiple platforms, and you want the structure definition to be consistent across compilers, CPU architectures, and bus width (32 vs 64 bit).

Shamoun came by a couple of days ago, saw that I had the disassembled program on my screen, and half jokingly asked, "how come it always comes down to the disassembly with you?" Well, here's why. After ruling out an invalid memory reference, inconsistent definition/use of the struct, use of memory after free, thread stack overflow, heap corruption, stack corruption, turning off the optimizer, and variable scoping conflict, I couldn't think of anything else to do but disassemble the function and break out the PA-RISC 1.1 Architecture and Instruction Set Reference Manual.

So here it is:

;;; void PollData::toBinary(BinaryPollData *bpd) {
0xfd920 stw %rp,-0x14(%sp)
0xfd924 stw,ma %r3,0x40(%sp)
0xfd928 stw %r4,-0x3c(%sp)
0xfd92c copy %r25,%r4
0xfd930 copy %r26,%r3
;;; memset(bpd, 0, sizeof(BinaryPollData));
0xfd934 ldi 0,%r25
0xfd938 ldi 0xce,%r24
0xfd93c call 0xfd8f8
0xfd940 copy %r4,%r26
;;; bpd->bpd_version = BPD_VERSION;
0xfd944 ldi 4,%r25
0xfd948 sth %r25,0(%r4)
;;; bpd->cpu_pct = 5;
0xfd94c ldi 5,%r26
0xfd950 stw %r26,2(%r4)

The crash happens at 0xfd950. The value of register 4 is 0x76547bd8. See it yet? Perhaps the definition of STW would help:

So the line in question:
0xfd950 stw %r26,2(%r4)
basically says "take the value in register 26 and store it in the memory location represented by taking the address in register 4 and adding 2. In other words, stick the value 5 into memory address 0x76547bda.

It boils down to the compiler not doing unaligned access properly. The address of the cpu_pct member (%r4) is 0x76547bd8 plus the offset 2. The resulting address 0x76547bda is not aligned on a 32 bit boundary, so the STW operation throws a bus error when attempting to write the content of register 26 (0x05) into that address.

What should it have done? Writing a foo.c shows the proper code emitted by the compiler should have been something along the lines of:

0x2bfc ldi 5,%r31
0x2c00 extrw,u %r31,15,16,%r19
0x2c04 sth %r19,-0x3e(%sp)
0x2c08 sth %r31,-0x3c(%sp)

The above code breaks the value into two halfwords and stores them separately.

Wednesday, February 13, 2008

Two Pedestrians Killed In Midtown

It turns up two out of three of the people hit last night on my street died.


Tuesday, February 12, 2008

Emergency vehicles

I don't quite know what happened, but there are a crapload of emergency vehicles on my street right now. There have got to be seven or eight fire trucks and half a dozen other ambulances and other trucks.

(click image to enlarge)

More ambulances showing up now. Not sure what's going on, but it's not a drill.

Update 11:07pm - it's coming across NBC News Channel 4 now. Apparently a garbage truck hit four people with at least one critically injured.

Flash and power consumption

So I was casually reading an article on CNet news last night, and all of a sudden I could hear the fan in my laptop go to full speed. To my surprise, even though I was just reading static text, some Flash applet that wasn't even in view was running my CPU at full throttle.

I have one of those Kill-a-watt devices you can plug things into to see how many power is consumed. My Macbook typically draws about 20 watts when I'm just surfing the web (it drops to less than 2 watts when it goes to sleep). As a test, I tested the consumption when sitting on the web page, and my laptop was drawing 45 watts. That's about what the laptop draws when I'm watching full screen HD quality television on my laptop.

I went to CNN.com's website, and sitting on a random article with a Flash banner ad consumes 27 watts. Sitting on the same page without the ad consumed 20 watts.

Fortunately, there is a FireFox addon to disable Flash. You can even put a button in your toolbar toggle it on/off at will, which is convenient in the rare case that you do want to see a flash animation.


The power consumption aside, flash ads are annoying and it's actually kind of nice to not see them.

Monday, February 11, 2008

Cracking open the TRS-80

CNet has a series of pics that show disassembly of a Tandy TRS-80


My first real computer was a Tandy Color Computer 2, which is pretty much just a cost reduced version of the TRS-80 with a slightly different ROM. My father bought it for me for Christmas when I was eight.


No hard disk. No floppy drive. The only secondary storage I could use to save my BASIC programs was a $12.00 cable you could buy to hook it to a cassette recorder. Radio Shack sold dedicated cassette recorders specifically for the purpose, but my old radio/tape player worked well enough.

That little computer held me over for quite a few years until I got my Apple IIc.

Sunday, February 10, 2008

Performance of unzip

I've been pondering the performance of my unzip implementation for the last few hours....

The use case that is most popular for my implementation is the unpacking of a 500 MB zip file. In most cases, the entire file is unzipped (as opposed to extracting individual files from the archive). My implementation uses standard fopen/fread/fseek for portability reasons (has to work on UNIX and Windows).

For those of you not familiar with the zip file format, it uses a central directory. Comparing it against something like tar/gzip, this means that you get the benefits of random access (meaning you can extract individual files without traversing the entire archive), but at the cost of not being able to use zip files in a streaming manner. So if you need a single file from a 500 MB archive, you just find it in the directory at the front of the file, and then fseek() to the proper offset and extract it.

However, in my case, most of the time the entire archive has to be extracted. If you are using a sequential access mechanism, this could be very expensive. Unless you are prepared to build a cache of the entire directory in memory and index the cache based on offset, your access pattern looks something like this:

Open zip directory
Process directory entry
fseek to proper offset where the compressed content is
decompress the content
fseek back to the directory
advance to next directory entry
} while (i < entries)

So you are continuously moving the seek pointer back and forth, which causes the OS to not be able to employ good buffering strategy.

mmap(), or the Win32 equivalent MapViewofFile(), could be pretty useful here. It would allow random access at fixed cost, but I am not sure what the effects are in terms of virtual memory buffering. There are also two types of buffering to consider - prefetching data I have not asked for yet and keeping in memory data I have already asked for. In my usage pattern, I really need good prefetching, since I generally only need to access a specific region of memory once. While you could argue that the regions comprising the zip directory is access multiple times, that probably uses less than 1% of the archive, so the goal would be to optimize for access to the compressed content.

More on this later...

Best Mailbox Ever!

Leave it to Aunt Cyndi and Uncle Frank to have the coolest mailbox:

Friday, February 8, 2008

cmd.exe throws a MessageBox on invalid executables?

Gotta love this. On Windows 2003 Server, you have a batch file that attempts to run an executable and the executable is corrupt, cmd.exe throws up a dialog box.

Ugh. Imagine what happens if you have a script that blocks on user input... At least on Windows XP, running the program just fails with an "Access Denied" error can lets the user handle the condition via %ERRORLEVEL%.

Why does it do this, and is there any way to tell cmd.exe to not behave this way. My original hope was perhaps there was some environment variable you could set along the lines of "NO_INTERACTIVE_MODE=1". So Let's take a closer look at what is going on under the hood by attaching a debugger to the cmd.exe while it's stuck at the dialog.

Here is the call stack taken from the MSVC debugger:

So we can see that under the hood, the command interpreter is making use of the the ShellExecute API, and that it is ShellExecute that is throwing the dialog, and not cmd.exe itself. So a quick look at the API reveals there is a flag you can send to prevent it from doing this:


If you set the SEE_MASK_FLAG_NO_UI flag in the SHELLEXECUTEINFO fMask field, errors will not cause an alert dialog to popup. Perfect.

So now we know that it's ShellExecute creating the dialog, and it can be disabled. Is there any condition under which cmd.exe will set that flag when calling ShellExecute()? Let's look at the diassembly in the debugger:

Or better yet, let's skip to two hours later when I gave up trying to analyze the above and just ran cmd.exe through the absolutely fabulous IDA Pro disassembler:

What a difference. It's moments like this when I realize how much better IDA Pro is compared to the disassembler built into Visual Studio.

From the above, it's pretty easy to see that the fMask field is hard-coded to the constant value 0x88140. So we take a look at shellapi.h in the Platform SDK, and we can see that the flag we are interested in is 0x400:

#define SEE_MASK_FLAG_NO_UI 0x00000400

So no, the flag is never set and since there is no conditional logic that dictates what gets stuck into the fMask field, its definitive that there is no way to not get the dialog box to come up if invoked through cmd.exe.

Guess I'll have to find some alternative to check the integrity of the executable before invoking it...

Wednesday, February 6, 2008

Debugging in live environments

Had to work late because a customer called and wanted me to look at a problem with a failed installation. It's always kind of cool to see your software in a real live environment. It's not the same seeing your product running on a test box in the lab as it is seeing it running on a customer's server. Partly it's the notion that someone finds your code to be valuable. You can see it's serving a real purpose - solving a real problem.

The fun happens when they call because it's not working as expected. You don't have all the safety nets and conveniences of your development environment. For one thing, if I screw up my development PC, who cares? I could just reinstall it. That obviously wouldn't fly with a customer's server. And you don't have your tools at your disposal. When code crashes in my development environment or QA, I just attach with my remote debugger and analyze the core dump. No such luck when you're VPN'd in to a customer environment.

I always like a challenge. Hard to figure out problems are what keep me coming to work every day. The best is connecting to a customer environment and then saying "What the fuck? How on Earth did the program get in this state?" Things just don't behave the same way in the wild like they do in the confines of your nice controlled environment.

Tonight's episode I encountered the Windows version of what UNIX developers would refer to as "Your listening socket got inherited across a fork call because it didn't close all the open file descriptors". It's a common problem if you've ever written a UNIX daemon - the daemon runs some other program via fork/exec and because you didn't close the file handles before calling execve(), that program inherited all the file handles from the daemon itself. This can result in problems such as socket communication errors and an inability to start the server if the original daemon gets restarted (because the listening socket is already in use).

Win32 has an equivalent - whenever you call CreateProcess(), there is a flag that dictates whether the new process inherits the handles of the parent. Setting the flag to false prevents the above described behavior. The problem comes in though when there is sufficient abstraction in your program that you don't realize this has occurred. For example, your code calls some code called RunSomePerlCode which calls a class called SpawnProcess, which calls _spawnve(), which doesn't even offer you the opportunity to tell it not to inherit the handles (so you don't think about the implications in regard to handles).

Enough rambling for one evening. Off to bed...

Friday, February 1, 2008

Wow HP-UX's aCC compiler is annoying...

You know how a traditional C compiler toolchain is typically a series of tools "chained" together. Typically it's something like:


This architecture suggests that you should be able to feed the output of one link into another. Not so with the aCC compiler that comes with HP-UX 11.0.

Check out the nice little C program (it's a proof-of-concept resulting from several hours of analysis into a problem building Perl):

#include <inttypes.h>

int main()
return UINT32_MAX ;

Compiling this with aCC shows some sort of problem with macro expansion during proprocessing:

bash-2.04# aCC -c foo.c
Error 20: "foo.c", line 5 # ';' expected before 'l'.
return UINT32_MAX ;
Error 172: "foo.c", line 5 # Undeclared variable 'l'.
return UINT32_MAX ;

So it's natural to want to see what the code looks like after preprocessing:

bash-2.04# aCC -E foo.c > out.c
bash-2.04# cat out.c
int main()
return 4294967295ul ;

Simple enough. Perhaps it doesn't handle "ul" as a type? This is where it gets interesting. The above preprocessed code compiles without any issue.

bash-2.04# aCC -c out.c

In other words, with HP-UX's compiler:

aCC -c foo.c

is not the same thing as:

aCC -E foo.c > out.c
aCC -c out.c

Very annoying.