Cracking open the TRS-80

CNet has a series of pics that show disassembly of a Tandy TRS-80

http://www.news.com/Photos-Cracking-open-the-TRS-80/2300-1042_3-6229672.html

My first real computer was a Tandy Color Computer 2, which is pretty much just a cost reduced version of the TRS-80 with a slightly different ROM. My father bought it for me for Christmas when I was eight.

http://en.wikipedia.org/wiki/TRS-80_Color_Computer#Color_Computer_2_.281983.E2.80.931986.29

No hard disk. No floppy drive. The only secondary storage I could use to save my BASIC programs was a $12.00 cable you could buy to hook it to a cassette recorder. Radio Shack sold dedicated cassette recorders specifically for the purpose, but my old radio/tape player worked well enough.

That little computer held me over for quite a few years until I got my Apple IIc.

Performance of unzip

I've been pondering the performance of my unzip implementation for the last few hours....

The use case that is most popular for my implementation is the unpacking of a 500 MB zip file. In most cases, the entire file is unzipped (as opposed to extracting individual files from the archive). My implementation uses standard fopen/fread/fseek for portability reasons (has to work on UNIX and Windows).

For those of you not familiar with the zip file format, it uses a central directory. Comparing it against something like tar/gzip, this means that you get the benefits of random access (meaning you can extract individual files without traversing the entire archive), but at the cost of not being able to use zip files in a streaming manner. So if you need a single file from a 500 MB archive, you just find it in the directory at the front of the file, and then fseek() to the proper offset and extract it.

However, in my case, most of the time the entire archive has to be extracted. If you are using a sequential access mechanism, this could be very expensive. Unless you are prepared to build a cache of the entire directory in memory and index the cache based on offset, your access pattern looks something like this:

Open zip directory
{
Process directory entry
fseek to proper offset where the compressed content is
decompress the content
fseek back to the directory
advance to next directory entry
} while (i < entries)

So you are continuously moving the seek pointer back and forth, which causes the OS to not be able to employ good buffering strategy.

mmap(), or the Win32 equivalent MapViewofFile(), could be pretty useful here. It would allow random access at fixed cost, but I am not sure what the effects are in terms of virtual memory buffering. There are also two types of buffering to consider - prefetching data I have not asked for yet and keeping in memory data I have already asked for. In my usage pattern, I really need good prefetching, since I generally only need to access a specific region of memory once. While you could argue that the regions comprising the zip directory is access multiple times, that probably uses less than 1% of the archive, so the goal would be to optimize for access to the compressed content.

More on this later...

Best Mailbox Ever!

Leave it to Aunt Cyndi and Uncle Frank to have the coolest mailbox:

cmd.exe throws a MessageBox on invalid executables?

Gotta love this. On Windows 2003 Server, you have a batch file that attempts to run an executable and the executable is corrupt, cmd.exe throws up a dialog box.



Ugh. Imagine what happens if you have a script that blocks on user input... At least on Windows XP, running the program just fails with an "Access Denied" error can lets the user handle the condition via %ERRORLEVEL%.

Why does it do this, and is there any way to tell cmd.exe to not behave this way. My original hope was perhaps there was some environment variable you could set along the lines of "NO_INTERACTIVE_MODE=1". So Let's take a closer look at what is going on under the hood by attaching a debugger to the cmd.exe while it's stuck at the dialog.

Here is the call stack taken from the MSVC debugger:



So we can see that under the hood, the command interpreter is making use of the the ShellExecute API, and that it is ShellExecute that is throwing the dialog, and not cmd.exe itself. So a quick look at the API reveals there is a flag you can send to prevent it from doing this:

http://msdn2.microsoft.com/en-us/library/bb762154(VS.85).aspx
http://msdn2.microsoft.com/en-us/library/bb759784(VS.85).aspx

If you set the SEE_MASK_FLAG_NO_UI flag in the SHELLEXECUTEINFO fMask field, errors will not cause an alert dialog to popup. Perfect.

So now we know that it's ShellExecute creating the dialog, and it can be disabled. Is there any condition under which cmd.exe will set that flag when calling ShellExecute()? Let's look at the diassembly in the debugger:



Or better yet, let's skip to two hours later when I gave up trying to analyze the above and just ran cmd.exe through the absolutely fabulous IDA Pro disassembler:



What a difference. It's moments like this when I realize how much better IDA Pro is compared to the disassembler built into Visual Studio.

From the above, it's pretty easy to see that the fMask field is hard-coded to the constant value 0x88140. So we take a look at shellapi.h in the Platform SDK, and we can see that the flag we are interested in is 0x400:

#define SEE_MASK_FLAG_NO_UI        0x00000400

So no, the flag is never set and since there is no conditional logic that dictates what gets stuck into the fMask field, its definitive that there is no way to not get the dialog box to come up if invoked through cmd.exe.

Guess I'll have to find some alternative to check the integrity of the executable before invoking it...

Debugging in live environments

Had to work late because a customer called and wanted me to look at a problem with a failed installation. It's always kind of cool to see your software in a real live environment. It's not the same seeing your product running on a test box in the lab as it is seeing it running on a customer's server. Partly it's the notion that someone finds your code to be valuable. You can see it's serving a real purpose - solving a real problem.

The fun happens when they call because it's not working as expected. You don't have all the safety nets and conveniences of your development environment. For one thing, if I screw up my development PC, who cares? I could just reinstall it. That obviously wouldn't fly with a customer's server. And you don't have your tools at your disposal. When code crashes in my development environment or QA, I just attach with my remote debugger and analyze the core dump. No such luck when you're VPN'd in to a customer environment.

I always like a challenge. Hard to figure out problems are what keep me coming to work every day. The best is connecting to a customer environment and then saying "What the fuck? How on Earth did the program get in this state?" Things just don't behave the same way in the wild like they do in the confines of your nice controlled environment.

Tonight's episode I encountered the Windows version of what UNIX developers would refer to as "Your listening socket got inherited across a fork call because it didn't close all the open file descriptors". It's a common problem if you've ever written a UNIX daemon - the daemon runs some other program via fork/exec and because you didn't close the file handles before calling execve(), that program inherited all the file handles from the daemon itself. This can result in problems such as socket communication errors and an inability to start the server if the original daemon gets restarted (because the listening socket is already in use).

Win32 has an equivalent - whenever you call CreateProcess(), there is a flag that dictates whether the new process inherits the handles of the parent. Setting the flag to false prevents the above described behavior. The problem comes in though when there is sufficient abstraction in your program that you don't realize this has occurred. For example, your code calls some code called RunSomePerlCode which calls a class called SpawnProcess, which calls _spawnve(), which doesn't even offer you the opportunity to tell it not to inherit the handles (so you don't think about the implications in regard to handles).

Enough rambling for one evening. Off to bed...

The only valid measurement of code quality

From OSNews Comics

Wow HP-UX's aCC compiler is annoying...

You know how a traditional C compiler toolchain is typically a series of tools "chained" together. Typically it's something like:

preprocessor->compiler->assembler->linker

This architecture suggests that you should be able to feed the output of one link into another. Not so with the aCC compiler that comes with HP-UX 11.0.

Check out the nice little C program (it's a proof-of-concept resulting from several hours of analysis into a problem building Perl):

#include <inttypes.h>

int main()
{
  return UINT32_MAX ;
}

Compiling this with aCC shows some sort of problem with macro expansion during proprocessing:

bash-2.04# aCC -c foo.c
Error 20: "foo.c", line 5 # ';' expected before 'l'.
      return UINT32_MAX ;
             ^^^^^^^^^^
Error 172: "foo.c", line 5 # Undeclared variable 'l'.
      return UINT32_MAX ;
             ^^^^^^^^^^

So it's natural to want to see what the code looks like after preprocessing:

bash-2.04# aCC -E foo.c > out.c
bash-2.04# cat out.c
....
int main()
{
 return 4294967295ul ;
}

Simple enough. Perhaps it doesn't handle "ul" as a type? This is where it gets interesting. The above preprocessed code compiles without any issue.

bash-2.04# aCC -c out.c
bash-2.04#

In other words, with HP-UX's compiler:

aCC -c foo.c

is not the same thing as:

aCC -E foo.c > out.c
aCC -c out.c

Very annoying.