Train ride home

Caught the 8:44pm from Belmar tonight to head back to the city.

I dozed off and woke up a few minutes after 9:15 and noticed the train was stopped and I could see the lights from Monmouth Park off in the distance. Half awake, my eyes drifted downward and I saw what looked like a headlight on the ground next to the train. And in fact, it was a headlight. There was a white car facing backward on the adjacent train track immediately outside my window.

At this point a cop car sped up to the road next to the train. And then another, and then a fire engine, and another cop car. Within fifteen minutes, there were seven cop cars, three fire engines and an ambulance there.

Apparently the driver wasn't in the car when it was hit by the train, but I could see a cop talking to some people around the car suggesting they were the owners (they did not appear to be injured).

It took about an hour for a tow truck to remove the car from the track, after which the train pulled away. From the way the firemen kept looking under the car with flashlights, it would appear that the midsection of the car was resting on the rail such that left two wheels were not in contact with the ground.

I finally walked through my door at 11:45pm tonight. Ugh.

I'll have to check the New Jersey papers tomorrow.

Em28xx analog device merge

Today Mauro and Douglas merged a bunch of analog device support from Markus's em28xx tree:

http://linuxtv.org/hg/v4l-dvb/rev/572c09c6861b

It's hard for me not to be conflicted about this decision. On one hand, I want to see as much device support as possible get into the mainline kernel. On the other hand, it continues to make worse the problem about duplicated functionality within the two em28xx trees. Markus put alot of hard work into making those devices work by testing them himself or working with people all over the world who have the devices.

While it is certainly permissible under the GPL to do what they did, this leaves a bad taste in my mouth for lack of a better analogy.

I wish there was a better way for the two groups to work together. I appealed to Markus again last week to see if we could find some middle ground, and let's just say it didn't go well at all.

In Philadelphia this weekend

In Philly this weekend.

Spent a good portion of yesterday analyzing the dib0700 Linux driver source code and researching the other components in the Pinnacle PCTV HD Pro USB device I own. I cannot debug the driver on my MacBook, so I figured I would at least do the prep work so I can analyze the traces. Sent some email to the linux-dvb mailing list to see if anybody knows where the GPIO pins are.

Also made some updates to the linux-dvb wiki for various devices I do work on.

My changes for the ATI TV Wonder 600 USB got merged in yesterday:

http://linuxtv.org/hg/v4l-dvb/rev/4898b1b582f2

This afternoon, Victoria and I are going to New Jersey to celebrate Lauren and Suzy's birthday.

The Last Hope

Attended The Last HOPE conference over the weekend.

Here's a list of the talks I went to:

• A Collaborative approach to Hardware Hacking: NYCResistor
  
• The New York City Taxi System: Privacy vs. Utility
  
• IPv6, The Next Generation Network Playground: how to connect and explore
  
• A Decade under the DMCA
  
• The Intersection of Culture Jamming, Hacking, and Hacktivism
  
• Escaping High Security Handcuffs
  
• Grand Theft Lazlow: Hacking the Media by Laughing at Them
  
• Featured Speaker: Adam Savage (co-host of MythBusters)
  
• Strengths and Weaknesses of Physical Access Control Systems
  
• No-Tech hacking
  

It was lots of fun. I told a coworker about the talk on high security handcuffs and his response was "What on Earth are you doing in your free time?"

In addition to the vendor area, they had an area setup where they were teaching people basic electronics, how to solder, and helping beginners build kits with microcontrollers. They also taught a class and helped 60 people take the ham radio license exam, of which more than 70% passed (which I hear is unusually high).

Low-tech

In response to that user's thermal problem I was talking about last night with the ATI TV Wonder HD 600, picked up a $23.00 meat thermometer and hooked it up to my Pinnacle PCTV HD card so I can do some analysis of thermal properties under various conditions (Windows driver vs. Linux, capturing versus idle, digital versus analog capture).



It ain't pretty, but it gets the job done. Besides, I'm a vegetarian so it's not like I have alot of other uses for a thermometer branded to "Cook meat, poultry, & fish safely and accurately with perfect results!" I guess when I'm done I can give it to a coworker, as long as they don't have a problem with the taste of rosin flux. Yummy!

Speaking of ridiculous...

So, this made its way into the mainstream press:

Hacker Holds Key to City's Network
An Alleged Hacker Won't Reveal Secret Password to Unlock San Francisco's Network
http://abcnews.go.com/Technology/story?id=5390020&page=1

Yeah, so I guess they've never heard of single user mode, or the equivalent for Cisco IOS. If you have access to the hardware, it takes about five minutes to reset the master password.

Repeat after me:

Physical Access is Total Access
Physical Access is Total Access
Physical Access is Total Access

em28xx frustrations...

For the last few days I've been working with an owner of an ATI TV Wonder HD 600 USB, yet another em28xx based device.

He's reporting that he believes the device runs hotter under Linux than under Windows. Now, I already know that we're not powering down the device at idle, so essentially it's always running at full power. And at least the HVR-950 I have runs pretty hot to begin with.

So, the question is: is this guy right? I'm certainly not suggesting he's lying, but is his assertion correct? I really need to find some way to quantitatively tell if the device's thermal profile differs, because if I'm driving components improperly it could shorten their life or burn them out. I guess I can rig up some sort of thermal probe that I can tape to the F-connector with some thermal paste.

But this again got me thinking about Markus and his em28xx driver.

The reality is that I don't know if the device is being properly configured. All of the power management is from reverse engineering via analysis of USB traces. The entire V4L driver is a product of guessing the functionality of various registers through trial and error.

In other words, if I had the damn datasheet I could just look up how to properly program the device registers.

Markus has the datasheet because he works for Empiatech. He maintains his own driver separate from the V4L mainline tree.

http://mcentral.de/hg/~mrec/em28xx-new/shortlog

It's licensed under the GPL, but for a number of reasons (some technical and some political), I can't see how it will ever get into the mainline Linux kernel.

So here we have me, Mauro, Douglas, Aidan and a few others hacking away at the "community" driver through reverse engineering, when the vendor of the chipset is paying someone to write and maintain a GPL'd driver based off of the datasheets. This is just stupid.

Can't we all just get along?

The reality is that I don't see this as a competition. It's not about whose driver is "better". It's about getting support into the mainline kernel so it "just works" for users. When I started this, I started working on the V4L driver because of some controversial licensing on the mrec driver and I wanted the support in the mainline kernel. Now that the mrec driver is under the GPL as well, why would we not want a vendor supported driver in the mainline kernel? It's likely to be of higher quality in a number of ways since they have a developer working full time on it and because the driver's based off of the datasheets instead of being reverse engineered.

I took a close look at the mrec driver tonight, and it's pretty good. So I asked myself, why wouldn't this be accepted in the V4L project (and the Linux kernel)?

So this doesn't sound like I'm bashing the driver, here is both a list of the good and the bad thing I see about the driver:

=== Good ===

• Significantly better device support - Markus has access to the actual hardware for many of these devices, spends time adding support for new devices, and since he works for the chipset vendor can in many cases just call the manufacturer of the product and ask them questions.
  
• Proper tuner locking - tuner locking was one of the big issues that caused infighting before Markus forked off his code. Let's face it - it's been over a year and most of the other devices don't do any locking at all. His scheme, while not unified across drivers, is better than nothing.
  
• Based on the chipset datasheets - He had the benefit of just being able to look up what the registers mean
  
• VBI support for analog - a recent addition in the mrec driver, but none at all in the V4L driver
  
• Support for other demods not currently in V4L - I don't think we have any devices yet that use the LGDT3304, but Markus's driver has support for devices that do.
  
• More thoroughly debugged - He's working on this full time. He's working bugs, dealing with issues, and putting in proper fixes based on reliable information instead of reverse engineering.
  

=== Bad ===

• Doesn't leverage common infrastructure such as videobuf (resulting in duplicate functionality and more difficult for those who have to maintain multiple drivers)
  
• Firmware blobs embedded in source - While it's easier for the user, many distributions do not allow firmware blobs in the kernel due to the belief that this is not GPL compatible. We would need to get permission from the vendor to redistribute the firmware as a file (in the V4L driver, we extract it from the Windows driver binary)
  
• Ambigious licensing - some of the files have headers from companies other than Empiatech which are very clearly not GPL compatible (like the Micronas drx3973d driver). Also, it's not clear that even the firmware blobs mentioned above are authorized to be redistributed by their rightful owners (Xceive and Micronas). While Empiatech may be ok with making a GPL driver, these parties have not consented to having their intellectual property in the kernel (they may have consented but the header files say just the opposite).
  
• It has its own xc3028 and xc5000 tuner driver. I don't know whether his driver is better than the one in V4L. Presumably he has the datasheets for those parts, but on the other hand the V4L driver allows loading of the firmware externally. The V4L drivers are also used by devices beyond the em28xx and may have functionality required by other companies products.
  
• What I'll call "Black magic" - lots of arbitrary code without any explanation as to what it is doing or why. Why does the DVB init routine write 0x77 to register 0x12? What does that do? A combination of poor use of constants and commented code combined with a lack of access to the datasheets leaves this a mystery. You just have to "trust that it's doing the right thing because it works"
  
• He's the only one who has access to the datasheets, so there is limited opportunity for peer review. The community driver is based on reverse engineering, and we can pass around USB traces we collect to justify/explain design decisions. How do you question a design when the basis of answers is essentially "because the secret document that I can't show you says so"?
  

Most of the above issues are not insurmountable, if people on both sides could just swallow their pride and do what's best for the users.

I keep thinking about how I could be working on other things, rather than duplicating the efforts of someone else.

Grrr....

I guess I'll get back to my ATSC support for Kaffeine now. At least there I have both the source code AND the specifications.

Kaffeine ATSC Close Captioning working!

I finally got a few cycles to integrate my buffering code into the Xine CC decoder.


(Click on the screenshot to enlarge)

Looks like it's generally working, although there's a bug in the quicksort somewhere because it occasionally crashes. Also, I think occasionally one of the characters is rendered out of order, which could just be that I need to tune the size of the buffer pools.

I need to get EIA-608 scrolling text to work since many broadcasts require it, but this is still a milestone...

Yay!

Saturday...

In the coffee house in Philly getting some stuff done. I think maybe they got some new Internet connectivity because I was able to download code at 2.4 Mb/sec.

Sent some email to the xine-devel mailing list explaining my ATSC frame ordering problem. Hopefully they have some existing infrastructure for reordering the frames. If not, then I'll use the code I wrote on Thursday night. If I'm lucky there will be a response by the time I get back tomorrow night.

Exchanged some email with the co-op board about the upcoming FIOS deployment and the performance differentials between Ethernet and 802.11b/g.

Analyzed the wire trace a user provided for the "ATI TV Wonder HD 600" USB ATSC receiver. If it indeed just uses the tvp5150 decoder than making it work should be a 20 line change to the list of known devices in em28xx-cards.c.

Victoria and I are going over to one of her friend's house tonight. Will play some Scrabble; should be fun.

More progress...

Made some more progress on my ATSC closed captioning tonight. I broke out my old man's copy of "Fundamentals of Data Structures" from 1983 because I couldn't remember how to code a quicksort (all the memory is statically allocated to keep load down on the heap manager and minimize memory fragmentation). The experience reminded me of the days of 5am coding at the Rutgers computer lab in the basement of Hill Center (except I'm old now so 1:00am feels like 5am).

Successfully stubbed out the buffering code in a test harness, which queues up a certain time range before announcing the group to the CC decoder (modeled it after TCP sliding window). It seems to be working but I want to test it some more before I expose it to a 19Mb/second MPEG stream. I'll also want to tune the window size, which shouldn't be hard once I feed it some real data.

I'm out of energy and I'm going to Philly tomorrow after work, so I guess I'll find out if it actually works on Sunday night when I get back.

Geek

Add one to the list of things that might make people think I'm a geek:

* Looking at your bookshelf for a book on the Linux kernel and realizing that you own four.

On a separate note, the Senate passed a bill granting immunity to all the telecoms for participating in illegal wiretapping for the NSA. How quickly we forget that the reason in the late 1970's that Congress came up with the FISA court in the first place was to stop the executive branch from illegal surveillance (See Project SHAMROCK). If you're really interested in how FISA came about, pick up a copy of James Bamford's The Puzzle Palace in paperback. It's a really good read.

I've decided: ATSC Closed Captioning is a bitch

So I didn't get much development work done last night. I did however get Leopard installed on my MacBook, reformatted my old iBook with 10.3 so I could give it to Packy for Parrot development, bisected the v4l-dvb tree to isolate a bug, and watched that really stupid National Treasure movie.

Tonight I got back to working on the ATSC closed captioning. "In our last episode", I had gotten Kaffeine to render the caption data, but the characters were not shown in the correct order. It turns up this is because the characters are arriving in transport order, and they need to be resequenced into picture frame order before they can be announced to the CC decoder (since the render information for a packet is dependent on previous events).

So now I'm going to need some sort of ordered linked list, where I insert packets as they arrive and periodically flush the list into the decoder if the render event with the lowest PTS matches the PTS currently being viewed. I will also have to keep track of when I last flushed the buffer so I don't insert events into the list that have already been rendered in the previous frame.

Oh yeah, and it has to share code with the existing DVD CC rendering, so I have to work within the constraints of the existing code and not break anything.

Kaffeine ATSC Closed Captioning Working... Almost

I made changes to Xine to start the libsupcc plugin and parse the EIA-608 closed caption info, and it's showing up in Kaffeine.

The only problem I'm having now is that because the frames can arrive out of order, the closed caption info is garbled (it's rendering the right characters but in the wrong sequencing). For example, "Hello world." is being rendered as "eHlloo Wlr.d"

I need to write some sort of sequencer to handle this. It might also have something to do with field1/field2 correlation. But it's certainly progress to see stuff show up on the display and it not be complete garbage.

More on ATSC closed captioning under Kaffeine

I got the Xine built from source now, and did some more debugging trying to figure out how the closed captioning support works. There is code for EIA-608 (used for DVD support), but there doesn't appear to be any way to load the plugin externally - it only kicks in if Xine detects closed captioning present in the stream (and the means of detection varies by standard).

Unfortunately, I don't think I can force it on/off, so I'm going to have to patch Xine to detect the info.

After some digging around through the spec and the MythTV source code, I figured out that the CC data is in the user_data section of the VOB. It's covered in ATSC a/53 Part 4 Section 6.2.3.1. For now I would be happy even if I just got the older style EIA-608 working.

I'm still not sure though how I'm going to correlate the CC stream locale information with the content contained in the PSIP EIT-0 entry (so the user knows which languages the various CC streams are in). I think the DVD code in Xine has some sort of convention for defining a callback, but I don't know how to use it yet.

At least in the short term, if I can get the content decoded I can just refer to them as "CC1,CC2,CC3,CC4", which is what Elgato EyeTV2 does (I don't have EyeTV3 so I can't say if they have improved it).

ATSC closed captioning under Kaffeine

Spent a good portion of last night reading about the various specs for closed captioning under ATSC (a/65, a/53, EIA-608, EIA-708). Pretty interesting stuff.

There are actually two separate versions of CC info sent in digital television, the first being the old fashioned "line 21" closed captioning with a single font, limited to Latin characters, and little control over how it is rendered (originally used for analog NTSC). The newer version of the protocol, which is targeted at digital, is much more advanced in that it supports eight fonts, different font colors/backgrounds, UNICODE characters, and control over where it is rendered on screen.

It's an area that I still needed to get working under Kaffeine.

Unfortunately both EIA-608 and EIA-708 are specifications that have to be paid for (a little over $100 each). It's annoying when a FCC mandated standard isn't freely available.

After digging into the code, it turns up though that Kaffeine relies on Xine to do the subtitle parsing and rendering (Kaffeine just lets you pick whether to turn it on/off and which stream to view). This is good in the sense that any work I do will benefit other applications that use libxine, but it's bad because it's another project I have to figure out how to build from source, and it appears I have already broken my Ubuntu box trying to compile it.

On the upside, there is already partial support for both EIA-608 and EIA-708 in libxine because it's used for DVD playback. So in theory I just have to get xine to detect the presence of CC from the PMT block. Once I do this, Kaffeine should "just work" without any changes. If this works out, I won't have to buy the EIA specs at all.

Why the hell did the ATSC bury the CC data in the video stream, instead of separating it out into another MPEG program stream like DVB does?

On a sidenote, debugging Kaffeine is a bitch:

• I don't know the ins/outs of libtool so I had to "make install" to get debug libraries
  
• I ran around in circles for half an hour only to realize my debug libraries weren't being used. I was running the kaffeine binary out of my working directory, but it was using the libraries that shipped with Ubuntu
  
• Kaffeine forks at startup so I have to run it in one window and then attach to the running pid ("--sync" doesn't appear to work)
  
• KDE Debug (kdbg) appears to be crap.
  
• I need a new office chair for my desk because my back is bothering me from using what used to be one of Gran's kitchen chair's.
  

Perhaps David Banner needs anger management therapy?

I think perhaps David Banner should stop focusing on exotic genetic or radiation treatments and get some anger management counseling.

Sure, I can appreciate him turning into the Hulk under some circumstances. Like in the pouring rain with a flat tire, he injures himself when his tire iron slips and then he turns into the Hulk. I can get that.

But last night I watched an episode where he turned into the Hulk because he didn't have enough change for a payphone:

1. David runs up to old rotary payphone and dials "0" for Operator. No answer.
   
2. Calls information and the operator gets the number for police department, but the person can't connect the call for him.
   
3. Dials police department. "Please insert another 35 cents."
   
4. "I DON'T HAVE 35 CENTS!!!!" Eyes turn white. Shirt starts to tear.
   

Yeah. I think maybe one season is enough. I think I'm going to cancel season 2 on my Netflix.

While we're on the subject, I can think of times when turning into the Hulk could be a good thing. Like this morning when some asshole in a Cadillac SUV blew through a red light while talking on his cellphone and nearly ran me over. Why couldn't I have turned into the Hulk then? I'd squish his car like a tin can!

Macgyver Gets Lazy

From xkcd: http://xkcd.com/444/