Oopsie

Last week at work I had, what in industry terms we might refer to as, a "big fuckup".  I can't go into the technical details but I will say that it was high-profile and customer-facing.

But before we talk about that, let's pause for a brief anecdote my uncle shared with me.

My Uncle Richie is what I would refer to as an "expert amateur woodworker".  He's been doing woodworking for over thirty years, done countless projects both large and small, and has a well-equipped shop with tools he is highly familiar with.  He's about as good at the craft as you can get without being a professional who does it for a living.

A few years back he was building some new steps for the back porch.  During the final assembly he was putting the various pieces in place and unexpectedly realized that one of the boards was a bit too long.  With the wood glue already drying he ran back into the garage, flipped on the table saw, and cut the board.

Bzzzt.  While cross-cutting the board he also managed to pass his hand over the saw and ran the blade across the palm of his hand.  Whoops. *

The point I'm trying to make is that even an expert can have moments where he/she/they becomes complacent (especially when in a hurry).  And even when doing something you're good at sometimes you still have that "oh crap" moment.

None of this is to offer an excuse or rationalization for what happened with the software I am responsible for.  The post-mortem I had to do revealed that there were five or six improvements I can make to increase the robustness of the program and avoid the issue in the future.  And of course there are improvements to the QA process that can help catch such issues before the code makes it into production.

But while I wanted to beat myself up over what was a small bug in the code that had big implications, sometimes I have to remind myself that sometimes, "shit happens".  Learn from it and move on.

* He's fine, by the way.  He needed a trip to the hospital and a bunch of stitches, but no permanent damage.

A short little rant on WordleBot

I've been playing Wordle since early January, and it's one of my favorite games (behind Spelling Bee).  I've spent alot of time thinking through various strategies, and in fact wrote my own little solver many months ago.

Last month the New York times introduced "WordleBot".  Available only to NY Times subscribers, it's a neat little tool which looks at your most recent game and offers some feedback on how you did, and how you could have done better.

New York Times WordleBot

It's fun to see how their solver attacked the problem, compare your approach to it, and see a bunch of underlying statistics on possible approaches.  That said, it's got one little property to it that really annoys me.

To illustrate my point, let's look an example I captured a couple of weeks ago:

Now in the above example, I actually out of dumb luck arrived at the same second word that the Bot would have chosen.  What caught my attention though was that it said there were only five possible solutions.  However, there are actually quite a few more possibilities:

Understanding why my tool said there were 46 possible solutions after the second word and why WordleBot claimed there were only five requires us to delve a bit into the definition of "solution".

There are actually two lists of words that come into play here.  The first is the list of possible five-letter words in the English language dictionary.  The Wordle dictionary has 12947 words, but that assumes you have extracted the list from the the source code to the app.  When I wrote my original solver I actually used the standard dictionary that ships on most Linux systems, which has 10230 five-letter words.  But while that's the list of possible words, many of those are far too obscure to ever guess with only six chances (people would be annoyed if the answer to today's puzzle was something like "amahs").  As a result, the author asked his partner to go through the list of words and pick out all the words she recognized.  This resulted in a much smaller list of possible solutions (about 2500 in total).

The key here is that when Wordle says there are only "five possible solutions", it's taking advantage of the fact that it knows this much smaller list.  So when it says things like, "there's only one solution left so you should get it on the next turn" it's making an unrealistic assumption that the user also knows which of the 12947 words make up the list of possible solutions.  This just feels unfair.

My suggestion is that the WordleBot would be on a more level playing field if it used the complete list of 5-letter words when narrowing down the list.

End rant.

Privilege, Nepotism, and Luck

As we come to the end of 2021, I've been giving a good bit of thought to my career development and how I started out.  Part of this is my pondering where I want to go next, and part is the byproduct of a conversation I had with someone I worked with more than twenty years ago.

A couple of fun facts about me which people probably don't know:

• I got my first job in high school doing bookkeeping for a business owned by my grandfather and uncle.
• I got my first "real job" in IT while in college because I happened to be dating the boss's daughter.
• I got my second real job because my father worked at the company in another department.

Now, none of this is to suggest I didn't do good work at any of the companies.  Looking back I think I did "better than average" for my age and level of experience at the time.  I worked long hours and made a considerable effort to prove my worth and exceed expectations.  And the reputation I established and the relationships I formed set me up for future jobs.

But none of this changes the fact that my first three jobs were acquired as a result of nepotism.  I likely wouldn't have gotten through the door at any of them without the personal relationship.  It's also fair to suggest that in my early years of employment I was a much more awkward and prickly person on an interpersonal level, and the personal relationships I had at those jobs might have spared me from some difficult conversations that my managers might have had with me otherwise.

It's all too common that successful people talk about how much they were able to achieve through hard work and determination.  It's also all too common that they neglect to mention other factors that they can't really take credit for.  Sure, I worked hard and was determined, but I was also lucky, had people who helped me, and always had a safety net in a supportive family (e.g. I could take risks knowing that if all else failed then I could always move back home with my father and live rent-free for a few months to get back on my feet).

It also didn't hurt being a white, straight, cisgender male.

Reflecting On My Recent Accident

The day after my fall I posted on Instagram a photo of me sitting in a wheelchair outside the skating rink, smiling with a bag of ice on my ankle.  This paints a relatively simple picture suggesting I just had an "oops" moment and now I needed to get my leg fixed up.  But while I was feeling ok at that moment, it doesn't really capture what I really felt only a few minutes prior.

Pain.  Searing, blinding pain.  Pain like I have never felt in my life.  The feeling that you're going to pass out from the pain it hurts so badly.  I remember laying on the ice with my skate folded underneath my other leg, and I couldn't even see straight.  The panic that CC might have been hurt too since I was holding her hand when we fell.  My heart was racing and my forehead was covered in sweat, despite just laying there on the freezing cold rink.  I remember Jeremy telling the referees that he's a physician (thank God for that), and I remember his voice as he talked through several attempts in straightening my leg and removing the skate from my foot.  I don't remember how they got me up off my back and into the wheelchair, but I am pretty sure I didn't help in the process.

And then I felt ok.  I guess it's all relative, but at that point I was feeling better and figured I just had a sprained ankle and I had been a baby.  I called my spouse from the locker area and informed her of "the dumb thing I did".

It wasn't just a sprain though, which became apparent after they did the x-rays and concluded I needed to be transferred to another hospital that was "better equipped to handle this level of trauma".

I spent five days in the hospital.  I should have been out of there in three, but on the third day I had an episode of syncope during my PT session.  Thus I had to wait until they got my pain meds right before I could be safely discharged.

I can feel myself starting to sweat whenever I think about those moments.

It's hard not to feel like it's just yet another reason for me to be depressed about 2021.  Like I didn't already have enough weighing me down.

I'm not writing any of the above to solicit sympathy.  Mostly it's just me trying to work through what I suspect is some mild PTSD.

Instagram Image Quality

At my day job I spend all day capturing live video, compressing it, distributing it, and showing it to users.  Hence when people ask questions like, "Why do my Instagram photos look crappy?", it tends to pique my interest.  In this post I'll briefly summarize the basic workflow and offer some guidance on how to debug such issues.

There are a variety of moving parts associated with contributing image content to a site like Instagram.  The following diagram shows the basic workflow.  While not specific to Instagram's specific internal process, all these sites basically work the same and have the same components.

(click to enlarge)

The important consideration here is that there are two largely disconnected processes:  the process by which the content gets into Instagram, and the process that users viewing the content go through.  Problems introduced as part of the contribution process will likely impact all viewers, but there are cases where the contribution phase may have worked perfectly but different viewers see different results due to the way content is delivered to the user.

Let's quickly go through the different stages and then we can talk about how to break down the problem.

Acquisition - This essentially encompasses taking the initial picture.  This is a combination of the content itself and the camera settings used.

Question:  Why might close-up photos of an object look ok but a picture of me standing in grass in front of some trees look crappy?  Answer:  image complexity.  Bear in mind that the encoder typically has a target image size so it has to make decisions about which parts of the picture to make look best while still creating an image that is N megabytes in size.  An object in front of a solid color background is comparatively simple to encode and the compression process emphasizes ensuring the object looks good.  When you take a picture in front of a complex background the compression engine may not know what areas to optimize, and thus the encoder will dedicate more bits in the final image to making the grass look less blurry at the cost of your face looking worse.  This is why it's often difficult to take good video of soccer games or other games played on a large field.  What the viewer cares about is the players and the ball (sometimes referred to as the "Region of Interest") but the compression engine doesn't know that and thus it spends as many bits encoding the entire field at the cost of quality of the individual players.

Ingest - This is the process where you take the picture from the camera and send it to Instagram.  There is processing that can happen within the phone before it gets sent out.  The image from the camera may be decoded and re-encoded to allow cropping (either as a result of user defined cropping, or to meet an expected aspect ratio), change the resolution (e.g. Instagram has a maximum width of 1080 pixels, so anything larger than that needs be rescaled before uploading), or reduce the size of the encoded picture (e.g. change the 5MB photo into a 1MB photo at 1080x1080 prior to upload to reduce bandwidth usage).  Aside from allowing the user to crop the photo, Instagram doesn't expose many user-configurable options to control the upload process, but decisions are being made behind-the-scenes which the user won't be aware of, and this behavior may change whenever the Instagram application is updated.

Server Side Processing - this is what the website does to the photo after it's uploaded from the Instagram application running on the phone.  The main task being performed is creating multiple renditions of the image at different resolutions and bitrates.  While the user is uploading a high quality photo, it's entirely likely that some users would be better suited to receive lower quality renditions, and the rescaling/re-encoding that occurs here is how that is accomplished.  Doing it at the server also allows for the user to only have to upload a single high quality version of the picture which helps from a bandwidth perspective.  That said, it's possible that bugs in this process may result in cases where a particular rendition of an image has problems (e.g. the 1080x1080 version looks fine, but the 640x640 version look poor).

Viewing - Unlike a desktop browser, a mobile application like the Instagram app has much more complex decision making when determining which rendition of an image to download and show to the user.  In order to provide an optimal viewing experience for all users, the app may automatically do things like downloading a low quality version of an image to minimize cellular data usage.  If there is a problem with this decision making though the user may end up with poor quality images (i.e. they were connected over Wifi and had plenty of bandwidth, but are being sent the 320x320 version of the image).

A note about "Pinch to Zoom":  When you zoom in on an image in the application, you're not really seeing a higher quality version of the image.  You're just zooming in on whatever image was already downloaded.  When the image you're zooming in on has a higher resolution than the display, this results in being able to see more detail.  However if you're zooming in on a very low resolution image, you'll see artifacts like blurring or jagged lines.  It's also important to note that zooming in on the original image prior to uploading is a very different experience than when viewing it after upload.  This is because in the former case you're zooming in on the original, highest quality image, while in the latter you're zooming in on whatever the website delivered which may already be of reduced quality.

"Ok, so now you've spent a bunch of paragraphs explaining how all this works in principle.  How do I actually figure out what's broken?"

Having explained the workflow, you can break up the problem to understand which stage is causing the issue.  This is especially true if you know definitively that the image quality was good at one point but has since gotten worse with new content.

For the purpose of offering an example, let's say you know definitively that image quality was fine in April, but now all the images you upload look crummy.  Here are a few concrete steps you can perform to narrow down the issue:

1. Go back to posts from April and make sure the image quality still looks good in the app (i.e. pinch-to-zoom and ensure proper detail is shown).  If those posts from April look good, then you've likely just cut the problem in half and confirmed that it's not an issue with the viewing experience and the problem is some aspect of the authoring half of the workflow.
2. Assuming Step 1 looked good, pick a specific photo from a post from April, and go back into your Instagram app and find that photo in your camera reel.  Upload that exact same photo again in a new post.  This cuts the problem in half again:  If the newly uploaded photo looks good, then you've got some problem with the camera app and how new photos are being taken (e.g. camera settings changed).  If the newly uploaded photo looks bad, then you know it's got something to do with how the photo is being processed by the Instagram app and/or the Instagram website.
3. If step 2 looked good then you need to figure out why the camera app is taking pictures that no longer are properly processed by Instagram.  Set up a tripod and be prepared to take a series of images of the same object but with different camera settings (putting a small post-it pad in the corner with a number will allow you to keep track of the pictures without impacting the image quality).  Create a spreadsheet to keep track of exactly how each possible setting in the camera application was configured for each image.  Upload the resulting images in a post and compare the quality to see which settings have an impact.  Hopefully you'll create 20 pictures and come to a conclusion like, "When the HDR+ option is enabled, the images look like crap" or "When the aspect ratio is changed from 4:3 to 16:9 the problem happens".
4. If step 2 looked bad, then photos that are known to have worked previously now don't work and something has changed either within Instagram's infrastructure, the Instagram app, or the app settings.  There are fewer dials available to the user, but I would suggest uploading the same photo multiple times with different settings for the "data saver mode", "high resolution photos", cropping settings, making sure you're on Wifi and not cellular during upload, etc.  It may also be worthwhile to completely uninstall the app after doing a "clear data" and "clear cache" in the Android Application Preferences screen.  This will wipe out all settings/configuration, in case there is some subtle or undocumented setting which is causing the issue (e.g. something which didn't get properly handled when upgrading from one version to the next).

I hope this offers some insight that will help in diagnosing such problems.

First Rant

Well now that I'm off LiveJournal, I can now proceed to complain about what a PITA that was.

Ten years ago there were tools written by a number of people to export LiveJournal content and import it to other platforms (WordPress, Blogger, etc).  Those tools relied on a relatively good API that LiveJournal provided, which has long since been removed.  Now the only functionality they offer is the ability to export posts one-month-at-a-time.

I spent a few hours one night trying out various tools from GitHub, Google Code, etc, and none of them worked.  So on a different night I broke down and started looking at the XML that LiveJournal made available, and hacked together a Perl script to download it all (750 posts in total).

Now I'm a software developer by trade, but the fact of the matter is I don't do much scripting nowadays.  I'm largely neck deep in video codecs, multi-threaded programming, and other low-level really technical complicated things.  A Perl script to scrape a website, download all the images, replace all image references in the XML, and generate an XML blob that can be imported into Blogger is the sort of task I would normally hand to a junior programmer as well as being something that I generally suck at.

So it took two evenings.

That said, it's done, and it didn't turn out half bad.  I suspect there are still a couple of broken images that reference the old blog, but I'll nail that down at some point.  If you wrote one of the 638 comments that were left on the old blog between 2005 and 2013, sorry friend, they ain't getting ported over.

First Kiss

Part of CC's bedtime ritual is I read her some stories, and then we go to bed. Typically this involves some combination of her cuddling with her bunny rabbit and/or Cookie Monster, and she falls asleep in my arms (after which I transplant her to her crib).

Tonight was a bit different... Sitting in the dark with her in my arms, she had her Yellow Bunny and Cookie Monster pretend to kiss each other. She gave a kiss to each of them, then held them up for me to kiss, and then she arched her head back, puckered her lips, and kissed me.

Daddy's heart melts.