Thursday, June 3, 2021

Instagram Image Quality

At my day job I spend all day capturing live video, compressing it, distributing it, and showing it to users.  Hence when people ask questions like, "Why do my Instagram photos look crappy?", it tends to pique my interest.  In this post I'll briefly summarize the basic workflow and offer some guidance on how to debug such issues.

There are a variety of moving parts associated with contributing image content to a site like Instagram.  The following diagram shows the basic workflow.  While not specific to Instagram's specific internal process, all these sites basically work the same and have the same components.

(click to enlarge)

The important consideration here is that there are two largely disconnected processes:  the process by which the content gets into Instagram, and the process that users viewing the content go through.  Problems introduced as part of the contribution process will likely impact all viewers, but there are cases where the contribution phase may have worked perfectly but different viewers see different results due to the way content is delivered to the user.

Let's quickly go through the different stages and then we can talk about how to break down the problem.

Acquisition - This essentially encompasses taking the initial picture.  This is a combination of the content itself and the camera settings used.

Question:  Why might close-up photos of an object look ok but a picture of me standing in grass in front of some trees look crappy?  Answer:  image complexity.  Bear in mind that the encoder typically has a target image size so it has to make decisions about which parts of the picture to make look best while still creating an image that is N megabytes in size.  An object in front of a solid color background is comparatively simple to encode and the compression process emphasizes ensuring the object looks good.  When you take a picture in front of a complex background the compression engine may not know what areas to optimize, and thus the encoder will dedicate more bits in the final image to making the grass look less blurry at the cost of your face looking worse.  This is why it's often difficult to take good video of soccer games or other games played on a large field.  What the viewer cares about is the players and the ball (sometimes referred to as the "Region of Interest") but the compression engine doesn't know that and thus it spends as many bits encoding the entire field at the cost of quality of the individual players.

Ingest - This is the process where you take the picture from the camera and send it to Instagram.  There is processing that can happen within the phone before it gets sent out.  The image from the camera may be decoded and re-encoded to allow cropping (either as a result of user defined cropping, or to meet an expected aspect ratio), change the resolution (e.g. Instagram has a maximum width of 1080 pixels, so anything larger than that needs be rescaled before uploading), or reduce the size of the encoded picture (e.g. change the 5MB photo into a 1MB photo at 1080x1080 prior to upload to reduce bandwidth usage).  Aside from allowing the user to crop the photo, Instagram doesn't expose many user-configurable options to control the upload process, but decisions are being made behind-the-scenes which the user won't be aware of, and this behavior may change whenever the Instagram application is updated.

Server Side Processing - this is what the website does to the photo after it's uploaded from the Instagram application running on the phone.  The main task being performed is creating multiple renditions of the image at different resolutions and bitrates.  While the user is uploading a high quality photo, it's entirely likely that some users would be better suited to receive lower quality renditions, and the rescaling/re-encoding that occurs here is how that is accomplished.  Doing it at the server also allows for the user to only have to upload a single high quality version of the picture which helps from a bandwidth perspective.  That said, it's possible that bugs in this process may result in cases where a particular rendition of an image has problems (e.g. the 1080x1080 version looks fine, but the 640x640 version look poor).

Viewing - Unlike a desktop browser, a mobile application like the Instagram app has much more complex decision making when determining which rendition of an image to download and show to the user.  In order to provide an optimal viewing experience for all users, the app may automatically do things like downloading a low quality version of an image to minimize cellular data usage.  If there is a problem with this decision making though the user may end up with poor quality images (i.e. they were connected over Wifi and had plenty of bandwidth, but are being sent the 320x320 version of the image).

A note about "Pinch to Zoom":  When you zoom in on an image in the application, you're not really seeing a higher quality version of the image.  You're just zooming in on whatever image was already downloaded.  When the image you're zooming in on has a higher resolution than the display, this results in being able to see more detail.  However if you're zooming in on a very low resolution image, you'll see artifacts like blurring or jagged lines.  It's also important to note that zooming in on the original image prior to uploading is a very different experience than when viewing it after upload.  This is because in the former case you're zooming in on the original, highest quality image, while in the latter you're zooming in on whatever the website delivered which may already be of reduced quality.

"Ok, so now you've spent a bunch of paragraphs explaining how all this works in principle.  How do I actually figure out what's broken?"

Having explained the workflow, you can break up the problem to understand which stage is causing the issue.  This is especially true if you know definitively that the image quality was good at one point but has since gotten worse with new content.

For the purpose of offering an example, let's say you know definitively that image quality was fine in April, but now all the images you upload look crummy.  Here are a few concrete steps you can perform to narrow down the issue:

  1. Go back to posts from April and make sure the image quality still looks good in the app (i.e. pinch-to-zoom and ensure proper detail is shown).  If those posts from April look good, then you've likely just cut the problem in half and confirmed that it's not an issue with the viewing experience and the problem is some aspect of the authoring half of the workflow.
  2. Assuming Step 1 looked good, pick a specific photo from a post from April, and go back into your Instagram app and find that photo in your camera reel.  Upload that exact same photo again in a new post.  This cuts the problem in half again:  If the newly uploaded photo looks good, then you've got some problem with the camera app and how new photos are being taken (e.g. camera settings changed).  If the newly uploaded photo looks bad, then you know it's got something to do with how the photo is being processed by the Instagram app and/or the Instagram website.
  3. If step 2 looked good then you need to figure out why the camera app is taking pictures that no longer are properly processed by Instagram.  Set up a tripod and be prepared to take a series of images of the same object but with different camera settings (putting a small post-it pad in the corner with a number will allow you to keep track of the pictures without impacting the image quality).  Create a spreadsheet to keep track of exactly how each possible setting in the camera application was configured for each image.  Upload the resulting images in a post and compare the quality to see which settings have an impact.  Hopefully you'll create 20 pictures and come to a conclusion like, "When the HDR+ option is enabled, the images look like crap" or "When the aspect ratio is changed from 4:3 to 16:9 the problem happens".
  4. If step 2 looked bad, then photos that are known to have worked previously now don't work and something has changed either within Instagram's infrastructure, the Instagram app, or the app settings.  There are fewer dials available to the user, but I would suggest uploading the same photo multiple times with different settings for the "data saver mode", "high resolution photos", cropping settings, making sure you're on Wifi and not cellular during upload, etc.  It may also be worthwhile to completely uninstall the app after doing a "clear data" and "clear cache" in the Android Application Preferences screen.  This will wipe out all settings/configuration, in case there is some subtle or undocumented setting which is causing the issue (e.g. something which didn't get properly handled when upgrading from one version to the next).
I hope this offers some insight that will help in diagnosing such problems.

Friday, July 24, 2020

First Rant

Well now that I'm off LiveJournal, I can now proceed to complain about what a PITA that was.

Ten years ago there were tools written by a number of people to export LiveJournal content and import it to other platforms (WordPress, Blogger, etc).  Those tools relied on a relatively good API that LiveJournal provided, which has long since been removed.  Now the only functionality they offer is the ability to export posts one-month-at-a-time.

I spent a few hours one night trying out various tools from GitHub, Google Code, etc, and none of them worked.  So on a different night I broke down and started looking at the XML that LiveJournal made available, and hacked together a Perl script to download it all (750 posts in total).



Now I'm a software developer by trade, but the fact of the matter is I don't do much scripting nowadays.  I'm largely neck deep in video codecs, multi-threaded programming, and other low-level really technical complicated things.  A Perl script to scrape a website, download all the images, replace all image references in the XML, and generate an XML blob that can be imported into Blogger is the sort of task I would normally hand to a junior programmer as well as being something that I generally suck at.


So it took two evenings.

That said, it's done, and it didn't turn out half bad.  I suspect there are still a couple of broken images that reference the old blog, but I'll nail that down at some point.  If you wrote one of the 638 comments that were left on the old blog between 2005 and 2013, sorry friend, they ain't getting ported over.

Monday, August 11, 2014

First Kiss

Part of CC's bedtime ritual is I read her some stories, and then we go to bed. Typically this involves some combination of her cuddling with her bunny rabbit and/or Cookie Monster, and she falls asleep in my arms (after which I transplant her to her crib).

Tonight was a bit different... Sitting in the dark with her in my arms, she had her Yellow Bunny and Cookie Monster pretend to kiss each other. She gave a kiss to each of them, then held them up for me to kiss, and then she arched her head back, puckered her lips, and kissed me.

Daddy's heart melts.

Sunday, April 27, 2014

The role of parents

I've been spending a lot of time lately thinking about what sort of parent I am going to be. Some of this is a result of a conversation I had with Victoria last week as we compared/contrasted the parenting style of our respective mothers. What qualities of our parents do we want to emulate in how we raise our own child -- and what situations are best served as an example of what not to do?

And the lessons don't stop coming on your 18th birthday. My father passed away when I was 23. At the time and for some years on I had not really appreciated the notion that there were still things he could teach me (after all, 23 year-olds know everything, right?). However after having CC I often found myself asking, "How did my father handle situation X when he was in my shoes?" In many cases I can just think back to an event where such happened to me and recall how he handled it. But there have been more than a few cases where I was too young to remember, or the answers are more about his thought process rather than his visible actions. I can't help but regret the inability to ask him.

Tuesday, October 29, 2013

Stuff and things and Stuff

Wow, time flies.

We moved to the Upper East Side last month. Still trying to get everything unpacked although making progress. Nice area, literally a block from Gracie Mansion. With the baby it's really nice having an apartment that actually has doors. I can now watch TV again without the closed captions!

Been working from the home office for the last couple of weeks since my business partner fell ill and I had to take over one of his projects. There just isn't enough space in my cubicle for the equipment required to juggle three projects at once.

Overall things have been very busy but good.

C.C. turned seven months a couple of weeks ago. She eats solid food now, is almost crawling, and has a much more evident personality. She now also sleeps through most of the night (until about 5am), which is a godsend after several months of colic followed by several more months of waking up every couple of hours.

More CC pictures on the Facebook album:
https://www.facebook.com/media/set/?set=a.4951733630027.1073741825.1199470342&type=1&l=84b4c09299

I've got a huge collection of photos in my camera, but I need to take some time and organize them all. All of the photos above are just from my phone. In the last few weeks we went apple picking in New Jersey, took CC to a community pumpkin picking in the nearby park, and last weekend spent Saturday at the Pumpkin Fest in Central Park. Tomorrow we're heading to New Jersey so CC can do her first trick-or-treating with Jayden and Lauren's kids.

TTFN.

Monday, July 22, 2013

A brief update on life and things

Baby C.C. turned four months last week. She's getting so big. It's really quite amazing to look at pictures from April versus pictures from this week. Note to self: I've got a bunch of photos on the camera Victoria bought me for my birthday -- need to get them downloaded and posted on Facebook.

Victoria received her New York license to practice today. I'm so proud of her. It's the end of a very long road that started nearly seven years ago. We started dating shortly before she went away to grad school in Philadelphia.

Business is good. Busy, but good. Any Linux driver gurus out there looking to do some consulting? Let me know.

Suzy got married in Allaire State Park on Friday, so we took a three day weekend down at "the shore". I'm looking at these photos of me in a white tee shirt holding CC by the pool. Scary how much they remind me of similar photos of my father. It has been thirteen years this week since he died - so a bit of introspection on my part for the last few days.

More later...

Sunday, April 7, 2013

Baby Cynthia Catherine

Figured it would be worth putting up some photos for my friends without Facebook to enjoy:
baby cynthia

All the other photos can be found here:
https://www.facebook.com/media/set/?set=a.4951733630027.1073741825.1199470342&type=1&l=84b4c09299

I've been effectively on paternity leave for the last month. Lots of sleepless nights, diaper changes. The baby has needed to eat every three hours (including waking her if she's asleep).

Going back to work full time on the 15th. Been doing enough to keep things moving, but it's been a challenge.