Debugging a Persistence Bug in a Mongo/Rails App

This blogpost originally appeared on the Substantial blog in 2014. I forgot to cross-post then, so I’m doing it now.

TL;DR Using the $push operator in mongo fails silently on null fields. Turn on safe mode in your mongoid.yml.

The problem

One of our Rails/mongo projects recently had a bug where one of the array data fields we were appending to was not persisting properly. The app uses Rails 3.2.x (one of our older projects) and Mongoid as an object document mapper. The following code snippet shows the has_many relationship between Fleets, Garages, Motorcycles and Automobiles. I’ve omitted the Motorcycle and Automobile classes here, but they have basically the same structure.

class Fleet
 include Mongoid::Document

 embeds_many :motorcycles

class Garage
 include Mongoid::Document

 embeds_many :automobiles

We’d recently converted a bunch of fleets of motorcycles to automobiles in garages (they needed some mechanical work done). The script looked like this:

fleets = Mongoid.default_session[:fleets]

fleets.each do |fleet|
 id = fleet.delete_id

 garage = fleet.deep_dup
 garage['_id'] =
 garage['automobiles'] = fleet['motorcycles']
 garage.delete 'motorcycles'


The problem we were having was with the newly created Garages. We were able to remove an automobile from a garage fine, but if there was a garage with no automobiles that was created from a fleet with no motorcycles, appending a new automobile to the garage wouldn’t save properly.

Writing a test script

Ordinarily, I would write a unit test to create a repeatable test scenario so as to automate the testing process. In this case, I wasn’t sure where in the code the problem was, so this was not an option. Manually adding and removing entries in the web app directly got tiresome pretty quickly, so I whipped up a quick script to run after every change. Every time we reloaded the model, g.automobiles would be an empty array again, no matter how many automobiles we appended to the Garage.

g = Garage.find('5165ea04abc5c2b85f00004a')
puts g.automobiles.inspect

g.automobiles << '2009 Ninja 500R')
puts g.automobiles.inspect!
puts g.automobiles.inspect

Going through mongoid logs

A coworker suggested going through the mongo logs to see if the app was sending the correct updates to the database. Mongoid allows you to set two variables within Rails’s application.rb to set the log level to DEBUG. See here for more details on Mongoid and logging.

module MyApplication
 class Application < Rails::Application
   Mongoid.logger.level = Logger::DEBUG
   Moped.logger.level = Logger::DEBUG

After tracing through each database query, I could see that the app was indeed telling mongo to $push a new Automobile document to the Garage’s :automobiles field. Finally, after googling “mongo array push not persisting”, I found the issue. In mongo’s documentation on the $pushoperator there is a note that says:

  • If the field is absent in the document to update, $push adds the array field with the value as its element.
  • If the field is not an array, the operation will fail.

When I pulled up some of the problematic Garage records to see what the :automobiles field looked like, sure enough, they were nil instead of unset like their corresponding Fleet records. I dug back into the earlier script to see why there were a bunch of nils hanging out in empty Garages and discovered the problematic line:

garage['automobiles'] = fleet['motorcycles']

If a fleet had no motorcycles in it to begin with, the script set garage['automobiles'] to nil, which caused any mongo $push operations to that field to fail silently.

We’ve now set safe: true in mongoid.yml as per the Mongoid’s documentation on safe mode, so these types of exceptions will be caught faster in the future. That being said, the docs do mention that mongo sometimes logs an error on the server, but sends a message back to the client that the operation was successful, so I’ll be watching the mongo logs more closely in the future.

Accordion Strap

So I was playing a gig with How Short last weekend and at the beginning of the second set, my right shoulder strap completely broke off. The strap itself was fine, but the screws that hold the strap in straight up tore out of the wood. I tried screwing the tiny screws back in, but they weren’t holding, so I ended up playing out the rest of the gig with a broken right shoulder strap. For all you non-accordionist readers out there (like all of you, probably) the right shoulder strap is what provides tension for me to pull the bellows, so playing without it was rather difficult. I figured out that I could sit on the strap to provide at least a little bit of tension, but it was still pretty tiring.

I look like I’m really into what I’m playing, but I’m really just trying to maintain tension while pulling on the bellows.

So the gig finished at 2pm and I had a party and jam to play that night, as well as a gig the next day. Luckily, Petosa (the world’s most awesome accordion shop) was open until 3. I hightailed it over there and they replaced the screws with much beefier ones. Subsequent jams and gigs went off without a hitch.

GroundUP Music Festival 2017


Last weekend, Rosanna and I flew to Miami for the first ever GroundUP Music Festival. I’d never traveled for music before, or for that matter been to a weekend long music festival, but it was a really fun experience. There was a tremendous amount of good music and it was fun to see some of my favorite artists live.

Aside from music, a number of things made the festival a really enjoyable experience. First, they capped the number of people at 1500, so I never felt like I was drowning in crowds. It felt like a very casual and intimate festival. Also, there were never two performances at the same time, so I never had to decide between multiple acts that I wanted to see. Lastly, the performers themselves attended the festival just like the rest of us, and we got to enjoy music together and talk to them too.

Of course the highlights were the shows themselves. John Medeski’s Mad Skillet was really fun with tasty Hammond organ lines supported by a funky sousaphone. Jacob Collier’s one-man show at the late night set was a phenomenal spectacle. He looped and played piano, upright bass, organ, drums, a bunch of percussion, as well as singing through a harmonizer pedal, running all around stage playing everything. Chris Thile and Michael Daves played a set too, singing sweet harmonies over breakneck bluegrass mandolin and guitar.

Snarky Puppy played all three days of the festival, each night better than the last. The band was impossibly tight and solos were beyond imaginative. Both the horns and rhythm section were in lockstep, not a note out of place. In particular, I really enjoyed watching Michael League direct the band while playing bass, cueing solos and when to move on in the form. There was no point at which it wasn’t crystal clear where the band was supposed to be, and they followed cues flawlessly. He’s also got the world’s best bass face, which was amazing to see in person. (Yes, he’s playing a baritone guitar in the picture below, with his other band Bokanté.)


The set Snarky Puppy played on the third night had special significance. It’d just been announced that they’d won their third Grammy and the band was visibly really excited about it. The audience whooped and cheered, while the band started to play Lingus, which the throngs of Snarky Puppy fans sang along with joyously.

In addition to the great performances, the festival included workshops led by the performers. Bill Laurance, one of the keyboardists in Snarky Puppy, gave a workshop on “Finding a Voice” in which he detailed his journey in producing record after record, trying to hone in on a voice that he felt was truly his. He insisted that persistence was one of the most important things in music. The anecdote that stood out to me most was that Snarky Puppy toured and recorded for seven(?) years before they started to see any financial success.

A bunch of percussionists and drummers from the various groups performing in the festival gave a little Q/A session as well. Larnell Lewis, one of the drummers for Snarky Puppy, talked about setting aside focused time for specific practice goals, like 15 minutes of metronome exercises or 20 minutes of learning tunes. He also talked about using mental practice, when a drum kit is not available, where he runs through rhythms and tunes in his head while on the road. Jamison Ross talked about how the multiple drummers in Snarky Puppy coordinated by listening to the other drummers and either matched rhythms to provide emphasis or filled space where the other drummer wasn’t playing. Larnell also mentioned that for certain pieces, they would trade off who played kick or snare as a way to not collide with each other. I’d always wondered how they coordinated so successfully.


Esperanza Spalding and Becca Stevens gave a workshop on their songwriting process, in which they talked about the importance of building a song around a story. They also noodled around with progressions and melodies on stage, building the first bits of a song. It was really encouraging to hear them talk about how songwriting is a slog for them as well, that they also don’t have a magic bullet that allows them to pump out endless material. Esperanza, like Bill, spoke of persisting and not giving up on a song even if it doesn’t sound good at first.

Snarky Puppy also ran a workshop with a band from a local music school. They had the band play a couple pieces and different members of Snarky Puppy gave feedback on how they could improve. Michael talked about the importance of synchronizing voicings between the guitars and keyboards on hits to get a much crisper, clean sound.

Michael League and David Crosby gave a workshop on “Chasing the Muse”, which was mostly a Q/A session on their songwriting process. They advocated for recording everything. Michael apparently keeps a collection of voice recordings of song snippets that he adds to whenever the idea strikes. Whenever he sits down to get some writing done, he listens through them until he finds one that inspires him.

All in all, the festival was a fantastic experience. Learning about the musicians’ writing and practicing habits in the workshops and hearing gorgeous music they create was endlessly inspiring. A lot of the information in the workshops wasn’t new to me, but it was somehow encouraging to hear that even brilliantly successful musicians go through the same practicing and writing grinds that I do. Time to make some music!

Switching graphics drivers


TL;DR Don’t be hasty when mucking around with graphics drivers

What caused problems

After updating to Ubuntu 16.04, steam was no longer starting. Some forum posts were suggesting that certain dependencies weren’t being linked correctly and suggested manually moving things around. That didn’t seem to do anything, so I tried updating the graphics driver. I was a little hasty, and being unsure of which driver was in use, I decided to apt-get remove a bunch of other drivers, including the nouveau one. This caused everything to break and lightdm wouldn’t start properly, which means I couldn’t log in anymore.

Fixing things

First order of business was to get a terminal, since lightdm hanging meant I couldn’t log in normally. Recovery mode was too locked down for me to get anything real done, and it would freeze when I tried things. I ended up editing the boot option and removing every additional argument past “ro” and replacing it with “text”. This allows you to boot into terminal mode.

Next, was connecting to wifi. Now, I think my system was automatically connecting anyway, but I figured it out at any rate. nmcli dev wifi list shows what are available. nmcli dev wifi connect “SSID” password “PASSWORD”

I ended up doing sudo apt-get purge xorg lightdm and reinstalling with sudo apt-get install xorg lightdm, which took FOREVER, but it did get me back to a graphical boot after restarting. I now think this was probably unnecessary. It ended up defaulting me to an older proprietary NVIDIA driver that steam was still interacting poorly with, but at least I could use the computer now!

Finally, I discovered through this post that I could run ubuntu-drivers devices which displays a list of available drivers, and sudo apt-get install whichever driver I wanted. Picking the latest proprietary NVIDIA driver finally let me run steam without issue.

Ubuntu and wifi

Occasionally when waking my laptop from sleep, wifi is disabled. It’s usually some kind of weird driver bug where the card refuses to reinitialize properly. In the past, I’ve just unloaded and reloaded the appropriate kernel module and that would cause it to load correctly.

However, this doesn’t seem to work on my laptop at work. Some googling found a page with the command “rfkill list all” which shows the list of devices on your computer that broadcast some kind of wireless signal (wifi or bluetooth). “rfkill unblock all” reenables all devices, which fixed my issue.