Debugging: The Little Things

Not the offending USB drive
photo: Gary Honey

Recently I was working on some embedded software that someone else wrote.  I needed to add some new functionality to the code, and I was working through how the existing code was working.  In particular, the software was parsing a file, and the previous developer had done a good job of sending out diagnostic messages to the terminal, and the code was parsing a line of the file as a bad record.

Well as a good little developer, the first thing I did was open the original file and check out the line that the embedded controller didn’t like, and of course it was a valid record.  Frustrating.  Well my workflow was to develop the code on a remote Linux computer, copy the files to my laptop via a network share, copy from my laptop to a USB drive, and finally to copy the code and files from the USB drive to the embedded target.

Next I needed to figure out why the “code was bad,” and I spent some time studying the C++ that does the file parsing to determine why it was failing on a perfectly good input file.  After some time, the code looked good, so I added a debug message to output the offending line of the file to the terminal.  I went through my build-copy-copy-copy-run cycle and ran the code – “Illegal Instruction.”  Now I had broken the code.  This had happened before, and I figured it was something I would have to resolve in the code later. I went back, rolled back all my changes, and re-added the one line of output I wanted, build-copy-copy-copy-run.

This time, the code executed, the file parsing errored out and the terminal dutifully spit out a line of garbage that deserved the error thrown.  Hmm, looks like my file is corrupt. I then copied the file from laptop to USB drive to target, ran the code, and got a similar error, different line number – different gibberish.  Things are getting strange now, so I copied the file from the target, back to my PC for analysis and sure enough, the file is corrupt, but how.  Original file is ok, it’s corrupt on the target, and since I copied it back, it’s corrupt on the USB drive.

Next step, I copied the original file from my laptop, to the USB drive, then to the target, then back to the USB drive, and back to the laptop, renaming the file at each step.  When I compared each version, it turns out that it corrupted copying from my computer to my very cool USB drive shaped like a construction machine that I had gotten at a trade show.  I went to the desk drawer and found a good (not free) USB drive, copied the file laptop to drive to target, ran the code: file processed correctly.  Since dumping the free USB drive, the random Illegal Instruction errors have gone away.  Turns out the free drive cost me several hours of debugging time.