Understanding USB Serial on Arduino Uno (There Is No Magic)

(source: http://gordonscruton.blogspot.com/2011/06/no-magic-please-part-1-learning.html)

A professor I had was fond of saying that the first thing to learn about computers is that there is no magic. From time to time, I am reminded of that truth. So let me tell you a story.

I've been playing around with a lighting project using some Supernight LED strips (unlike NeoPixels, these light strips allow you to control the hue but don't allow for individual LED clusters to be controlled; the entire strip takes on one color). After having some fun with them using the packaged-in controller, I decided to wire up my own based on some previous work and a tutorial from Make on how to control these lights (they're basically four-channel: one channel is +12v, and the other three can accept a PWM signal for color intensity). I used an Arduino UNO as the driver for the light control circuit. Everything was going well: I did a breadboard test, wired it all up, and put it in a nice project enclosure box.

To control the lights, I'm running a web service on a Raspberry Pi that communicates via USB serial to the Aruduino (in theory, the Pi could send the PWM signals directly, but I'm squeamish about wiring my harder-to-replace Pi to a 12 volt signal). I wrote a small state machine to accept some serial signals on the Arduino, and a small go service to take web requests and translate them into opening the serial port, sending the data, and closing the port.

When it works, it works great!

Trouble is, it didn't work. I could see my server sending the signals, and it believed they were sent correctly, but the Arduino wouldn't respond. I could even go into the Serial Monitor in the Arduino IDE and send the characters individually; they'd trigger the lights, but when I sent them from my server, the lights would go out. I couldn't debug by printing messages back over the serial connection, because I was using it to control the board (PROTIP: always add a couple LEDs to your enclosure; you never know when you're going to need the ability to just get a simple status signal from your Arduino without using serial).

The breakthrough came from adding a bit of code to the setup() function that caused the lights to quickly pulse red-green-blue to indicate setup was complete. I quickly learned that every time I tried to send a signal to change the light color from the Pi, the lights were pulsing---the Arduino was doing a full reboot. Why?

I had assumed that the Arduino was (at the hardware level) talking directly to the USB, and the Serial library I used emulated a serial connection in software. In reality, that's not the case at all. The secret is described clearly (well, clearly enough ;) ) in the wiring diagram for the UNO itself.

(Source)

The main processor is an ATMega 328P, but there's an entire second processor on the chip---an ATMega16U2 that sits between the main processor and the USB hardware. That processor has (normally) non-modifiable firmware that translates the USB serial protocol to a traditional 5v serial logic protocol. It also has a useful feature, sitting right in the middle of the diagram:

"There's yer problem."

It's a bit hard to tell in all the visual noise, but that's pin 13 on the USB controller wired to RESET on the main processor. So when does the USB controller reset the main processor? When a USB connection is initiated, the controller sends a DTR signal on pin 13. This signal reboots the processor. Every time my web service opened the serial device in Linux, it was causing the USB controller to trigger a DTR---the subsequently-sent bytes were then either being consumed or dropped on the floor. Fixing this was as simple as re-coding my service to open the device once and hold onto it for the entire life of the process. No sweat!

So this still leaves us the question: why is DTR wired to RESET? Here's the magic I was missing: The Arduino UNO uses the USB connection for two things: sending and receiving serial communications, and programming the Arduino itself. How does it know that the signal coming in is supposed to be new program logic and not serial commands for the program to consume? It turns out that the IDE reprograms the Arduino by starting a serial connection (causing DTR to pull the RESET pin). The first thing the Arduino's main processor does on reset is run firmware that listens for a few milliseconds for a sequence of bytes indicating new code incoming (these bytes are documented in the STK500 communication protocol v1)---if it receives those bytes, it writes them to flash memory, otherwise it gives up and begins running userland code. The Arduino Serial Monitor triggers the reset also, but doesn't send the new-code bytes. Previous to this feature being added, reprogramming the Arduino required the developer to time resetting the Arduino and starting the code upload somewhat precisely, which was annoying. Incidentally, if you own an Arduino and don't want this feature enabled, there are a couple of convenient ways to disable it so that your processor won't reboot every time it starts a serial connection.

So there you go: no magic, just a pair of processors with some interwoven activation logic that allows the same wire to serve as both communications and as the channel to download new program versions. Which, when it all operates smoothly, is pretty magic!