Saturday, October 23, 2021

Setting up Xmonad on a Lenovo ThinkPad X1 Carbon: Getting Started

From time to time, I have profoundly bad ideas. One of those ideas is to change out the window manager on a Linux PC.

For many tasks (such as maintaining this blog), I keep a Lenovo ThinkPad X1 Carbon running Linux. It’s been a good little workhorse, running Ubuntu 20.04. I decided to switch my window manager to Xmonad, because I enjoy fast manipulation of my windows without needing to tell the window manager where they should live.

In principle, this should be very straightforward. In practice, it turns out that switching out your window manager causes a wide variety of seemingly-unrelated things to break. As best I understand it, this is because several basic “housekeeping” tasks of a desktop Linux environment actually route through the window manager, and when the WM goes away, the configurations that launch those tools goes with it. Here’s a short list of things that no longer “just work” when you switch off (I keep discovering more as I go):

The rest of this post will go into how to get a basic Xmonad setup done. Over the next while, I’ll fill in more information on how to get the remaining functionality back without Canonical’s default window manager in place.

First steps

I drew heavily from a beginner’s guide to Xmonad, which generally steered me in the right direction. I was able to install the relevant components with apt-get. This got me a configuration running with Xmonad, Xmobar (to display content at the top), and Stalonetray (an icon tray, which is necessary for some X11 apps to have a place to show system status).

After finishing this config, some things worked but some took some banging on. In particular,

  • Xmonad has been updated since this guide was released, and the way “docks” attach has changed. My xmonad.sh factors this change in.
  • xfce4-power-manager icon is black-on-black, so I thought it was missing. It was simply invisible in the default all-black color scheme of the top-bar. Switching to a dull grey helped the missing icons appear. Once I knew what the issue was, I was able to install lxappearance and run it to change the icon theme configuration; switching to “Ubuntu-Mono-Dark” provided an icon that was visible.
  • To attempt to integrate with the sound control buttons on the ThinkPad, I pulled in “xmonad-extras,” which integrates against “Third party extensions for xmonad with wacky dependencies” (I’m leaving it as an exercise for the reader whether ‘controlling volume via keyboard’ should be considered “wacky,” but you do you). This proved more challenging than I expected because apt-get actually knows this package as libghc-xmonad-extras-dev, but once I found it it worked fine (though I will probably extract it for reasons I’ll get into when I drill open audio).

Put it all together and the end result so far isn’t terrible!

Screenshot of my desktop with xmonad running: status bar at top, three windows open
I don't expect it to win any awards, but it's pretty functional


The Configs

Below are the relevant config files I have so far.

Monday, October 18, 2021

Moving big chunks of text: how I use emacs

(This post is a followup to a previous post on the topic of using emacs for blobs of text) 

As a programmer, sometimes your job is to move blobs of text.

This is unavoidable for many reasons: while we strive for approaches that maximize brevity and we do our best to avoid repeating ourselves, there are tradeoffs. .Configuration files, in particular, tend to be "flat;" Logic that isn't repeated can be harder to comprehend (seeking information in two or more files to understand a concept) and if there is a mechanism for expanding a meta-configuration language into a "flat" configuration, someone has to build and maintain that mechanism.

As an example of a task I have to do often, let's assume I have the configuration for a dashboard program. The dashboard has multiple configurations in which it runs (monitoring several environments, perhaps).

A list of directories displayed in the emacs *dired* mode

Each directory might look a little different, but they all have a 'config' subdirectory.

An environment subdirectory in emacs *dired* mode

The config subdir has several files, one of which is an environment config:

A config subdirectory ein emacs *dired* mode, showing the env.conf file

Finally, env might have several key-value pairs, which vary from file to file.

The contents of an env.conf file

Let's say that I need to update where we get our data from, and for all of these environments, I now need to pull from the copernicus datasource instead of the tycho datasource. I could go through, one by one, and make the edit, but that takes time (and repeatedly re-typing the same small set of keystrokes is error-prone). When I have to manage nearly the same configuration for multiple parallel processes, I infrequently find myself needing to copy the same config change across many files. When I need to do that, I turn to emacs keyboard macros.

Because emacs is built as a text editor running in a virtual LISP machine (as discussed in my previous post), it is capable of recording and replaying every input. We can start a macro at any time with Ctrl-X, then (, (for brevity, I'm using emacs abbreviations from here forward, where the previous macro is C-x (). C-x e can then be used to replay a macro. At the end of macro replay, emacs is in a special "mini-mode" where hitting e again will play the macro again, over and over. I like this approach because it lets me break down the task into smaller steps and spot-check the work; doing these kinds of edits as a shell script is some people's preference, but I feel the shell solution is usually a bit like hammering a few nails by renting a steamroller to drive over them: quite a bit of setup, and if you mess up, you really mess up.

So here's how I approach this task:

  • Navigate to the top-level directory
  • Do the task by hand one time to check for sharp edges
  • Before doing it the second time, Ctrl-X ( to start a macro
  • Now, record the macro. As we go, we're going to be sensitive to how individual configs could vary and lean towards using commands, not the arrow keys to navigate. Arrow navigation will fail us if a subdirectory has too many files or the env.conf file has too many parameters.
    • Enter to descend to the dashboard subdir (dev, in my case)
    • Ctrl-X [, move cursor to beginning of buffer
    • Ctrl-s, interactive search forward. Type config and hit enter. Tricky bit: I have to be careful typing, if the search comes up empty here, my macro will "Bell." More on that in a bit, but if I "bell," I usually just C-x ) to stop editing the macro, finish this one out by hand, and start a macro on the next one.
    • Enter to resolve search, then enter again to descend directory
    • Same plan: beginning of buffer, C-s, env.conf, enter-enterNow that we're in the config file, C-x [ to beginning of buffer, then C-S and search for DATASOURCE=. Enter to confirm search, which moves the edit point to after the equals sign
    • C-k, kill the whole line
    • type copernicus
    • C-x C-s to save the buffer, updating the file
    • C-x k enterC-x k enterC-x k enter to get to the dashboards directory again (closing up all the subdirectory buffers and the env.conf file as we go)
    • (This is key) Hit down arrow one time to move the directory cursor to the next directory
    • C-x ) to close macro


Now that we've done it one time, I can just hit C-x e, then e e e e to update prod, remote, staging, and test.

Checking your work and the zen of emacs: buffers are state machines

So why bother with this instead of a shell script? What I like about this approach is that if something goes wrong, it's much easier to recover than a shell script. If an error occurs while a shell script is running and the script bails, I'm now in a not-great state: running the script again will try to re-edit the files that are already edited, which is rarely what I want. There's no way for the script to know; it has no context on previous runs. But emacs keeps context in the form of the point (cursor)  position in the top-level directory buffer, which doubles as a progress-tracker. This is a valuable piece of the zen of using emacs, which is worth highlighting:

Emacs buffers are stateful. The point doubles as a progres tracker.

This gives emacs a nice tradeoff between fully-manual and fully-automatic edits for repetitive tasks. The command-line is a sword; a shell script is a machine gun nest. Emacs keyboard macros are a semi-automatic weapon: every push of e is a pull of the trigger. If something unexpected happens (i.e. a search fails because the env.conf file or the DATASOURCE row is missing), emacs will take a "bell" event and the macro will interrupt, which allows me to correct the state of the world and then run the macro on the next line instead of starting over.

Using the buffer point as state opens up a couple clever tricks that I find significantly harder to do in shell. Say, for example, that instead of switching everything from tycho to copernicus, I needed to set each file to its own DATASOURCE. In shell, this'd be a little tricky; I'd have to do a lookup file of some kind. With emacs, I just create a new temporary buffer, *datasources*, in which I put a sequence of environment-name / datasource name pairs ("admin: newton","dev: einstein", etc.). Then, I'd change the procedure I described previously as follows:


  • Open the temporary buffer in a second window
  • At the beginning of the macro: before opening the directory, select the directory name and use M-w (metakey-w, usually "alt") to save the directory name
  • At the step where I'd insert copernicus, instead do C-x o to switch to the second buffer
  • C-x [ to go to the beginning of the buffer, then C-s C-y enter to search the buffer for the name of the directory
  • rightarrow to the beginning of the value next to the directory name, then C-space, C-e, M-w to select the value and copy it to the "kill ring" (emacs' concept of a paste buffer)
  • C-x o to go back to the env.conf buffer and do the copernicus replacement, but use C-y to paste ("yank") the value copied from the other buffer

Here's what it looks like.

There are many ways to solve this problem, and different coders will have different favorite approaches. I'll definitely find people who swear by using a shell script for all this. But I think the important thing is to talk about it; something I've noticed in my career is how rarely developers talk about the way they do their craft. If you're a professional developer, I definitely recommend taking some time to look into other people's favorite approaches and find what works best for you.

Friday, October 15, 2021

Moving big chunks of text: why I use emacs

This one ran a little long, so I'm turning it into a two-parter. This part is about emacs and the theory of choosing your tools in general. In the next one, I'll show off a demo of how I use emacs to automate a task that is annoying toil but I don't do quite frequently enough to justify fully automating it.

In a previous post, I mentioned that half my job is moving big blobs of text around, and how important it is to automate that process. I'm going to provide a concrete example using my multi-tool of choice, emacs. Before I continue, of course, I should address the elephant in the room.

Why not vi?

I never really learned it.

That's not very satisfying

Yeah, sorry. Twenty years on from when I first encountered both tools in college, My use of emacs is pretty much just luck. I started learning it first, continued learning it, and now I have accreted a 103MB configuration directory that follows me around from computer to computer. If you tell me vi is great, I'll agree. If you tell me vi is better than emacs at something, I'll believe you.

But if you tell me there's something vi can do that emacs definitely cannot, I'll be modestly surprised and want to know more. Because that's the thing: these tools are more similar than different. And passionate arguments about which of two things that are 90% the same are better is a young person's game. In fact, on topics like this in general, as a word of advice:

For software of the same category, choose one and go deep on it. Know enough about its neighbors to learn more if you need to.

I find I cover more ground when I do this.

So why emacs?

emacs logo


What emacs offers that matters is the following:

Ancient

Emacs was written in 1976. It's older than MS-DOS and only three years younger than the XEROX Parc Alto, the first personal computer to offer a graphical user interface (at an affordable $32,000). Nothing in software is forever, but anything that has stuck around for 25 years in this industry has crocodile DNA. The reason this matters is that community matters for the longevity of software tools; if there's no community, you're building and maintaining it yourself, which is fine if that's your goal but a massive cost otherwise. If you're going to invest heavily in something, it should be something that probably won't fall out of fashion next week.

Degradable

Because of its age (and a community mentality of keeping it maximally compatible), emacs runs on UIs ranging from a full GUI to a decent-quality TTY emulator (i.e. command line). As a result, I rarely find a computing environment I can't bring up emacs on (with just a little configuration). It's comfortable to be able to quickly spin up a familiar development and editing environment on every machine I sit down at, and decreases friction from having to learn something novel.

Network-aware

I don't merely mean "can do TCP/IP" here. Emacs is constructed as a distributable tool; the editor engine and the client can be running in different programs and even on different machines. Furthermore, emacs is aware of SSH and can remote-control the filesystems of other machines, meaning I can edit remote files without having to spin up a shell on them or copy files back and forth. That proves to be a big deal pretty often in my modern ecosystem of remote controlling computers.

Deeply, deeply reconfigurable

This is the important property of the tool, the one that justifies keeping a 100MB config directory around and learning all the esoteric space-alien-language keyboard shortcuts and interface metaphors that predate such exotic topics as "mouse" and "desktop" (to say nothing of "cut", "copy" and "paste"). Emacs stands for Editor MACroS. There's a joke that emacs is really a virtual LISP machine that someone wrote a text editor in.

It's not a joke. It's the law and the whole of the law and it's magic.

Emacs lets you reconfigure basically everything. Every command, every menu option, every human activity is running a command that can be overridden. Even individual keystrokes are just bound to default "insert this character at the current point in the current buffer" functions. That massive extensibility means emacs is something more than a text editor: it's a development environment in which you can develop it. I've seen a full implementation of Tetris running in it. I'm in the process of writing a shell wrapper in it (more on that... Possibly never, I make no promises. ;) ).

The text of the "self-insert-command" help documentation
emacs is built on self-insert fictions


And really, that's the most important thing. In fact, second word of advice:

Seek out the most powerful tools. The most powerful tools can modify themselves.

The tradeoff of doing such a thing is that you eventually end up with an emacs with one user. I don't expect anyone to understand or help me with my emacs configuration; there's 100MB of custom behavior on top of it. Hence, the need to go deep: to maintain a well-customized tool, you have to become an expert in it. if you really drill down on a tool, you may find that in the end, you're the only one who can fully understand what you've done to it.

A screenshot of my emacs editor. On the top is a view of the emacs_config directory I've built. On the bottom is some of the text of this blog post.
This is my editor, this is my GNU. I built it for me and not for you.

What other tools do you use?

In my day-to-day, I have a handful of other things I develop with.

Chrome

Chrome browser split into two panels. On the left is a pair of wheels. On the right is the inspector.
This screenshot is a lie; I'm not debugging anything here. But the wheels are real

This is a bit domain-specific, but: I use Chrome for my web development. Chrome has some overlap with emacs in terms of its extensibility, and the developer tooling is pretty solid. I've even built a (shudder) debuggable build of Chromium. Once. For a very special reason.

... and yes, working at Google is a huge part of this choice. I have no strong opinion on Firefox vs. Chrome, other than the ugly market truth that if you're trying to provide a service to the most people, debugging your website on Chrome and Safari first will cover more people's usage environment faster than Firefox.

VSCode

A VSCode editing screen, showing navigation bar, some text, and some hints

It's weird that I use a second text editor, I know. But I've been pretty impressed with VSCode; for the kind of development I do (mostly TypeScript, JavaScript, and Java), it's a very effective IDE. The formatting, autocompletion, and static checking integration out of the box is simpler to configure than emacs (the one downside of a hyper-flexible editor: many ways to do a thing means many ways to do it wrong). And it shares emacs's comfort with operating in a networked environment, being able to connect to a remote instance of itself over SSH. It's possible that in 25 years, it'd be the kind of thing I'd recommend over emacs if it stands the test of time. I keep emacs launched in a second window for the few tasks VSCode makes harder (it can be very opinionated about auto-formatting files that already have a format set up that is incompatible with the VSCode defaults, and rather than either turning off the formatter or resetting its rules to match the ones in the target file, it's easier to just pop the file open in emacs and let its lack of auto-formatting help me out).

In my next post, I'll demonstrate a task I have to do not-infrequently and how I speed it up a bit with emacs.


Tuesday, October 5, 2021

Things a computer science degree didn't teach me about software engineering

Agent K and James Edwards III, soon to become Agent J, in an elevator. Scene from movie: Men in Black
"Whatever you say, Slick, but I need to tell you something about all your skillz..."

While I'm very happy with / grateful for my computer science degree, I eventually became a software engineer. And the thing about software engineering is that it's related to, but quite distinct from, computer science. Following up on my previous post about that topic, here's the topics and skills I wasn't taught in my computer science undergrad that proved important in my career so far. Along with each one, I'll try and provide the closest thing I remember from undergrad.

Reading other people's code

Almost no professional software engineering is done alone, which means reading other people's code is basically a necessity. And other people's code can be atrocious. While I learned to write well enough to be understood, the skill of fitting my mind around someone else's mental model to understand their code was rarely exercised.

Incidentally, one study has suggested that aptitude to learning other programming languages is related to aptitude in learning natural languages significantly more than aptitude in the mathematics that makes up the core of computer science.

Closest I got: We did some debugging exercises which touched on this space, but the biggest match was becoming a teaching assistant (TA) for one of our courses involving a functional programming language gave me a lot of practical experience in understanding other people's code. I recommend it highly.

Choosing what problems to solve

It turns out, there are infinite problems to be solved with computers, and succeeding in a software engineering career involves a certain "nose" for which ones are the most valuable. Growing this skill can involve both introspection and retrospection, and is generally not a challenge that comes with undergrad, where problems will be handed to you until you complete coursework. Any software engineering career track will eventually expect you, the engineer, to generate the problems that should be solved; those are the engineers that a company needs, because they underwrite the bottom line for the next five years.

Closest I got: I undertook an independent study, but unfortunately it put me off of grad school and PhD pursuit. The gaping maw of the unknown was too much to stare at. But over time, I've been able to learn this skill better on-the-job.

Deciding on a solution, designing it, and defending it

With coursework, there is generally one metric by which your solution is evaluated: passing grade. But engineered software will be maintained by other people and often interlocks with a huge edifice of already-existing technology operating closer to or further away from the end-user. The right way to solve these problems is your responsibility to create, and your design will often require explanation or even defense of its fit vs. alternatives. These aren't software skills; they're people skills and they involve understanding not just the system you're building but the people interested in seeing it built, as well as time and cost tradeoffs.

Closest I got: As a TA, I had an opportunity to design new lessons and put them in front of the professor. This was an excellent opportunity to refine my ability to explain and defend something novel I'd created.

Huge systems

Corporate software accretes features. Once something is built, it is incredibly rare for the company to decide to stop doing whatever the thing is that the software supports.

As a result, becoming a member of a large team at a mature company will often mean struggling with software of absolutely enormous complexity. This isn't useful to do to undergrads; pedagogy is better served by approaching the elephant one piece at a time, and any support systems needed for a class tend to be slim and focused. But big-world software (especially web software) is not designed to be understood by one person; that's why companies hire teams, and it's why reading other people's code becomes so important. A huge system requires some limited expertise in a broad skillset, because eventually an abstraction breaks and, for example, you have to care about the network layer to fix the UI, or the 3D rendering layer to build the schema for a new dataset, or the package management layer to write any code at all. And industry software often has to make practical tradeoffs that more abstract, academic software can skip... Does your system have a time limit in which it can operate, because you're building banking software and your peers turn off on weekends? Is your website breaking because you've hit the limit for max URL size (which is implementation-dependent)? Are you building on a system that lacks a floating-point coprocessor (or, for that matter, a "divide" operator)?

Closest I got: Reading the source code and design notes of big systems. The Linux kernel is open-source. So is Docker. So are multiple implementations of MapReduce. Reading and really trying to understand all parts of one of those elephants can give a person perspective on how wide the problem space can be.

Handling user feedback and debugging user bug reports

Real-world systems have real-world users, and users are the greatest way of finding faults that have ever been devised. Software of sufficient quality to pass academic exercises is often not nearly well-tested enough to be easily maintained or improved in the future, or grow to handle new requirements; it's write-and-forget software. Related to this: academic projects that require code often provide relatively straightforward feedback on problems. User feedback is anything but straightforward (when it isn't outright abusive, because users are emotionally angry that your tool got them 99% of the way to where they wanted to be and then ate their data).

Closest I got: While I actually had a few classes that taught expecting the unexpected, nothing was as educational as building real software that real users use. Whether it's maintaining a system relied upon by other students or building a library used by people on GitHub, getting early experience with the bug-debug loop with strangers in that loop is useful.

Instrumentation

More challenging for a software engineer than vague or angry bug reports is the bug report that never comes, where the user just gets frustrated and stops using your software. Software operating at huge scale can't rely on users to have the time or patience to file reports and must police itself. The practice of instrumenting systems (encompassing both robust error capture and reporting and observation of users' interactions with the tools, what pieces they care about vs. what goes unused, whether they often get halfway through a designed flow and stop, etc.) is as much art as science, but it demands its own attention to design and implementation.

Closest I got: This is a big piece that was basically never touched on in undergrad. Debugging exercises were close in that they could involve temporary instrumentation to understand code flow, but this is a skill that I think may be under-served (unless pedagogy has changed in fifteen-odd years).

Automating toil and transforming big blobs of text

While code isn't text, much code is represented as text and mechanically, a software engineer spends a lot of time moving text around. A significant difference between a stagnant software engineer and a growing one is whether they get better at that process. That can involve both practice and recognizing when a particular task is a "hotspot" that is repetitive drudgery enough that it could itself be automated. There's no rule against writing code to write code; half of software is probably exactly that. I like to rely on tools that make this kind of growth easier, and I'll touch on that in a later blog post.

Closest I got: Computer science exercises tend to try and avoid being the sort of thing that can be solved with moving big blobs of text, trying to instead focus on problems solvable with the right (relatively) few lines in the right place. One exception is an operating system design course I took, and that one gave the most incentive to learn how to quickly replace phrases globally, or move an entire data structure from one form to another (when we the developers didn't expect that change so it wasn't cleanly structured to do it).

Growing your tools

Related to the previous topic: the industry will change over time, and the tools available to you will change with it. Like any good practicing laborer in a technical field, be it medicine, agriculture, or manufacturing, learning of new tools and improving the ones you already have is a continual exercise. The ride doesn't end until you get off.

Closest I got: Everything and nothing. Improving and seeking to improve one's tools has to become like breathing air to a software engineer; this is the "piano, not physics" part of the practice of programming itself. Better IDEs, better languages, best practices, new frameworks, new ideas... Seek them out, and may they stay exciting to you your entire life.

Tuesday, September 28, 2021

Things I wish I'd Known Before Getting A Computer Science Degree

 Hi, my name is Mark, and I struggled with impostor syndrome for decades.

("Hi, Mark").

In my case at least, impostor syndrome came from the discontinuity between being good at things in high school and far more mediocre in college. I came from a place where I thought I had things figured out to a far more challenging environment in a different city, with peers who knew way more than I did about a subject I thought I was passionate about. For awhile, I seriously considered changing disciplines.

These feelings don't really chase me anymore, but I've thought for some time about why they came up and what I wish I'd known—in the high-level sense, not the day-to-day sense—about the path I'd started out on.

So, dear Mark in 2001, here's some things Mark in 2021 gets to know.

Computer science is more math than engineering

You're into computers because of what they can do. That's cool. Computer science is about why they do it. And the why brushes up against the limits of what they can do. You've never thought too much about that aspect of things. You'll be thinking about it a lot these next four years. Some of this will be hard, because you have a mind that thrives on learning things you can apply right now, not on abstraction and theory. 

You'll adapt. 

It'll hurt like hell. 

One thing that helps adaptation: tie the whys to the whats. Learn the history of how we came to know these things (oftentimes, we learned them because someone was chasing a what, got stuck, and had to chase a why to understand the reasons "what" wasn't working). Computer science as a pedagogy isn't interested in its history, so it won't hand those pieces to you... But you are. Seek them out.

Software engineering isn't computer science


You're gonna be real discouraged in freshman year because a full half of your classes are deep math you haven't seen before. That's okay. It gets good junior and senior year when you start applying the theory on larger projects. But understand: that theory is computer science. Your peers that go on to grad school or to pursue a PhD, they aren't likely to be designing the next generation of developer tooling—they might, but they can also make a career out of never publishing any mainstream-known software. This discipline is about understanding the pure-math space of computation as much as the machines, "meat and bones" of how to make a computer go.

Computer science isn't software engineering


On the flip side, there are things you will absolutely need to know to be a successful writer of software people use and maintain that a computer science undergraduate degree doesn't need to teach you. I've read some brilliant code by some very smart researchers that is incredibly bad for communicating the concepts behind the code to other people—single-letter variable names, zero comments, no encapsulation, and so forth. It's not necessary to do those things to publish novel research, so those things aren't done. I'll write another article on this topic in the future.

Computer science relies on mathematics not always taught in high school

Palpatine talking to Anakin Skywalker from Star Wars. "Have you heard the tale of the Travelling Salesman? It is not a story the calculus teachers would tell you."
Professor Palpatine introducing a freshman to discrete mathematics

Computer science is the science of algorithmic solution of problems: what problems can be solved by algorithm, how hard it is, and what algorithms are the same thing in disguise. Of the mathematics an average US high school student will know, the least likely in this set is discrete mathematics. Discrete has few practical applications relative to algebra or calculus, so it often falls by the wayside. But it's fundamental to computer science, because it deals with the mathematics of processes that progress in orderly, distinct steps.

You don't really know the term "discrete math" right now. You'll be comfortable enough with it to make jokes about it soon.

If there were a way to bootstrap yourself into this before getting to college—summer class, elective in high school—I can't recommend it enough. They'll teach it to you in college, but if you get there knowing it, it'll be one fewer thing to worry about while you're trying to stuff everything else into your brain.

The industry is young, and pedagogy varies wildly


Your father made a career out of writing software on the back of an English undergraduate degree. As a disciplined field, computer science is young; pedagogy varies wildly between institutions that offer computer science degrees. You'll take an internship in Philadelphia where you'll meet several students from Drexel; they'll run circles around you in terms of prolific code authorship, because they've been using the tools the company uses since freshman year. But Carnegie Mellon intends to graduate potential grad students and PhD candidates from their undergrad program; they start computer science on fundamentals of theory and mathematics, and most code can be written in a plaintext editor. Drexel, to my observation, biases towards getting students productive earlier; they have classes intent on teaching more intricate and esoteric parts of tools like Visual Studio (whereas CMU's opinion on such things is that Visual Studio didn't exist 25 years ago and may not be in use 25 years from now). So you'll be slogging through "How to set up a project" while they're scripting the debugger. Tuck in.

The field is moving so fast, it's impossible to predict what will be relevant

When you start, you'll meet some grad students who are very excited about computer simulation of electronic circuits. The idea they're chasing is that instead of building a circuit by exhaustively laying out logic gates on a chip, you instead use a high-level language to describe what the chip has to do, and an algorithm could "grow" a chip's physical design by heuristically selecting logic gate configurations to satisfy the requested input. This process will become standard in huge swaths of hardware design for new custom circuit fabrication. You'll also watch sloppy prototypes of voice recognition and image recognition explode into bog-standard technologies that come on every smartphone as a pack-in deal.

Faced with this rapid pace, the only option is to tuck in and prepare to be learning something new continuously, your whole career, forever. As one of your mentors freshman year will say, "Programming is piano; it's not physics." This is a field where what is possible is expanding at a rate we can barely keep up with. Enjoy the ride; don't get lazy.

Set aside time to get good at the tools

Corollary to the previous: a lot of classes will focus on teaching theory and rely on the students to learn the mechanics of their tools on their own. If it can be read out of a manual, it's not something the college is focused on expositing (the manual is, after all, right there).

Set aside time to get good at using your tools of choice and exploring their nooks and crannies. The biggest favor you'll do yourself in your undergrad is taking the time one summer to learn the esoteric language used in the class you'll be taking next semester. It will still be a challenge, but you won't be struggling over syntax and theory at the same time.

Put thought into where you want to be in four years

College can give you tools to succeed but won't tell you who to be. It can be easy to get so heads-down on the tasks and classes that you wake up one day and realize you don't have a life path beyond "student." That can be your life path, but if you want that, seek out grad students and PhD candidates and talk to them. You might find what they do with their time is what you want to do; you might not! Research and deep theory isn't for everyone.

If, instead, you choose industry, you will never be freer in your life to choose which one you pursue than graduation year. You will have an incredibly valuable talent to share with the world; it's up to you who benefits from that. Make sure it's someone who deserves it.


Whatever you're interested in doing, computer science will get you closer to it

Software is eating the world. There are very few things humans do now—in industry, in recreation, in travel, in finance—that don't (or can't) involve a computer somehow. And the addition of computers opens entirely new ways of doing things that weren't even considerable previously. The skills you will acquire to understand the fundamentals underpinning all of these new approaches and technologies will be invaluable navigating the world that is evolving in front of us.

This is going to be a hell of a good time. Enjoy it.

Wednesday, July 7, 2021

How to keep your gnome-terminal from hanging in SSH connections

I've been having problems with my GNOME terminal hanging when on an extended SSH connection. Finally realized today there's probably a keepalive configuration somewhere.

There is.

Go into your ~/.ssh/config file and add this stanza under your Host config:

  ServerAliveInterval 240

(note the two spaces to make it a sub-property of Host)

If you want every SSH connection to stay alive, add the rule for every host

Host *
  ServerAliveInterval 240

FAQ

  1. Why isn't this the default?

    Really? You're using a Linux environment and you're asking why the default isn't "Don't just break?" 

    😉

Thursday, November 5, 2020

Adding music to a NES game

Petris now has a theme song, courtesy of my niece. :)



I asked for some help from my nieces to come up with a small bit of music for the game, since I'm not much of a composer myself. After I described the constraints of the NES to them (two square waves, one triangle, one noise), my oldest niece was able to get it to me as a recording she put together on an iPad; I just had to get it transcribed into a format the NES would accept and a music engine to play it.

Enter FamiStudio.

FamiStudio

FamiStudio is a nifty combination of tool and framework for constructing NES music and sound effects. It consists of a couple pieces:

  • A desktop application for composing music or sounds against the NES hardware
  • Several audio engines in assembly that can be embedded in an existing NES source code
The system supports both the in-hardware features of the chip and a few software-implemented effects (such as arpeggio). Music is broken down into "patterns" and "instruments," where patterns allow for saving memory by repeating the same musical phrase and instruments combine "voicing" rules with notes to create various timbres the sound circuitry can support. I put together a FamiStudio project and dove in.

My niece gave me a fairly simple piano-and-drum piece, so I only needed two instruments for it. For the drum, I used a bassy frequency and tweaked the volume envelope to give a sharp hit with a quick die-off. The volume is quiet for one frame to give a "squishy" feel, which sounds more bass (soft mallet hitting a big membrane, maybe? I don't know much music physics ;) ).


To get the sound a bit more "drummy," I used the arpeggio effect to tweak the noise frequency higher for two frames, then back down to the selected frequency. It puts a bit of hiss on the front, like a metal rattle that dies off; not quite snare drum, but a bit more color than a flat "boomph" sound.



The square wave carries the two-note melody and only has a few tweaks. Duty cycle selects what percent of time the square wave is "on" and can vary between 12.5%, 25%, 50%, or 75% (which is just 25% again; human ear can't tell the difference). 50% gives me a good old-fashioned "pure" square wave with no flavor, which is very piano. For volume, I went with a sharp attack and a long-tail decay; no deep thought on this I just doodled on the waveform until it "felt" good.



Now that the song exists, I can export it as FamiStudio Music Code using the NESASM format, since Petris is written in nbasic and nbasic includes an escape hatch to NESASM. The result is a data file containing a representation of the music, but not the commands to play it.


Halfway there

The nbasic language doesn't include a "file import" operator, so I copied-and-pasted the content of that file between asm and endasm directives. Now, I just needed some code to play it.

FamiStudio Sound Engine

FamiStudio's GitHub repository includes an implementation of the audio engine in several different NES assembly languages. I grabbed the one for NESASM and pasted it directly into Petris's source file. The audio engine is quite featureful, and quite large; everything I'd written for Petris to that point now composes 19% of the soruce file. So I compiled and..... The assembler choked.

So I'm not sure if I'm using an older version of NESASM (version 3.1) or if FamiStudio was built against a newer version, but NESASM itself has a hard limit of about 32 characters worth of label that the FamiStudio Sound Engine completely ignores. I made and documented 14 search-replaces to trim the label lengths down to something acceptable to NESASM, and we were off to the races.

To connect the audio engine to my game, I had to run some procedures exposed by the engine. The engine documents these pretty well, but a quick summary follows. Since we're operating at assembly level and don't have a standard function-call protocol, each function documents what the a, x, and y registers have to be set to for the function to work.

  • famistudio_init sets everyting up. It accepts a selector of 0 for PAL timing and 1 for NTSC timing (a), and a pointer to the label at the top of the music data spit out by FamiStudio (low-byte x, high-byte y; in my case, that's untitled_music_data.
  • famistudio_music_play begins playing a song. It takes the index of the song to play (a); I only have one, so I just had to load a 0 into the accumulator.
  • famistudio_music_stop kills playback and takes no input params
  • famistudio_update is a "meat and potatoes" function that must be called once per animation frame (i.e. once per non-maskable interrupt cycle) to allow the engine to move the music forward. More on this later.
The nbasic language is very good about bridging to assembly, so calling these was no more complex than throwing down a small wrapper subroutine to make juggling variables easy.

It's almost certainly overkill to save all the registers, but no harm done

I added gosubs to these routines to the relevant locations, through a call to the updater into my nmi interrupt logic, and... Started an earthquake.


Here's what happened. I made the mistake of adding the update function into my code here, after drawing is completed:

You feel like you're gonna have a bad time...

Problem is, that means I'm running the audio update loop inside the vertical blanking period (the NMI fires at the start of vertical blank. This is, generally, setting yourself up for failure; if the audio processing takes so long that the vblank period ends, any more manipulation of the PPU before the next NMI fires is going to fight the PPU itself for control of state, as the PPU uses its memory latches to generate a frame of video output. Specifically in my case, vwait_end does some cleanup and sets the scroll register correctly, so firing it after vblank ends causes offset artifacts as video rendering continues with the wrong scroll value.

Fortunately, the solution is real straightforward: don't do that.


Now we're safely setting up audio while the PPU is doing the heavy-lifting of emitting one frame of video, and life goes on.

Lessons Learned

This was a fun exercise to finish, and I'm honestly very surprised at how good the public tooling is for NES audio. The audio engine available under the MIT license was a huge win, and I want to extend a huge thank you to Mathieu Gauthier (a.k.a. BleuBleu) for making that exist. He doesn't appear to have a Patreon, so drop a note on one of his video tutorials and tell him what a cool guy he is.