Setting up Xmonad on a Lenovo ThinkPad X1 Carbon: Getting Started

From time to time, I have profoundly bad ideas. One of those ideas is to change out the window manager on a Linux PC.

For many tasks (such as maintaining this blog), I keep a Lenovo ThinkPad X1 Carbon running Linux. It’s been a good little workhorse, running Ubuntu 20.04. I decided to switch my window manager to Xmonad, because I enjoy fast manipulation of my windows without needing to tell the window manager where they should live.

In principle, this should be very straightforward. In practice, it turns out that switching out your window manager causes a wide variety of seemingly-unrelated things to break. As best I understand it, this is because several basic “housekeeping” tasks of a desktop Linux environment actually route through the window manager, and when the WM goes away, the configurations that launch those tools goes with it. Here’s a short list of things that no longer “just work” when you switch off (I keep discovering more as I go):

• screen locking
• rendering the desktop (yes, if you give it no commands, the window manager just draws nothing!)
• auto-mounting of plugged-in USB storage
• sleep-on-close-screen
• layout of second monitors
• connecting second monitors
• battery indicator
• sound volume, mute, output control
• Bluetooth connections

The rest of this post will go into how to get a basic Xmonad setup done. Over the next while, I’ll fill in more information on how to get the remaining functionality back without Canonical’s default window manager in place.

First steps

I drew heavily from a beginner’s guide to Xmonad, which generally steered me in the right direction. I was able to install the relevant components with apt-get. This got me a configuration running with Xmonad, Xmobar (to display content at the top), and Stalonetray (an icon tray, which is necessary for some X11 apps to have a place to show system status).

After finishing this config, some things worked but some took some banging on. In particular,

• Xmonad has been updated since this guide was released, and the way “docks” attach has changed. My xmonad.sh factors this change in.
• xfce4-power-manager icon is black-on-black, so I thought it was missing. It was simply invisible in the default all-black color scheme of the top-bar. Switching to a dull grey helped the missing icons appear. Once I knew what the issue was, I was able to install lxappearance and run it to change the icon theme configuration; switching to “Ubuntu-Mono-Dark” provided an icon that was visible.
• To attempt to integrate with the sound control buttons on the ThinkPad, I pulled in “xmonad-extras,” which integrates against “Third party extensions for xmonad with wacky dependencies” (I’m leaving it as an exercise for the reader whether ‘controlling volume via keyboard’ should be considered “wacky,” but you do you). This proved more challenging than I expected because apt-get actually knows this package as libghc-xmonad-extras-dev, but once I found it it worked fine (though I will probably extract it for reasons I’ll get into when I drill open audio).

Put it all together and the end result so far isn’t terrible!

I don't expect it to win any awards, but it's pretty functional

The Configs

Below are the relevant config files I have so far.

Moving big chunks of text: how I use emacs

(This post is a followup to a previous post on the topic of using emacs for blobs of text) 

As a programmer, sometimes your job is to move blobs of text.

This is unavoidable for many reasons: while we strive for approaches that maximize brevity and we do our best to avoid repeating ourselves, there are tradeoffs. .Configuration files, in particular, tend to be "flat;" Logic that isn't repeated can be harder to comprehend (seeking information in two or more files to understand a concept) and if there is a mechanism for expanding a meta-configuration language into a "flat" configuration, someone has to build and maintain that mechanism.

As an example of a task I have to do often, let's assume I have the configuration for a dashboard program. The dashboard has multiple configurations in which it runs (monitoring several environments, perhaps).

Each directory might look a little different, but they all have a 'config' subdirectory.

The config subdir has several files, one of which is an environment config:

Finally, env might have several key-value pairs, which vary from file to file.

Let's say that I need to update where we get our data from, and for all of these environments, I now need to pull from the copernicus datasource instead of the tycho datasource. I could go through, one by one, and make the edit, but that takes time (and repeatedly re-typing the same small set of keystrokes is error-prone). When I have to manage nearly the same configuration for multiple parallel processes, I infrequently find myself needing to copy the same config change across many files. When I need to do that, I turn to emacs keyboard macros.

Because emacs is built as a text editor running in a virtual LISP machine (as discussed in my previous post), it is capable of recording and replaying every input. We can start a macro at any time with Ctrl-X, then (, (for brevity, I'm using emacs abbreviations from here forward, where the previous macro is C-x (). C-x e can then be used to replay a macro. At the end of macro replay, emacs is in a special "mini-mode" where hitting e again will play the macro again, over and over. I like this approach because it lets me break down the task into smaller steps and spot-check the work; doing these kinds of edits as a shell script is some people's preference, but I feel the shell solution is usually a bit like hammering a few nails by renting a steamroller to drive over them: quite a bit of setup, and if you mess up, you really mess up.

So here's how I approach this task:

• Navigate to the top-level directory
• Do the task by hand one time to check for sharp edges
• Before doing it the second time, Ctrl-X ( to start a macro
• Now, record the macro. As we go, we're going to be sensitive to how individual configs could vary and lean towards using commands, not the arrow keys to navigate. Arrow navigation will fail us if a subdirectory has too many files or the env.conf file has too many parameters.
• Enter to descend to the dashboard subdir (dev, in my case)
• Ctrl-X [, move cursor to beginning of buffer
• Ctrl-s, interactive search forward. Type config and hit enter. Tricky bit: I have to be careful typing, if the search comes up empty here, my macro will "Bell." More on that in a bit, but if I "bell," I usually just C-x ) to stop editing the macro, finish this one out by hand, and start a macro on the next one.
• Enter to resolve search, then enter again to descend directory
• Same plan: beginning of buffer, C-s, env.conf, enter-enterNow that we're in the config file, C-x [ to beginning of buffer, then C-S and search for DATASOURCE=. Enter to confirm search, which moves the edit point to after the equals sign
• C-k, kill the whole line
• type copernicus
• C-x C-s to save the buffer, updating the file
• C-x k enter, C-x k enter, C-x k enter to get to the dashboards directory again (closing up all the subdirectory buffers and the env.conf file as we go)
• (This is key) Hit down arrow one time to move the directory cursor to the next directory
• C-x ) to close macro

Now that we've done it one time, I can just hit C-x e, then e e e e to update prod, remote, staging, and test.

Checking your work and the zen of emacs: buffers are state machines

So why bother with this instead of a shell script? What I like about this approach is that if something goes wrong, it's much easier to recover than a shell script. If an error occurs while a shell script is running and the script bails, I'm now in a not-great state: running the script again will try to re-edit the files that are already edited, which is rarely what I want. There's no way for the script to know; it has no context on previous runs. But emacs keeps context in the form of the point (cursor)  position in the top-level directory buffer, which doubles as a progress-tracker. This is a valuable piece of the zen of using emacs, which is worth highlighting:

Emacs buffers are stateful. The point doubles as a progres tracker.

This gives emacs a nice tradeoff between fully-manual and fully-automatic edits for repetitive tasks. The command-line is a sword; a shell script is a machine gun nest. Emacs keyboard macros are a semi-automatic weapon: every push of e is a pull of the trigger. If something unexpected happens (i.e. a search fails because the env.conf file or the DATASOURCE row is missing), emacs will take a "bell" event and the macro will interrupt, which allows me to correct the state of the world and then run the macro on the next line instead of starting over.

Using the buffer point as state opens up a couple clever tricks that I find significantly harder to do in shell. Say, for example, that instead of switching everything from tycho to copernicus, I needed to set each file to its own DATASOURCE. In shell, this'd be a little tricky; I'd have to do a lookup file of some kind. With emacs, I just create a new temporary buffer, *datasources*, in which I put a sequence of environment-name / datasource name pairs ("admin: newton","dev: einstein", etc.). Then, I'd change the procedure I described previously as follows:

• Open the temporary buffer in a second window
• At the beginning of the macro: before opening the directory, select the directory name and use M-w (metakey-w, usually "alt") to save the directory name
• At the step where I'd insert copernicus, instead do C-x o to switch to the second buffer
• C-x [ to go to the beginning of the buffer, then C-s C-y enter to search the buffer for the name of the directory
• rightarrow to the beginning of the value next to the directory name, then C-space, C-e, M-w to select the value and copy it to the "kill ring" (emacs' concept of a paste buffer)
• C-x o to go back to the env.conf buffer and do the copernicus replacement, but use C-y to paste ("yank") the value copied from the other buffer

Here's what it looks like.

link

There are many ways to solve this problem, and different coders will have different favorite approaches. I'll definitely find people who swear by using a shell script for all this. But I think the important thing is to talk about it; something I've noticed in my career is how rarely developers talk about the way they do their craft. If you're a professional developer, I definitely recommend taking some time to look into other people's favorite approaches and find what works best for you.

Moving big chunks of text: why I use emacs

This one ran a little long, so I'm turning it into a two-parter. This part is about emacs and the theory of choosing your tools in general. In the next one, I'll show off a demo of how I use emacs to automate a task that is annoying toil but I don't do quite frequently enough to justify fully automating it.

In a previous post, I mentioned that half my job is moving big blobs of text around, and how important it is to automate that process. I'm going to provide a concrete example using my multi-tool of choice, emacs. Before I continue, of course, I should address the elephant in the room.

Why not vi?

I never really learned it.

That's not very satisfying

Yeah, sorry. Twenty years on from when I first encountered both tools in college, My use of emacs is pretty much just luck. I started learning it first, continued learning it, and now I have accreted a 103MB configuration directory that follows me around from computer to computer. If you tell me vi is great, I'll agree. If you tell me vi is better than emacs at something, I'll believe you.

But if you tell me there's something vi can do that emacs definitely cannot, I'll be modestly surprised and want to know more. Because that's the thing: these tools are more similar than different. And passionate arguments about which of two things that are 90% the same are better is a young person's game. In fact, on topics like this in general, as a word of advice:

For software of the same category, choose one and go deep on it. Know enough about its neighbors to learn more if you need to.

I find I cover more ground when I do this.

So why emacs?

What emacs offers that matters is the following:

Ancient

Emacs was written in 1976. It's older than MS-DOS and only three years younger than the XEROX Parc Alto, the first personal computer to offer a graphical user interface (at an affordable $32,000). Nothing in software is forever, but anything that has stuck around for 25 years in this industry has crocodile DNA. The reason this matters is that community matters for the longevity of software tools; if there's no community, you're building and maintaining it yourself, which is fine if that's your goal but a massive cost otherwise. If you're going to invest heavily in something, it should be something that probably won't fall out of fashion next week.

Degradable

Because of its age (and a community mentality of keeping it maximally compatible), emacs runs on UIs ranging from a full GUI to a decent-quality TTY emulator (i.e. command line). As a result, I rarely find a computing environment I can't bring up emacs on (with just a little configuration). It's comfortable to be able to quickly spin up a familiar development and editing environment on every machine I sit down at, and decreases friction from having to learn something novel.

Network-aware

I don't merely mean "can do TCP/IP" here. Emacs is constructed as a distributable tool; the editor engine and the client can be running in different programs and even on different machines. Furthermore, emacs is aware of SSH and can remote-control the filesystems of other machines, meaning I can edit remote files without having to spin up a shell on them or copy files back and forth. That proves to be a big deal pretty often in my modern ecosystem of remote controlling computers.

Deeply, deeply reconfigurable

This is the important property of the tool, the one that justifies keeping a 100MB config directory around and learning all the esoteric space-alien-language keyboard shortcuts and interface metaphors that predate such exotic topics as "mouse" and "desktop" (to say nothing of "cut", "copy" and "paste"). Emacs stands for Editor MACroS. There's a joke that emacs is really a virtual LISP machine that someone wrote a text editor in.

It's not a joke. It's the law and the whole of the law and it's magic.

Emacs lets you reconfigure basically everything. Every command, every menu option, every human activity is running a command that can be overridden. Even individual keystrokes are just bound to default "insert this character at the current point in the current buffer" functions. That massive extensibility means emacs is something more than a text editor: it's a development environment in which you can develop it. I've seen a full implementation of Tetris running in it. I'm in the process of writing a shell wrapper in it (more on that... Possibly never, I make no promises. ;) ).

emacs is built on self-insert fictions

And really, that's the most important thing. In fact, second word of advice:

Seek out the most powerful tools. The most powerful tools can modify themselves.

The tradeoff of doing such a thing is that you eventually end up with an emacs with one user. I don't expect anyone to understand or help me with my emacs configuration; there's 100MB of custom behavior on top of it. Hence, the need to go deep: to maintain a well-customized tool, you have to become an expert in it. if you really drill down on a tool, you may find that in the end, you're the only one who can fully understand what you've done to it.

This is my editor, this is my GNU. I built it for me and not for you.

What other tools do you use?

In my day-to-day, I have a handful of other things I develop with.

Chrome

This screenshot is a lie; I'm not debugging anything here. But the wheels are real

This is a bit domain-specific, but: I use Chrome for my web development. Chrome has some overlap with emacs in terms of its extensibility, and the developer tooling is pretty solid. I've even built a (shudder) debuggable build of Chromium. Once. For a very special reason.

... and yes, working at Google is a huge part of this choice. I have no strong opinion on Firefox vs. Chrome, other than the ugly market truth that if you're trying to provide a service to the most people, debugging your website on Chrome and Safari first will cover more people's usage environment faster than Firefox.

VSCode

It's weird that I use a second text editor, I know. But I've been pretty impressed with VSCode; for the kind of development I do (mostly TypeScript, JavaScript, and Java), it's a very effective IDE. The formatting, autocompletion, and static checking integration out of the box is simpler to configure than emacs (the one downside of a hyper-flexible editor: many ways to do a thing means many ways to do it wrong). And it shares emacs's comfort with operating in a networked environment, being able to connect to a remote instance of itself over SSH. It's possible that in 25 years, it'd be the kind of thing I'd recommend over emacs if it stands the test of time. I keep emacs launched in a second window for the few tasks VSCode makes harder (it can be very opinionated about auto-formatting files that already have a format set up that is incompatible with the VSCode defaults, and rather than either turning off the formatter or resetting its rules to match the ones in the target file, it's easier to just pop the file open in emacs and let its lack of auto-formatting help me out).

In my next post, I'll demonstrate a task I have to do not-infrequently and how I speed it up a bit with emacs.

Things a computer science degree didn't teach me about software engineering

"Whatever you say, Slick, but I need to tell you something about all your skillz..."

While I'm very happy with / grateful for my computer science degree, I eventually became a software engineer. And the thing about software engineering is that it's related to, but quite distinct from, computer science. Following up on my previous post about that topic, here's the topics and skills I wasn't taught in my computer science undergrad that proved important in my career so far. Along with each one, I'll try and provide the closest thing I remember from undergrad.

Reading other people's code

Almost no professional software engineering is done alone, which means reading other people's code is basically a necessity. And other people's code can be atrocious. While I learned to write well enough to be understood, the skill of fitting my mind around someone else's mental model to understand their code was rarely exercised.

Incidentally, one study has suggested that aptitude to learning other programming languages is related to aptitude in learning natural languages significantly more than aptitude in the mathematics that makes up the core of computer science.

Closest I got: We did some debugging exercises which touched on this space, but the biggest match was becoming a teaching assistant (TA) for one of our courses involving a functional programming language gave me a lot of practical experience in understanding other people's code. I recommend it highly.

Choosing what problems to solve

It turns out, there are infinite problems to be solved with computers, and succeeding in a software engineering career involves a certain "nose" for which ones are the most valuable. Growing this skill can involve both introspection and retrospection, and is generally not a challenge that comes with undergrad, where problems will be handed to you until you complete coursework. Any software engineering career track will eventually expect you, the engineer, to generate the problems that should be solved; those are the engineers that a company needs, because they underwrite the bottom line for the next five years.

Closest I got: I undertook an independent study, but unfortunately it put me off of grad school and PhD pursuit. The gaping maw of the unknown was too much to stare at. But over time, I've been able to learn this skill better on-the-job.

Deciding on a solution, designing it, and defending it

With coursework, there is generally one metric by which your solution is evaluated: passing grade. But engineered software will be maintained by other people and often interlocks with a huge edifice of already-existing technology operating closer to or further away from the end-user. The right way to solve these problems is your responsibility to create, and your design will often require explanation or even defense of its fit vs. alternatives. These aren't software skills; they're people skills and they involve understanding not just the system you're building but the people interested in seeing it built, as well as time and cost tradeoffs.

Closest I got: As a TA, I had an opportunity to design new lessons and put them in front of the professor. This was an excellent opportunity to refine my ability to explain and defend something novel I'd created.

Huge systems

Corporate software accretes features. Once something is built, it is incredibly rare for the company to decide to stop doing whatever the thing is that the software supports.

As a result, becoming a member of a large team at a mature company will often mean struggling with software of absolutely enormous complexity. This isn't useful to do to undergrads; pedagogy is better served by approaching the elephant one piece at a time, and any support systems needed for a class tend to be slim and focused. But big-world software (especially web software) is not designed to be understood by one person; that's why companies hire teams, and it's why reading other people's code becomes so important. A huge system requires some limited expertise in a broad skillset, because eventually an abstraction breaks and, for example, you have to care about the network layer to fix the UI, or the 3D rendering layer to build the schema for a new dataset, or the package management layer to write any code at all. And industry software often has to make practical tradeoffs that more abstract, academic software can skip... Does your system have a time limit in which it can operate, because you're building banking software and your peers turn off on weekends? Is your website breaking because you've hit the limit for max URL size (which is implementation-dependent)? Are you building on a system that lacks a floating-point coprocessor (or, for that matter, a "divide" operator)?

Closest I got: Reading the source code and design notes of big systems. The Linux kernel is open-source. So is Docker. So are multiple implementations of MapReduce. Reading and really trying to understand all parts of one of those elephants can give a person perspective on how wide the problem space can be.

Handling user feedback and debugging user bug reports

Real-world systems have real-world users, and users are the greatest way of finding faults that have ever been devised. Software of sufficient quality to pass academic exercises is often not nearly well-tested enough to be easily maintained or improved in the future, or grow to handle new requirements; it's write-and-forget software. Related to this: academic projects that require code often provide relatively straightforward feedback on problems. User feedback is anything but straightforward (when it isn't outright abusive, because users are emotionally angry that your tool got them 99% of the way to where they wanted to be and then ate their data).

Closest I got: While I actually had a few classes that taught expecting the unexpected, nothing was as educational as building real software that real users use. Whether it's maintaining a system relied upon by other students or building a library used by people on GitHub, getting early experience with the bug-debug loop with strangers in that loop is useful.

Instrumentation

More challenging for a software engineer than vague or angry bug reports is the bug report that never comes, where the user just gets frustrated and stops using your software. Software operating at huge scale can't rely on users to have the time or patience to file reports and must police itself. The practice of instrumenting systems (encompassing both robust error capture and reporting and observation of users' interactions with the tools, what pieces they care about vs. what goes unused, whether they often get halfway through a designed flow and stop, etc.) is as much art as science, but it demands its own attention to design and implementation.

Closest I got: This is a big piece that was basically never touched on in undergrad. Debugging exercises were close in that they could involve temporary instrumentation to understand code flow, but this is a skill that I think may be under-served (unless pedagogy has changed in fifteen-odd years).

Automating toil and transforming big blobs of text

While code isn't text, much code is represented as text and mechanically, a software engineer spends a lot of time moving text around. A significant difference between a stagnant software engineer and a growing one is whether they get better at that process. That can involve both practice and recognizing when a particular task is a "hotspot" that is repetitive drudgery enough that it could itself be automated. There's no rule against writing code to write code; half of software is probably exactly that. I like to rely on tools that make this kind of growth easier, and I'll touch on that in a later blog post.

Closest I got: Computer science exercises tend to try and avoid being the sort of thing that can be solved with moving big blobs of text, trying to instead focus on problems solvable with the right (relatively) few lines in the right place. One exception is an operating system design course I took, and that one gave the most incentive to learn how to quickly replace phrases globally, or move an entire data structure from one form to another (when we the developers didn't expect that change so it wasn't cleanly structured to do it).

Growing your tools

Related to the previous topic: the industry will change over time, and the tools available to you will change with it. Like any good practicing laborer in a technical field, be it medicine, agriculture, or manufacturing, learning of new tools and improving the ones you already have is a continual exercise. The ride doesn't end until you get off.

Closest I got: Everything and nothing. Improving and seeking to improve one's tools has to become like breathing air to a software engineer; this is the "piano, not physics" part of the practice of programming itself. Better IDEs, better languages, best practices, new frameworks, new ideas... Seek them out, and may they stay exciting to you your entire life.