I'm Matthew Davidson

The Thai language doesn’t have a “v” sound. So, many Thai people use the “w” sound as a “v” substitute when pronouncing English words with “v”.

The thing is, they’re not close. “V” is formed by placing the lower lip against the upper teeth. “W” involves making a circle with the lips, then relaxing them. What’s confusing is, the Thai language does have an “f” found, which is much closer to the English “v”. (Try them all out in your mouth to feel the similarity.)

I’ve seen a variety of explanations online that make sense historically, but not linguistically (at least, not to this amateur). They range from: RTGS transliteration of Indian/Pali/Sanskrit-origin words that involve “v”, Chinese diaspora pronunciation of a combined w/v sound, regional dialects, and just plain “we were taught that w and v are the same”.

So why would a less-similar consonant be chosen over a more-similar one? Back down the linguistic rabbit hole…

Nix

I started using nix last year when consulting for MIT. The lab had the perennial Python packaging problem of distributing their work to others, and used nix to solve it.

For those who aren’t familiar, nix builds a hashed tree of every input to building a program. It knows exactly what versions of everything to use, and neatly avoids almost every version resolution problem ever.

I’d heard of nix but never tried it. I got set up with nix-darwin, and ran immediately into issues of poor and outdated docs, and a community kind of split on whether to recommend flakes yet.

Another downside for nix-darwin in particular, is nonstandard filesystem locations, which caused issues for any app that just wanted to see things in common locations. This was especially bad for GUI apps, which don’t have a good way to set up with environments, the standard way nix determines which actual programs to run.

The first major casualty was Alfred. I might have muddled through, but the tech support staff were kind of rude, so I switched to Raycast. I don’t like Raycast’s choice to make everything multiple steps, but it works well enough.

Weirdly, even LLMs failed to produce great results. I suspect this is because they’re trained on the existing docs and blog posts, which are already poor and confusing.

I muddled through for almost a year, but any time I wanted to do something even slightly unusual (like overlays), I’d have to dig through the docs and community to figure out how to do it, or use the LLMs and pray.

I could have bit the bullet and become a nix expert, but that just wasn’t what I wanted from a package manager.

And Back Again

Some people have Homebrew issues, but I can’t remember the last time it caused a problem, so I gave up on nix, and switched back to homebrew. (Saving almost 200 GB of disk space in the process.)

Nix-darwin was actually using a Brewfile to install mac GUI apps, so I kept that and expanded it.

My packaging has largely settled on a single command:

alias brew-me-up='brew update && brew upgrade && brew bundle dump --global --force --describe'

This updates Homebrew, upgrades out-of-date packages, and records everything in a global Brewfile for later.

Learning Thai

I’ve been (slowly) learning Thai since moving here. Partly because of time constraints, and partly because it’s a very difficult language for westerners. It has tones, a different alphabet, meaningful vowel durations, and more.

I started with the immediate goal of communicating when ordering food, and so I began with phonetic vocabulary learning. This is when you write something that sounds like Thai, but using Roman characters, e.g., “kohr moo nueng jaan krap” (which means “Can I have a plate of pork?”)

The problem with this is if you don’t get the tones right, you’re highly likely to be misunderstood, since they will sound like different words. The official Royal Thai General System of Transcription (RTGS) is the standard for writing Thai in Roman characters, but it’s not consistently used outside of government signs, it doesn’t cover tones, and depending on your dialect of English, it’s not even the closest-sounding match. For example, the word for island is usually written as “koh”, but the Thai is pronounced more like “gaw” for American English speakers.

Incorporating IPA

Eventually, I got a new tutor, and started to learn to read actual Thai. My new tutor’s materials all used the International Phonetic Alphabet (IPA), which I’d heard of, but never used. I found it a godsend to vowel pronunciation.

See, English is actually one of the harder languages in the world when it comes to its loose relationship between spelling and pronunciation. Some of this is its dual Romance/Germanic heritage, but some of it is just quirkiness. I explained to my tutor that “cupboard” and “cup board” are pronounced differently, but could not for the life of me say why.

For the most part, all Thai vowel sounds are in English (or pretty close), but there’s no consistent way they’re spelled or named. A lot of beginner Thai learning materials use English examples, like “โ = o as in ‘row boat’”. Of course, every time the examples are for UK English pronunciation, it just adds to the confusion.

I hadn’t planned to learn IPA, but I saw the characters in my new study materials, and started to learn them. Within a week, my pronunciation was way better. I hadn’t realized how much extra difficulty the RTGS and other inconsistent systems had been adding to the learning process.

The IPA is not perfect by a long shot. The IPA characters have changed in recent decades, so some older materials may be unfamiliar. The use of rotated letters was no doubt helpful in the typesetting era, but probably hostile to people with dylexia. It has a bit of a older eurocentric bias, since it was created by English and French people a century ago. Plus, even the IPA’s use isn’t consistent, since apparently linguists who have criticisms with it, are prone to altering the pronounciations occasionally to suit their needs. Not to mention it’s difficult to both type, and write by hand.

Helpful IPA/Thai resources

1. https://www.ipachart.com/
2. https://en.m.wikipedia.org/wiki/Help:IPA
3. https://en.wikipedia.org/wiki/Help:IPA/Thai
4. https://thai-alphabet.com/

When we make a change to code, we should only run tests that can be proven to be relevant. Running unrelated tests tells you nothing useful, wastes electricity, and makes your scrum master cry.

So why do we run irrelevant tests to begin with? Our testing habits came from unit tests, where the goal is to make tests so fast, you can rerun them all frequently. But this broke down as test suites got longer and slower.

I have personally seen CI for PRs take up to 120 minutes to complete, and full end-to-end tests so slow they could only be run overnight.

If your tests finish in under 15 minutes, great! …this post isn’t for you, close the tab.

What can we do instead of pulling our hair out?

There’s a solution that deserves to be better-known outside FAANGs: test-impact analysis (TIA). With TIA, you use code and file dependencies to skip tests that cannot be affected by changes in a PR.

TIA is not new. Variants of it are used in Google’s TAP. Jest and Vitest support it. Microsoft coined the term itself, and offers it in Azure DevOps. Thought leaders have thought about it.

How does test-impact analysis work?

You might already have a pitchfork in hand, shouting “It’s unsafe to skip tests!”

But done correctly, TIA will only skip tests it knows are irrelevant. You can trust it as much as you trust your compiler (and some of the TIA methods are the same as your compiler’s).

Automatic

Automatic TIA methods use static code analysis or run-time coverage to figure out dependencies. You don’t have to change anything with your tests for them to work. However, they don’t usually understand non-code changes, so they run all tests to be safe in that case.

1. File dependencies

File dependencies are fast, safe, and simple, but can “overtest”. Not every test in a file may be affected by a change in a depended-on code file. It’s best for codebases that don’t import/export more than they need to (e.g., Python __init__.py files that import everything under the sun.)

2. Program dependency graphs

Program dependency graphs are what compilers use in optimization, but they work for TIA, too. By analyzing data dependencies and control flow, they can match up tests to individual lines of code. They won’t overtest, though they still have to be cautious when non-code files change.

3. Coverage

Coverage can map code dependencies that aren’t amenable to static analysis, like which branch of an if a test always takes.
This comes with a downside, though; to collect coverage info requires a full test run to start. If you want to use TIA in a fresh CI environment, you need to run all tests beforehand, store that coverage data, and share it with CI. (If you don’t, CI has to run the full test suite, defeating the whole purpose.) Coverage methods work well, at the cost of more complexity, and harder CI integration.

Manual

In manual specification, you write out all the dependencies yourself. (This is more common in full build systems like Bazel.) The major advantage is you can use TIA for non-code changes.

But the major disadvantage is that unlike the automatic methods, which can’t miss tests, if you specify the dependencies wrong, you can.

Complementary speed-up methods

TIA is safe, cheap, requires minimal changes, and combines well with other acceleration methods.

1. Test suites

The traditional speed-up approach divides tests into suites. When updating front-end-related code, run only the front-end suite. While simple, this method is coarse and needs manual setup.

Earlier this year, I received a grant from Clojurists Together to modernize Aleph and bring HTTP/2 support to it. These are some of the interesting tidbits of the journey so far.

HTTP/2: Now Twice as Hypertextual!

I began with a deep dive into RFC 9113 and its predecessors, and absorbed everything I could about the new updates for HTTP/2: streams, frames, flow control, server push, prioritization, connection status, pseudo-headers, etc. Some of the underlying changes are handled invisibly by Netty (Aleph’s underlying network lib), like the new binary frames and header compression, but the rest required serious updates to Aleph to get working.

Streams

HTTP is based on the simple concept of: send a request, wait for a response, repeat. Unfortunately, having to wait for the previous response before getting to send the next request is suboptimal. In theory, HTTP/1.1 supports pipelining, where you can fire off multiple requests without waiting for their responses, and eventually get the responses back in the same order. In reality, that breaks intermediate caches and proxies, so it’s rarely enabled. The general solution is to just open multiple TCP connections to the server. Browsers typically limit it to a maximum of six connections per server, putting a cap on parallelism.

HTTP/2 solves this by assigning each request/response pair to a unique stream (not to be confused with Manifold streams), allowing you to tell which frames are related and safely interleave them. Unfortunately, this broke Aleph’s deep assumptions about how things worked.

When you make a request, Aleph returns a Manifold deferred (like a CompletableFuture), but under the hood, it transforms the Ring request map, places it on a requests Manifold stream, then a Manifold consumption callback pulls requests off the stream, transforms them some more, converts for Netty, and then places them on the Netty pipeline.

Each put! on the requests stream is followed by its corresponding take! from the responses stream. This works under the assumptions of HTTP/1 order, but breaks as soon as you have multiple requests in flight with HTTP/2, since an incoming response could be for an out-of-order request.

Fixing this required removing much of the underlying Manifold stream code for HTTP/2 connections.

HTTP Version Negotiation

Supporting multiple HTTP versions requires changes to how how SSL/TLS is used. With just HTTP/1, you can connect to a web server and start transmitting. But RFC 9113 requires that you use TLS’s Application-Layer Protocol Negotiation (ALPN; RFC 7301) with secure HTTP/2.

In essence, the client and server agree on a protocol during the TLS handshake process. This is handy, but it broke Aleph’s setup process, since you can no longer set up the Manifold code and Netty pipeline for a TCP connection in advance.

What about insecure HTTP/2? Aleph doesn’t support it yet, but the spec allows it if you know in advance that the server supports it (i.e., servers you control in your own internal network).

Lies, Damned Lies, and Specifications

I learned a ton of material from the RFCs and design documents, and then I promptly had to unlearn a quarter of it. With evolving specs, there’s a real danger in reading outdated information. It’s not as simple as ignoring old RFCs, either.

I play games pretty slowly these days. Never finished Myst. Still working on Cuphead.

My wife and I played Undertale a couple years after it came out, and loved it.

I started playing Zachtronics’s EXAPUNKS a few years ago, and finally beat it last fall, only to hear that Zachtronics was closing down and moving on (UPDATE: The Zachtronics games are still available, and Zach eventually returned to making games with his new company, Coincidence.) EXAPUNKS is essentially “Assembly language: the game” meets Neuromancer. After a long day of high-level abstraction in your code, sometimes you just want to twiddle some bits.

One of the missions involves hacking a Virtual Boy-like game system, but you can also make freeform games for it, so I made this “demo” of Undertale’s Asriel battle in EXAPUNKS.

For my own amusement, I’m pretending to have “found” it while hopping dimensions; we’ll see if anyone’s fooled.

Enjoy!

Test-Induced Design Damage (TIDD) is not a new concept. DHH of Rails wrote about TIDD back in 2014. This is not even a new concept for Clojure, as Eric Normand wrote a newsletter about it in the past. Unfortunately, Normand’s post didn’t have the impact I’d hoped for, nor did it go into enough detail for me, so I’m going to try and give examples that will help people understand the issue and the trade-offs a bit better.

What is test-induced design damage?

I suggest you read DHH’s post above, but in short, it’s altering code to better support tests at the expense of other aspects of the system. Communities like Test-Driven Development (TDD) have taken it as a priori gospel that better testing is a primary goal, while downplaying the consequences. As they say, software developers know the value of everything and the cost of nothing.

For example, extracting hidden/private/closed-over code so it can be mocked for testing also carries detriments like requiring names (it usually can’t be anonymous any more), cluttering docs, expanding argument lists (since support objects must be passed-in or injected), potential misuse (if users can now directly access/create/use an object when they shouldn’t), and indirection overhead (both mental and code-based).

To be clear: alterations that support testing may have other benefits that justify their use, but this needs to be evaluated on a case-by-case basis, rather than assuming improved testing is a sufficient justification itself.

It’s even possible that complicating your code to support testing actually increases the number of bugs, despite more testing. This is because, all other things being equal, larger codebases have more bugs¹.

The rest of this post will look at changes made solely for testing. I’ll show you some examples of TIDD, in order of increasing complexity.

Examples

Imagine you have some function that takes too long to use in local tests. Maybe it makes a network call that takes a while.

Let’s say you made a plain function:

(defn my-fn 
  []
  (call-prod-endpoint))

;; usage
(my-fn)

You decide you need to mock it for testing, so what are your options?

Redefining via with-redefs or with-redefs-fn

Using with-redefs, you can temporarily replace the root definition for testing without touching the original code at all (alter-var-root can work, too, though it’s more cumbersome to use). This sounds like the perfect way to leave non-testing code clean, right? Eric Normand suggested this in his original newsletter.

(defn my-fn 
  []
  (call-prod-endpoint))

;; test
(deftest redef-ing-my-fn
  (with-redefs [my-fn #(call-mock-endpoint)]
    (is (= (my-fn) some-expected-result))))

Unfortunately, with-redefs requires a lot of care with multi-threaded / lazy code, since the var root definition is changed for all threads for a limited time. Code in other threads that run after the with-redefs ends can easily use an unintended value. Tim Baldridge wrote a long post on how vars work under the hood and why redefs can be tricky, and it’s worth reading before using functions like binding/with-redefs in any context.

You could safely use with-redefs if you can guarantee all of the following:

I promised a friend at a coffee shop I’d write up a list of my thoughts on IDM, aka “Intelligent Dance Music”. It only took a year, but here it is.

The term “Intelligent Dance Music” emerged in the early 90s to describe a lot of the Warp label’s artists. At the time, Warp was aiming for a post-club, at-home kind of music. The idea was that the music had too many beats to be ambient, but was usually too rhythmically complex to actually be danced to. (While many of the artists involved dislike the term, no better term ever stuck. For a hot moment in the late 90’s, it seemed like it would be “electronica”, but that was broader in scope, and didn’t last.)

Many of the artists seem to follow a certain progression: over time they become ever more experimental, and the music becomes more skittery and abstract. I personally love when artists successfully walk the fine line between ambient and dance, hooks and experiments, so I tend to favor most artists’ early/middle work. I personally believe that music should have a visceral component, that becoming too experimental robs it of something in favor of a head-heavy approach, and my reviews reflect that. But that’s just me.

What follows are my personal favorite IDM artists and albums from that era, with random notes and asides.

Aphex Twin

Richard D. James is amazingly talented, but also a bit of a cheeky wanker. He once lived in an old bank vault and drove a tank around town.

• Richard D. James Album - this is the one that kicked it off for me. Growing up in southern Virginia, I’d never heard anything like it. This is still my favorite era of Aphex Twin.
• …I Care Because You Do - the album that preceded Richard D James, and similar in style. (Also check out the unreleased Melodies From Mars if you like this period.)
• Selected Ambient Works 85-92 - his first album is very danceable, since much of it was written when he was a teenage DJ.
• Selected Ambient Works 2 - the album that made him famous - More ambient in tone, but with many interesting experiments that go way beyond Eno’s original ambient concepts.

Aphex Twin had multiple aliases, including AFX and Polygon Window. His own label released a zillion Analog Bubblebath compilations.

Boards of Canada

Despite the name, they’re a couple of Scottish dudes who made music for sitting around the campfire with their art collective. Apparently they were big fans of the Film Board of Canada.

• Music Has the Right to Children - their first album, and their best. They coaxed a warmth rarely heard from electronic instruments.
• Twoism - an early EP off the Skam label, quite good

Autechre

Why these guys weren’t bigger, I’ve never understood. Both Aphex and Boards of Canada achieved breakout success, but not Autechre. The rumor was their name was short for “Audio Technology Research”, but they claimed the name was formed starting with “au” and then bashing the keyboard, which makes more sense, given their song titles.

Last summer, I was doing HackerRank for fun and whiteboard practice, and I came across a nifty little trie exercise. The challenge was to add a list of contacts to a trie, and report on the number of contacts beginning with a list of prefixes.

I used my favorite language at the time, Clojure, and quickly arrived at the correct solution, but many of these coding exercise sites have time constraints, and the idiomatic Clojure was too slow.

What follows is how you take beautiful Clojure, and accelerate it when needed. (NB: Only do this for hot paths, this is not general Clojure style advice.) All the code is available here.

A Clojure Performance Journey

For those who don’t recall, tries are specialized data structures that excel at storing data with common prefixes (e.g., words). Conceptually, it’s a tree, where each node represents part of the prefix, and the complete path to a terminal node represents the data.

For a list of English words, the most straightforward implementation is a tree of nodes, where each node (other than the top) has a letter, a terminal flag to indicate whether the node is the last letter in a word, and an array of 26 pointers to other nodes, representing the alphabet. (Various optimizations exist to compress long chains, but we will focus on this implementation for now.)

In this diagram, you can see a representation of a trie storing the words: a, ale, all, alley, are, art, at, and ate. (Terminal nodes are tinted.)

Solutions

Standard data structures

Here’s the basic implementation. It has functions that add new words, locate the partial subtree with a given prefix, and count the number of words beginning with a prefix. In this example, db is a series of nested hash-maps, and the :* key indicates the node is terminal.

(defn add [db name]
  (update-in db (seq name) (fnil assoc {}) :* true))

(defn count-terminations [db]
  (let [terminations (if (:* db) 1 0)]
    (reduce +
            terminations
            (map count-terminations
                 (vals (dissoc db :*))))))

(defn find-partial [db partial]
  (println
   (if-let [sub-db (get-in db (seq partial))]
     (count-terminations sub-db)
     0)))

This works, but was way too slow.

Switch to eager over lazy evaluation

Clojure defaults to lazy evaluation, which requires a certain amount of overhead. What if we force eager evaluation with transduce instead of reduce?

(defn count-terminations [db]
  (let [terminations (if (:* db) 1 0)]
    (transduce
     (map count-terminations)
     +
     terminations
     (vals db))))

That shaves off a few seconds, but still not good enough.

Switch to a record

Alright, well, what about using a record with named fields and cache the default empty node?

I’ve been going through the Cryptopals challenges recently. For those who don’t know, they’re a series of exercises based on real-life cryptography breaks, and they’re lots of fun. You can see my code here.

Clojure is one of my favorite languages, so I decided to tackle them with it. Here are some thoughts on the first 3 sets, plus a few issues I ran into along the way. I hope this helps anyone else going through them without giving too much away.

General challenge notes

Challenge 3 - Single-byte XOR cipher

When doing frequency analysis, the instructions make no mention of the chi-squared statistic, though that’s probably a good starting point, and generalizes beyond alphabetic frequencies to any expected distribution. You can use it later when detecting ECB vs CBC, since you can use it to detect deviations from uniformity.

Challenge 7 - AES in ECB mode

The instructions do mention this, but after chatting with someone else doing the challenges, realize that you do not need to write AES yourself. Just use a default implementation for your language. You only need the single block transform (ECB), which you’ll use to build on top of. For Java, all you need is javax.crypto.Cipher and SecretKeySpec, using “AES/ECB/NoPadding”.

Challenge 9 - Implement PKCS#7 padding

When implementing PKCS#7 padding, the instructions make no mention of what to do when your data lines up perfectly on a block boundary. Only later, when doing the padding oracle challenge, did I realize you have to add an entire padding block (sixteen 16’s) in that case.

Challenge 17 - The CBC padding oracle

When altering the first byte of a block to see if it produces valid padding, you’re aiming to alter it such that the padding ends in a single 1 byte. But remember, if you test the original, unaltered byte, you will also get valid PKCS#7 padding (since it was correct to begin with), which is probably not 1.

Challenge 18 - Implement CTR, the stream cipher mode

Look carefully at the examples. At first glance it looks as if you’re XORing a bunch of zeros, but the ninth byte is actually incrementing.

Challenge 23 - Clone an MT19937 RNG from its output

Inverting the Mersenne Twister’s temper function has a computationally simple version. It’s not necessary to use brute force, inverse matrix multiplication, or a constraint solver. It can be done with just bit operations. The key insight is that XORing an integer with a shifted version of itself results in many of the bits in the output being identical (since the shifted-in bits are 0). You can then shift those bits and XOR again to obtain more bits, repeating until you have the whole thing.

Clojure / Java-specific issues

Poor primitive support in JVM and Clojure

The bread and butter of cryptography is byte-level manipulation. Unfortunately, the JVM treats bytes as second-class citizens (e.g., many methods on Integer/Long have no Byte counterpart), and lacks support for unsigned integer types. On top of this, Clojure defaults to 64-bit longs everywhere, necessitating a lot of conversion. I ended up writing many conversion routines just for basic support, and leaving integers in the default long type for compatibility.