Wesley Aptekar-Cassels

End-to-end testing emails

Fri, 16 Feb 2024 00:00:00 -0500

I came up with a really slick trick to write E2E tests that deal with sending/receiving emails recently. This is the sort of thing that seems like it's probably usually sort of a nightmare — I wanted to write a test for registering a account on a website, where part of the flow was clicking on a validation link in a email.

The slick trick is a test SMTP server which is also a HTTP server. It's 39 lines of Javascript and extraordinarily simple:

const SMTPServer = require("smtp-server").SMTPServer;
const http = require("http");

let emails = {};

new SMTPServer({
  authOptional: true,
  onData: (stream, session, callback) => {
    const chunks = [];
    stream.on('data', (chunk) => chunks.push(Buffer.from(chunk)));
    stream.on('end', () => {
      const data = Buffer.concat(chunks).toString('utf8');
      console.log(data);
      const email = session.envelope.rcptTo[0].address;
      if (email in emails) {
        emails[email].push(data);
      } else {
        emails[email] = [data];
      }
      callback();
    });
  }
}).listen(2525);

http.createServer(async (req, res) => {
  const email = req.url.slice(1);
  if (!(email in emails)) {
    emails[email] = [];
  }

  res.writeHead(200, {"Content-Type": "text/plain"});

  const initial_num_emails = emails[email].length;
  while (emails[email].length == initial_num_emails) {
    await new Promise(resolve => setTimeout(resolve, 10));
  }

  res.end(emails[email][initial_num_emails]);
}).listen(2424);

This allows you to make a request to http://localhost:2424/test@example.com, which will block until the SMTP server on port 2525 receives a email to test@example.com, at which point it'll return the full text of the email.

This allows for writing a super simple E2E tests — here's my helper function to register a new user:

async function registerAndLogin(page) {
  await page.goto('/register');
  const email = `${randomUUID()}@example.com`;
  const password = randomUUID();
  await page.getByPlaceholder('Email').fill(email);
  await page.getByPlaceholder('Password').fill(password);
  const verification_email = fetch(`http://localhost:2424/${email}`);
  await page.getByRole('button', { name: 'Sign Up' }).press('Enter');
  const verification_email_response = await verification_email;
  const verification_email_text = await verification_email_response.text();
  const verification_link = /(https?:\/\/[^\s]+)/gm.exec(verification_email_text)[1];
  await page.goto(verification_link);
  return {
    email: email,
    password: password
  };
}

Note the nice use of async/await to make the fetch request before pressing the "Sign Up" button.

As soon as I came up with this I immediately started showing it to everyone — my friend said "I feel like I just learned a new skateboard trick" — excellent praise, in my book.

As a sidenote, if you haven't used Playwright for E2E tests, you're missing out. It's my opinion that the existence of Playwright moves E2E testing of webapps from being something that's usually tedious and flaky but sometimes worth the costs to something that's viable as a primary testing strategy.

Unit tests are best when code is decoupled and complex, but web code tends to be simple but tightly connected. This makes E2E testing a extremely useful strategy, and Playwright makes it simple to write non-flaky E2E tests with a small amount of effort — check out the codegen tool and the trace viewer if you haven't.

If you want E2E tests to be your main tests, it does require some somewhat careful planning — since I use SQLite and don't rely on any cloud tools, I can have tests that run extremely quickly (and even in parallel), without relying on any external resources, which is key for this kind of strategy. If you can swing it, though, it's pretty nice.

Motivating X3DH

Sat, 13 Jan 2024 00:00:00 -0500

Signal uses a protocol called "Extended Triple Diffie-Hellman" (X3DH) for key agreement¹. This essentially takes standard Diffie-Hellman key exchange, and enables it to be used where one party may be offline while they're receiving a message, while providing forward secrecy.

While the paper clearly explains how to implement X3DH, the why is slightly less clear. In this post, I'll briefly explain how X3DH works, then investigate what security property each of the steps involved is aimed at providing, and why particular choices were made around the signature algorithm.

A quick summary of X3DH

(This summary requires a understanding of Diffie-Hellman key exchange — if you are unfamiliar or need a refresher, take a look at this footnote²)

In X3DH, the receiving party (who may be offline when they receive a message) uploads the following Curve2519 public keys to a server:

A identity key (IK_r)
A "signed prekey" (SPK)
A signature for SPK with IK_r
A number of "one-time prekeys" (OPK1, OPK2, …)

When the sending party wants to send a message, they download from the server IK, SPK, the signature for SPK, and one of the one-time prekeys (which the server deletes and does not use for future runs).

The sender has two keys:

A long-term identity key (IK_s)
A ephemeral key used only for a single X3DH handshake (EK)

After validating the SPK signature is correct, the sender can generate a shared secret by invoking the Diffie-Hellman function four times:

DH(IK_s, SPK)
DH(EK, IK_r)
DH(EK, SPK)
DH(EK, OPKn)

The outputs of these functions are concatenated and run through a key derivation function to generate a shared secret. If the server has run out of OPKs, only the first three functions are performed, to the slight detriment of the forward-secrecy properties. This image summarizes the Diffie-Hellman functions that are preformed:

Why, though?

This is a lot of stuff, and it's understandable to ask what it's all doing. There are two basic properties that we want our key exchange to provide:

The first is "forward secrecy" — that is, if the private keys are compromised after the X3DH handshake has been completed, a attacker will not be able to decrypt the message. This is why we need the ephemeral key, and why we don't just do standard Diffie-Hellman with the identity keys and call it a day.

The second is "mutual authentication" — that is, both of the parties know the identity of the other party that they're talking to, and know that they are not being impersonated. This is why we want long-term identity keys — it's much simpler to prove ownership of a single long-term key than it is one of many ephemeral keys.

So, we want to use long-term keys for mutual authentication, and ephemeral keys for forward secrecy, and we can understand the one-time prekey as the ephemeral key for the receiving side. What's the signed prekey doing, then? It primarily serves as a backup, in case the server runs out of one-time prekeys. The signed prekey is replaced every time the client uploads new one-time prekeys, and thus is is more ephemeral than the identity key (which hangs around indefinitely), but not so ephemeral that we might run out.

So, that's the explanation for why we have each of these keys. Once we have that, it's actually pretty clear why we do the set of Diffie-Hellman operations we do. If we look at the case without a one-time prekey, it's very clear:

We need to do a exchange between the ephemeral keys (EK and SPK), in order to get forward secrecy.
We need to involve IK_s and IK_r, to provide mutual authentication.
We don't want to use DH(IK_s, IK_r), because doing so does not provide any forward secrecy, and not doing so provides better deniability (since anyone could forge a convincing looking transcript of a X3DH session without knowing either of the identity private keys, by generating all the ephemeral keys themselves)

With those constraints, we end up with the only thing we can have:

However, the case where we do have a one-time prekey offers us more options. In particular, we could swap the roles of SPK and OPK, giving use either of these:

Signal chooses to take the somewhat simpler approach of keeping the role of SPK the same regardless of whether we have a OPK, just adding in DH(EK, OPK) in the case where OPK is available. While this has slightly worse forward secrecy properties on its own (the attacker can crack more of the DH outputs if they capture the SPK private key), this doesn't really matter, because the DH outputs are fed through a KDF anyways, which necessitates cracking all the DH outputs to capture the key³.

On signatures

One thing we haven't talked about is the reason that the signature of SPK is needed. Here's the attack that's intended to prevent:

Alice tries to send a message to Bob.
The server replies with Bob's IK, but values of SPK and OPK that it generated for itself, and thus knows the private keys for.
Alice sends a message. The server doesn't relay this to Bob (since he wouldn't be able to decrypt it), but instead holds on to it.
Later, the attacker compromises Bob's IK private key. Now they can calculate the shared secret, and read the initial message.

Signing the SPK is a sensible way to prevent this. However, there are a few questions to ask about the particular way in which it's implemented in X3DH. Firstly: is it safe? Reusing keys for multiple purposes is can lead to disastrous results, and it's important to rigorously prove that any such construction is safe. My understanding is that most cryptographers believed this construction to be safe, but it was only relatively recently that a actual proof for these specific algorithms was provided (and even then, I'm not entirely sure that the construction in that paper applies to X3DH, although I'd be quite surprised if it didn't).

The second question relates to how one even goes about using the same keypair for Diffie-Hellman and EdDSA signing. The standard approach (other than "don't do that", which is generally recommended) is to take a EdDSA key (which is a point on a Twisted Edwards curve) and convert it into X25519 key (which is a point on a Montgomery curve). This is convenient, since a point on a Twisted Edwards curve maps to a single point on Montgomery curve, but notably, a point on a Montgomery curve maps to two points on a Twisted Edwards curve⁴.

Signal chooses to solve this by defining a scheme called XEdDSA, which specifies which of the two possible points to use.

One has to ask, though, why go through the effort, when you could just start with a Twisted Edwards curve? My guess is that the signature step wasn't added until Signal was already being deployed, so using Montgomery curves for identity would be a backwards incompatible change. My understanding it that Matrix uses Twisted Edwards curves from the start in their implementation of X3DH, which I think is quite sensible, given that they had the luxury of designing their protocol with Signal's hindsight.

This still raises the question of why they chose to use the same key at all — it's easy to imagine a system where the identity key is exclusively used for signing, and a separate (signed) key-exchange key is used. Given recent proofs of the security of composing EdDSA and X25519, I don't think there's any advantage to doing so at this point, and Signal probably didn't go down this path for the same reason that they needed to invent XEdDSA in the first place — they already had a bunch of Montgomery keys deployed, and didn't want to switch them out for Twisted Edwards keys.

Well, they now use a post-quantum secure version, but it's a similar idea. ↩
Diffie-Hellman gives us a function DH(private, public) that takes a private key and a public key, and generates a secret value from them. Notably, two parties can each run the function with their own private key and the other party's public key, and they will both get the same value. ↩
Ideally don't be like SimpleX and accidentally forget to apply the KDF. ↩
This is what lead to a fun vulnerability in the Monocypher library, where giving a all-zero signature would be considered valid for 50% of messages due to a incorrect internal conversion between the two forms. ↩

If you think Bluesky is decentralized, you’ve been conned

Fri, 28 Apr 2023 00:00:00 -0400

Bluesky is a purportedly-decentralized social media company that spun out of Twitter, beginning in 2019. It makes sense that people are hunting for new social media platforms right now — Elon Musk's disastrous takeover of Twitter has demonstrated how fragile relying on private VC-funded companies is for something so critical as communication infrastructure. BlueSky's answer to this is that they are decentralized — thanks to the AT Protocol, anyone can run a server. Unfortunately, it's decentralized in name only:

The AT Protocol is entirely controlled by Bluesky the company, with no community governance structure or plans for a governance structure. Want to change how it works? Tough luck, buddy.
~~Want to run a server yourself? Well, the server is closed-source, so you'd better enjoy reimplementing it all yourself~~.¹ (Edit: I'm told that this is not true, and that the atproto has the server code. However, it's unclear to me how much of what's running on bsky.social is public — the identity parts appear to be, but not the web frontend, that I can find.)
The apps? Also closed source, you don't have any control over them.²

This isn't how you build a decentralized ecosystem, it's how you build a system that you can claim is decentralized while still holding all the power yourself.

There's a lot more that's wrong with Bluesky that I'm not going to get into, since I honestly don't really care. But if you care about not giving more power in your life to a VC-funded company started by a guy who said that "Elon is the singular solution I trust" to running Twitter, you probably want to stay away from Bluesky.

You might reasonably be fooled into thinking that indigo is the server, but a closer look reveals that this does not seem to be the case. ↩
There does appear to be one open-source Android app from a third-party, but there doesn't seem to be any plan to open-source the official apps. ↩

Storage Space PSA for Mastodon Admins

Sat, 12 Nov 2022 00:00:00 -0500

Do you administer a Mastodon instance? Is it falling over all the time now because it's running out of disk space because everyone is joining the network? Here is a quick PSA: tootctl media remove does not remove all the remote media. It removes all remote media more than 7 days old. It used to be that, on the server that I run, recurse.social, that would leave a handful of GB of media. Now it's more like 25GB of media. If you don't want to have to resize your server to deal with the peak of the influx of new users, consider changing your cronjob to something like tootctl media remove --days=1, to more aggressively delete remote media that's unlikely to be needed.

Subdomains as identity

Thu, 07 Apr 2022 00:00:00 -0400

A common identity scheme for websites is to give users pages like example.com/username. Twitter, Instagram, Twitch, Pinterest, and many other websites use this model. Reddit, TikTok, Mastodon, Lobsters, and several others use a slightly more conservative version of this, with a prefix (either @ or u/) before the username.

I think that in many cases, this is a mistake, and using subdomains, like username.example.com is better. Particularly, I think that the subdomain-based model is best for websites where there is minimal interaction between users (blogging, publishing, etc), or where that interaction happens via an open protocol.

Using subdomains for identity offers a few advantages:

It makes it easier for users to own their own identities, since you can very easily support using custom domains
It makes it easy to securely allow users full control over the CSS/HTML on their page, without worrying about XSS
It lets you cleanly have information about your website on the top-level domain, without adding the noise of a u/ or @ prefix to usernames

This would be most advantageous for systems like Mastodon — one of the most fundamental mistakes that mastodon made was tying your identity to whoever happens to sysadmin the server you use, and using domains as identity would have almost entirely solved that¹. However, I think that there are notable advantages even for less explicitly federated systems as well.

I personally think that these advantages are quite significant, and software that gives users control over their identity and expression is inherently worth building, but this blog post isn't about that. I figure if you're making this decision, you're capable of thinking through the tradeoffs yourself. This blog post is about concretely, how to implement a system that uses subdomains as identity, and how to extend that system to allow arbitrary custom domains. This is what I do for thoughts.page, which has been running successfully like this for a year now.

The first question is how to integrate subdomain information into whatever routing system you use. thoughts.page uses warp, which has out-of-the-box support for routing based on domain name, but you might not be so lucky in your framework of choice — searching "[framework] wildcard subdomain routing" should point you to relevant documentation. If you don't find a easy way to do this, a common alternative is to rewrite URLs internally, so that username.example.com/foo gets rewritten to example.com/user/username/foo before it's handed to your routing library. If your framework doesn't provide a way to do it, nginx can do the same thing with a rewrite command. However, I'd recommend doing it in your app if possible, since that will make development significantly easier.

Speaking of development, you won't be able to just use localhost to see your server anymore, since it will require subdomains for routing. Some systems will allow URLs like username.whatever.localhost to work, but if not, you can use lvh.me or vcap.me, which both return 127.0.0.1 for all requests.

When you deploy your app, you'll need to get a wildcard HTTPS certificate, instead of one that's restricted to a particular subdomain. If you use Let's Encrypt, this makes the process slightly more complicated, but there are good instructions online for doing so.

That's all you need to use subdomains for identity. If you want to go the whole hog and allow users to use custom domains, though, you need to do a little bit more. The main problem here is HTTPS: you need to have a certificate for whatever domain they choose to use, and your web server needs to be able to dynamically pick which HTTPS cert to use based on the domain name. Nginx added support for this in 1.15.9, which is relatively recent — it's in the most recent Debian version as of this writing (bullseye), but not in the previous version (buster). You can set this up by using the $ssl_server_name variable in your ssl_certificate and ssl_certificate_key variables, like so (assuming the default certbot setup):

ssl_certificate /etc/letsencrypt/live/$ssl_server_name/fullchain.pem; 
ssl_certificate_key /etc/letsencrypt/live/$ssl_server_name/privkey.pem;

That's not enough, though, since you also need to actually get the keys. I do this by having a small script² that reads the database to look for users with custom domains, and attempts to renew their certificates:

#!/usr/bin/env bash

cd $(dirname "$0")

certbot certonly --noninteractive --keep-until-expiring --dns-digitalocean --dns-digitalocean-credentials /root/digitalocean-secret.ini -d thoughts.page -d '*.thoughts.page'

sqlite3 thoughts.sqlite3 "SELECT domain FROM domains" | grep -v ".thoughts.page$" | while read DOMAIN
do
    certbot certonly --noninteractive --keep-until-expiring --webroot --webroot-path /var/cert-webroot/ --domain "$DOMAIN"
done

# Needed because nginx SNI doesn't run as root
chmod -R 755 /etc/letsencrypt/live/
chmod -R 755 /etc/letsencrypt/archive/

# Nginx needs a reload to pick up new certs
systemctl reload nginx

This is run every five minutes by a cron job. I could have the user adding a new custom domain trigger this, but so far, I haven't felt the need to do that.

The /var/cert-webroot/ directory referenced in that script is served by nginx, using the following snippet:

location /.well-known/acme-challenge/ {
    root /var/cert-webroot/;
}

This allows the app server to keep on running while the certs are being renewed, free of any interference from certbot.

This relatively simple change makes it easy for people to own their own identities if that's important to them, without imposing any sort of burden on less technically-inclined users to think about it. It also creates a softer gradient towards self-hosting, allowing people who are comfortable setting up a DNS record but not administering a server to own their own identities. Next time you're making a website with users, I hope you'll consider using subdomains as identity.

I say "almost entirely", since there should still ideally be some way for clients to have cryptographic proof of their identity, which they can use to delegate the ability to manage their account to a particular server. Implementing this well is quite complex, but I think doable and important, as well. If you're working on this kind of thing and want to chat about it, please do drop me a line :) ↩
One of the many advantages to using SQLite is that it makes writing small scripts like this really easy ↩

Minify and Gzip

Tue, 22 Feb 2022 00:00:00 -0500

I've been doing a bit of work cleaning up the CSS on this blog in the past couple days — the CSS is from a template that I copied more than five years ago at this point, and it was pretty crufty even when I first got it.

When I was poking around, I noticed that I had two separate media queries to check if the browser width was something that looked like a mobile device. This makes the CSS nicer to read, since the overrides can be right next to the things they're overriding, but it also makes the resulting CSS slightly larger. I minify my CSS using tdewolff/minify during the build, so I checked to see if it would collapse the two media query blocks into one. Doing so would be a somewhat complicated optimization, since the order that CSS rules are defined in can sometimes matter, but it wouldn't be too tricky. I made a quick test file:

.foo {
    color: red;
}

@media all and (max-width: 767px) {
    .foo {
        color: blue;
    }
}

.bar {
    color: green;
}

@media all and (max-width: 767px) {
    .bar {
        color: blue;
    }
}

And checked what it minified to on the latest release:

.foo{color:red}@media all and (max-width:767px){.foo{color:blue}}.bar{color:green}@media all and (max-width:767px){.bar{color:blue}}

Two media queries — how inefficient!

But then I thought to check — my website is served using gzip compression, so I shouldn't be checking the size of the minified file, I should instead be checking the size of the minified and gzipped file. The gzipped file that minify generated was 107 bytes, and when I applied my "optimization", it grew to 115 bytes. Huh.

However, compression is an area where the efficiency can vary wildly depending on filesize. I figured it might just be that the sample file wasn't large enough to have much structure for gzip to exploit in the first place. As a second test, I grabbed the CSS file for my blog, which weighed in at 1858 bytes gzipped, applied the optimization to it, and got… 1864 bytes.

This is a pretty common thing if you try to do a lot of filesize micro-optimization on the web — there are built-in ways to make code smaller that are designed for humans to use to make code more understandable, but those tools are often less efficient that letting gzip (or brotli, or whatever other compression you use) operate directly on the larger version. If you care about reducing the size of the network transfer, you need to actually measure the network transfer, and not anything else.

This same principle holds true for any sort of optimization work — if you measure a proxy for what you want to optimize, you need to be sure that the proxy actually correctly correlates with the thing you're trying to optimize! And once you've been bitten by this problem enough times, you end up realizing that it's often less work and pain to just measure the thing you care about directly.

To my knowledge there aren't any minifiers that operate with awareness of what compression algorithm the final file will be served with — I think that'd be a interesting area to explore. It's unlikely to actually be "worth it" in any reasonable cost/benefit analysis, but hey, I don't micro-optimize my personal websites because there's any benefit to it, I really just do it for the principle of the thing.

How to reset the USB stack on Linux

Fri, 18 Feb 2022 00:00:00 -0500

I've recently started having issues with the USB stack on my Linux laptop getting itself wedged. Rebooting fixes the problem, but I'd rather not have to reboot, so I looked into ways to reset the USB stack without needing a reboot. Almost all of the answers on the internet about how to do so either do not work on modern Linux systems, or did not reset at low enough of a level to fix the specific problem I was having.

The solution is to unbind and rebind the USB controller to the xHCI driver. First, find the PCI bus number of the misbehaving USB controller, using lspci -D. Then, write that bus number to /sys/bus/pci/drivers/xhci_hcd/unbind, for instance, by running echo 0000:00:14.0 | sudo tee /sys/bus/pci/drivers/xhci_hcd/unbind (replacing 0000:00:14.0 with whatever the PCI bus number you found in the previous step was). This will disable the USB controller. Once you've done that, reenable it by writing the same bus number to /sys/bus/pci/drivers/xhci_hcd/bind.

Here's a script that will do that all for you, in case it's useful:

#!/usr/bin/env bash

set -eo pipefail

DEVICE=$(lspci -Dm | grep "USB controller" | cut -f1 -d' ' | head -n1)

echo $DEVICE > /sys/bus/pci/drivers/xhci_hcd/unbind
echo $DEVICE > /sys/bus/pci/drivers/xhci_hcd/bind

You may need to modify the grep for "USB controller", depending on how your USB controller identifies itself.

SNES Development Part 6: Sprites

Thu, 10 Feb 2022 00:00:00 -0500

This is part six of my series on SNES development. You may want to start with the first post.

In the sixth, and likely final segment of this SNES tutorial, we're going to draw a sprite (or rather, an "object") on the screen! Once we're through with this, you should have basically everything you need to go off into the world and make SNES games all on your own :)

As usual, I recommend the Retro Game Mechanics Explained video on Objects as prerequisite background material.

We're going to allocate some space to save a copy of our OAM, which we'll DMA to the actual OAM memory every frame. We'll need to add a new palette for the sprites, and some new tiles. Then, we'll set sprite zero to use our tile and palette, and set its X and Y position. We'll make it a 16x16 sprite, because that's more interesting :)

You can find the code and diff in the usual places.

The code for this isn't too tricky. The first step is to allocate some space for a copy of our OAM table in the BSS segment. The BSS segment is just a place to put variables, similar to the zero page that we used previously. (We can't use the zero page for our copy of the OAM, since it's not large enough). Unlike code that's running in an operating system, our BSS isn't going to be set to zero before we get out hands on it, we need to add a loop to zero it out ourselves.

We need to add some new palettes, since sprites use a separate set of palettes from tiles. The sprite palettes start at CGRAM index 128. I add a black, which is unusable (since the 0th palette color on sprites is treated as transparent) and a red for the foreground color. You should already have some system of your own that's nice than this for setting up palettes, so feel free to use that instead. Mesen-S comes with a nice palette viewer tool if you want to check your work.

We also set the OBSEL register, which controls the object size options (we choose 8x8 for small sprites and 16x16 for large sprites) and some offsets for selection which tiles to use, which we leave at zero for simplicity. Leaving the offsets at zero means that the backgrounds and the sprites will use the same set of tiles. One thing to note is that if you're using a background mode that has a number of colors other than 4bpp, this'll get a little bit messy. I recommend using Mode 1 to start with, which sets BG1 and BG2 to 4bpp, but the demo code is running in Mode 0, which is 2bpp. It's not a big problem to mix them, it's just something you'll need to be thinking about when indexing into the tiles.

After that, the only remaining setup is setting the bit on the TM register that enables sprite rendering.

With the setup out of the way, we can write our sprite data into the OAM on each frame, and use DMA to copy our OAM data to the OAM. You can find documentation for the OAM format here, although the bits that you need to know to immediately start hacking are commented in the code.

Once you have the OAM integrated into your code, you basically have enough to start making a game! You can hook up the sprite so you can move it with the controller, start adding some sprites for enemies, making maps with tiles, and really doing whatever you want!

SNES Development Part 5: DMA

Thu, 10 Feb 2022 00:00:00 -0500

This is part five of my series on SNES development. You may want to start with the first post.

In part three, I mentioned that in the future, we'd switch from using a loop to copy data into VRAM to using DMA. Well, the future is now!

Using DMA is much faster than using a loop, which makes it easier to load large amounts of data from ROM to memory, whether that's graphics, sprite data, palettes, levels, or anything else your heart desires. It'll be useful in the next part, where we'll want to quickly send the sprite data to the SNES.

As usual, I recommend watching the Retro Game Mechanics Explained video on DMA & HDMA first. You can stop watching once you get to the HDMA part if you'd like — only the DMA section is needed to understand this part.

The code for this part might seem a little scary, but once you have a solid understanding of what DMA is and how it works, it's not too bad. You can find the code and diff on GitHub as usual. This time, we'll walk through each step of the diff, since it's small but dense.

lda #%00000001
sta DMAP0

This sets the DMA flags. The first bit is a zero, to indicate that we're transferring from the CPU memory to the PPU memory. The next two bits are unused. The next two bits are zero to indicate that we want the address we're writing to to auto-increment (we could instead choose to auto decrement, or use a fixed address). The final three bits, 001, indicate that we want to transfer two bytes at a time.

lda #<VMDATAL
sta BBAD0

This tells the SNES that we want this DMA to write to the VMDATAL register, and since we set the mode to 001, it'll also write to VMDATAL + 1, which is VMDATAH.

ldx #.loword(charset)
stx A1T0L
lda #^charset
sta A1B0

These lines set the starting address to charset, which is our label for the location of the charset data. The .loword and ^ are our way of telling the assembler to just pull the bits that we want — you can read more about them in the ca65 manual if you'd like. Note that we write to A1T0L with the X register since it's in 16 bit mode — this lets us write to the A1T0L and A1T0H registers at the same time.

ldx #(charset_end - charset)
stx DAS0L

Here, we set the amount of data we want to transfer.

lda #1
sta MDMAEN

And finally, we kick off the DMA!

This has been some more anticlimactic infrastructure work — as with the previous time we did this, everything will look exactly the same when you run it, but if you look in the Mesen-S event viewer, you should be able to see the change :)

This time, though, there's some stuff you can do on your own:

Write a macro to set up and run a DMA transfer
Use your shiny new DMA transfer to write some tiles to VRAM_BG1
Replace your manual palette setting code with DMA

Once you've done all that, it's time to move on to making some sprites in part six!

SNES Development Part 4: NMIs and vblank

Thu, 10 Feb 2022 00:00:00 -0500

This is part four of my series on SNES development. You may want to start with the first post.

Last time, we started reading controller inputs! However, we weren't able to get a tile to move around on the screen in response to the controller, because our code was running too fast. In this part, we'll look at how to slow down our main code so that it runs once per frame. We'll do this by modifying our NMI handler, which is called whenever the "vblank" period starts, once per frame. For context on what vblank is and how it works on the SNES, I recommend this video. This post will assume that you've watched it, or found the information in it from some other source.

We already have a NMI routine, which we set up in the very first part of this series. What we want to do is to make sure our code runs once per NMI. The most obvious way to do this is to just take the code we have and stick it in the NMI routine, but this isn't a good idea — doing so can lead to strange behaviour if your NMI handler takes to long, and a second NMI gets fired while you're inside it, among other problems.

Instead, we can use the wai instruction, which pauses execution until a interrupt is fired. However, there are more types of interrupts than just NMIs, so what we'll do is keep a count of the number of NMIs, which we increment in the NMI handler, and then have our main code check to see if the NMI count changed whenever it wakes up from the wai instruction.

We implement this NMI count variable by reserving some space in the zero page — a small, 256-byte section of memory that's fast to access an can be accessed from anywhere. In the future, you'll likely want to keep a few other global variables in the zero page, like the player's position, etc.

With that in mind, the code and diff should be pretty self explanatory! The only extra bit is that we need to tell the SNES to send us NMIs by flipping a bit in the NMITIMEN register.

...how do we know that did anything, though? Everything looks exactly the same!

This is a great time to introduce another useful tool in Mesen-S: the event viewer. It provides a visualization of what type of code is running, and where in the frame it's running. Open up "Debug → Event Viewer", and compare what it looks like before and after the change. It should be pretty obvious what the difference is :)

This wasn't that exciting of a section, but it unlocks a bunch of exciting things! On your own, try making a tile that you can move around with the controller — at this point, you should have all the tools you need!

Once you get that working, we'll return to some boring infrastructure by replacing our slow loop to copy data into VRAM with some fast, sexy DMA in part five.