Watch on YouTube

Spreadsheets are magical tools that can not only help you organize your finances, but they can also help you prototype video game logic. No really-- they can. That is what I spent a good chunk of my time this past iteration doing: modelling the new part of the base game and balancing it. All done without a line of code or even opening a code editor.

I knew that spreadsheets were the glue that kept software projects together, but never realized how they would be so much more critical in video game development projects.

The more time I spend making a video game, the more I have come to realize that video games start with spreadsheets. Not code. Not pictures. But spreadsheets.

Spreadsheets for funding. Spreadsheets for balancing. Spreadsheets for localization. Spreadsheets, spreadsheets, and more spreadsheets.

Working in enterprise software, I am all too familiar with spreadsheets for managing projects, organizing teams, and all sorts of business activities.

They are the glue or duct tape that hold the project together, in a lot of cases.

For video games, I keep finding myself and the various experts I learn from using spreadsheets to build the video game itself, not just organize the business activities around it.

I'm sure there are more examples than I know about, but in this post I want to share where spreadsheets seem to make sense as the default tool to use.

If you have other examples, reach out to me on Bluesky, LinkedIn, or YouTube and tell me how you use spreadsheets to make your video game. I am genuinely interested in learning where else they fit into the development process.

Text Localization

I have only dabbled in this one so far, but as I experimented with various dialogue systems, I saw how any dialogue or text needs to fit into a spreadsheet at some point to be localized.

A spreadsheet is a table, plain and simple. Using it to store data and then mapping that data to other data just makes logical sense.

This is why when I look at tools to help me with dialogue or text in my game, I make sure that they support export to a spreadsheet-friendly format like CSV.

Gameplay Models and Balancing

This one is a little more involved. I heard Jesse Schell on a podcast talking about how he had used a spreadsheet to determine the balance of a trivia game he was making.

I figure if you can create a model to simulate trivia gameplay, you must be able to do it for other systems.

And so, with my current project Passenger, I have spent time modelling out how to crunch the numbers in a new part of the game I am trying to prototype.

After a few evenings of fiddling, I managed to get a spreadsheet that showed me how the formulas balanced out and what variables I had to adjust to make things easier or harder for the player.

Let me emphasize what this has done for my project-- before adding more content to my game, I know how the gameplay system will be balanced. I don't think I know; I actually know how it will be balanced.

That doesn't mean the work is actually done. I mean, I still have to make it fun and all that. But I have a formula that I can adjust to make things easier (or harder) depending on player feedback and the feeling I am going for in the scene.

Understanding Your Audience

In my last devlog, I talked about documentation helping me define scope for Passenger by doing a competitive analysis on games I thought seemed comparable to the vision I have for Passenger.

This is all done in spreadsheets.

With the competitive analysis, I pulled all the tags for the comparable games and started crunching the data to figure out what tags aligned with my vision and what tags appeared more frequently with the comparable games. The overlap between these two sets of tags gives me the features I should include in my game.

Example

For example, you can see that Dating Sim is one of the tags I was considering for the game-- however, after doing some analysis on the comparables, I can see that it doesn't show up nearly as much as I thought it would.

This means that in terms of prototyping priority, the dating sim mechanics have dropped pretty low on my development schedule.

This saves me the effort of trying to figure out if that even works, as the audience I am trying to reach with Passenger doesn't really care about dating sim mechanics as compared to other features.

Conclusion

Spreadsheets are the duct tape of software development in general-- but they appear to be more than that for video game development.

They provide more than just the glue for video games. They provide the interface for working with non-game developer types and allow you to prove out your game idea before you even start drawing it out.

Every project needs an end, and that includes video games. But how do you decide what "done" means for your prototype, your vertical slice, or even your whole game? I might not know much about game development—but I have a lot of experience defining software scope. For my most recent iteration, I spent time out of the code editor and in a Markdown editor. I even used spreadsheets! *gasp* 😜

As I went through this past two-week iteration, I spent the majority of my time trying to nail down the project scope. More specifically, trying to answer three questions:

• What is this game?
• What are my project constraints?
• Where is my audience?

Although I am not experienced in game development, I have quite a bit of experience defining scope for non-gaming projects. I tried to map that experience to what I am trying to do with the Passenger prototype.

So, what do you do to define scope in software?

You write documentation, of course! Which is exactly what I did this iteration.

Here's what I worked on in terms of documentation.

GDD (Game Design Document)

A GDD is sort of the obvious one. I am not going to go into depth here about how to write one, as much more experienced folks out there have already done that. However, if you are unfamiliar with them, I suggest doing some research.

I based mine on the one shared at develop.games, which is still available as a Google Doc here.

The GDD provides the scaffold for the project and, more importantly, a reference you can always return to when figuring out if something belongs in your game or if it's scope creep.

Project Plan / Schedule

I have grouped these together because they help define the constraints for the project.

With projects, you always have constraints. The usual suspects are time and money. In my case, I am most curious about time (mainly because I have very little to no money for the project at this point).

I use GitHub for the project plan, and it helps me understand how much work I need to do to complete the various milestones of the project. The schedule shows me how much time I have available and what beats I want to hit with my game. Together, these let you do "project math" to determine if you have enough time in your schedule based on how long you estimate the project will take.

Again, not going to elaborate on project math—and estimates, by definition, are not facts or truth. However, they can help you determine if you have too much scope in your plan and need to trim things down a bit.

You trim them down by reviewing the GDD and the issues list in your project plan to figure out if every feature is needed.

Competitive Analysis

I learned to put together a competitive analysis for a game through a masterclass and through local resources with the business development group New Media Manitoba.

When you learn about what it takes to get funding for a video game, you find that it takes more than just a great pitch and a demo. It also takes a business plan.

The competitive analysis is what publishers and investors are going to want to see (amongst other things) to determine if your game is commercially viable, because it helps answer the question of finding your audience.

Once you know where your audience is (i.e. they are on the Steam pages of those comparable games), you can analyze the tags and figure out if your game idea aligns with the games you defined as comparable.

In my case, I assessed the tags that I wanted to use for my game and figured out if they aligned, in some capacity, with the games I believed to be comparable to Passenger. From there, I can start to deduce if my scope is big enough, too big, or too small for that audience.

It helped me answer the question where is my audience? when it comes to Steam. I can see the size of other games I believe to be comparable and figure out if my scope is realistic, too big, or too small.

Story Outline

This might not work for every game project, but it works for the ones with a narrative as a core feature of the game.

I started writing down the story to figure out how I would tell it through the gameplay mechanics I had defined so far and those I had planned. This exercise of understanding all my story beats helped me clarify the game I have in my head and, ultimately, whether I have all the pieces to deliver that story.

If I do, then my scope is probably in a reasonable state to deliver the vision I have in my head.

If I don't, then I either need to change my story or add the features I think I need.

Related Devlog

I recorded and published a video on YouTube. I go into a bit more depth about some of these documents if you want more specific context about how I use them for Passenger.

Have a great rest of your day.

As familiar as I am with scalable software design for non-game development projects, in this past iteration, I started to assess whether or not scalability is something I need to consider for video game design.

Video games are software, but they're not like any other software.

I have been dealing in software development for almost two decades at this point, but the more I tackle creating a commercial video game, the more I see how game development is just not the same.

My journey in game development continued with another completed iteration where game design was the focus. As the game started taking shape and a project started to form, the idea of scalable game design entered my mind.

Is scalable game design a thing?

This is what I reflect on in the devlog for this iteration.

With Prairie Dev Con (PrDC) 2026 complete, there was a clear theme in everything I managed to see and share—the future of software development is agentic. This doesn't necessarily mean it's all "doom and gloom," but rather an evolution has taken hold in our industry.

With Prairie Dev Con (PrDC) 2026 complete, there was a clear theme in everything I managed to see and share—the future of software development is agentic. This doesn't necessarily mean it's all "doom and gloom," but rather an evolution has taken hold in our industry.

Three Perspectives on AI: My first conference panel

I am always grateful to Prairie Dev Con for accepting my wild ideas for talks and whatnot and allowing me the privilege of speaking to a fantastic audience. This year, I proposed something different—something unfamiliar to me.

I proposed a panel.

The Hypothesis

Why a panel? you ask? Good question.

If I am being honest, there are two reasons:

1. I wanted to try something different from a lecture-style conference submission.
2. I wanted to compare and contrast different viewpoints on the spot, rather than compile the data myself and share it.

The hypothesis was that different perspectives will focus on different areas when it comes to AI, specifically with large language models (LLMs).

With that hypothesis, I found three different people who I have discussed AI with previously and brought them together. We had an enterprise CIO, an investor, and a small business founder.

And then came the panel.

The Result

I asked a series of questions, and each of them presented their own take on the subject, and the answers were very interesting in my humble and likely biased opinion (considering it was my panel). Some answers landed almost exactly the same and just complemented each other, while others focused on the pressing concerns from their unique space.

What was most interesting was when they all landed on or near the same answer—considering such different experiences and different problems they are trying to solve. It was really fascinating.

You can check out the slides in the talks section of the website. One of my panelists did manage to record the session, but it is going to take some audio cleanup, so I'll see what I can do and hopefully share that if I can make it usable.

Takeaway 1: Agents aren't the future, they are the standard.

I have been working with LLM-flavoured AI for quite some time. Even though I was very sure agentic workloads were coming, what PrDC made me realize is that not only are they here—but they are standard when it comes to software development.

A large majority of the sessions this year were flavoured with AI, including (but definitely not limited to):

• Generation Is Cheap. Judgment Isn't. (This was the keynote)
• How product engineers at Microsoft are using AI
• Agent Skills, Plugins & Marketplaces
• Building AI-Powered Business Intelligence: Transform Your Data Warehouse into an Intelligent Decision
• Agentic AI Raises the Stakes for API Security
• Build an MCP Data Source
• AI Enablement in the SDLC: A Strategy Sampler for 2026
• The Tokens are Coming From Inside the House
• Securing the Software your Agent Wrote
• Getting the Most Out of AI Coding Agents: Lessons From a Year in the Trenches

The list continues but it is clear that AI and agents are the future based on these developers, one way or another. But it wasn't their words that really caught my interest, but the demonstrations and stories of "how" they are using agents today.

Agents are the New Keyboard for Code

I don't really know how else to put it, but from the demonstrations I saw, senior engineers were using coding agents as a replacement for their keyboards. So, rather than typing the code in themselves, they were getting the agents to write the code and then reviewing it.

This has made the new bottleneck the review process, which we see in open source projects ultimately reflecting what I saw at the conference.

But isn't that a bad thing?

I don't think so, no.

Although the software developers and senior folks were using agents as their primary way to change code, they were still actively using their skills to review and assess those changes. They were just doing multiple changes in parallel and reviewing them when they were ready.

It is still software development, but just in an accelerated form, not bound by hands at a keyboard.

Takeaway 2: It is Still Changing

Although it seems clear to me that agentic work is inevitable in software development, the future isn't written in stone. In fact, many developers dropped a line akin to "it will probably be different within the next 6–12 months."

That is wild.

The pace at which the way we code is changing, only to be accepted by thought leaders working (not owning) the space, is incredible.

When some new tech or standard would come along, it would take a year or two before people would really start adopting it en masse. But with agents? We literally don't use our code editors the same way anymore, and we know it is going to change more.

Conclusion

I love Prairie Dev Con, and this was no different.

I am grateful to have been selected, and also grateful to have been able to learn so much from so many experienced people.

In the end, I learned that LLMs and AI aren't just an evolution—but a revolution in how software engineering will work. The question is, will I (and you) be able to keep up?

Stacked pull requests helped me organize and merge a set of dependent changes to this website—and gave me a little faith in GitHub as a platform again.

GitHub introduced a non-AI-focused feature called Stacked Pull Requests that helps you merge multiple PRs at once. This is the first feature since the introduction of Projects that has interested me in years, and it has given me a bit of faith in GitHub as a platform again.

Why Did I Use Stacked PRs?

I heard about Stacked PRs through the Western Devs (yes, we still talk, we just don't blog as much) and thought I would use them to "stack" a bunch of changes I wanted to apply to my website. My website has been something of a forever hobby project that I use to experiment with various technologies and keep my hands a little dirty when it comes to web tooling.

In any case, there were a number of features I wanted to apply to relaunch the site, but I knew it would be a massive change to the project. Between data migrations, new styling, and a new site structure, the change logs were going to be massive.

So, instead of one gigantic PR or having a bunch of PRs that I have to keep track of myself, I decided to try stacked PRs to keep track of the merge order for me.

Here are a couple of images of the CLI and Web UIs for stacked PRs from GitHub that I used:

Were Stacked PRs Useful?

Put simply: yes.

It sounds like a simple task, but even keeping my side project branches organized is difficult for me. I have tried keeping notes in the PRs themselves and organizing myself around milestones, but nothing really worked because it didn't tie everything together on the workflow side of things.

With my CI/CD pipelines in place, stacked PRs made it easy for me to order the branches properly, add new ones when necessary, and highlight where things would fail early when merging them altogether.

This Isn't Groundbreaking Though-- Is It?

No, of course not.

Honestly, I had been considering moving away from GitHub since they only seem to talk about generative AI coding. I'm not against AI coding, but that isn't the ONLY reason I use GitHub. In fact, I don't use GitHub Copilot; I use Codex for AI code assistance.

My point is that GitHub is a social coding platform with DevOps tooling, and if they aren't in that business anymore, then I should take the time to learn a toolset that does care about DevOps tooling.

I will save that for another post.

The cloud make this so much easier. We could spin up a server in minutes and with platform as a service we didn't have to worry about underlying compute or updates to operating systems, or patching. It was a dream. I'd tell clients "You're an insurance company leave being an IT company to Microsoft/Amazon/Google".

Recently though I've been looking at the cost of hosting an application in Azure. Even for our small applications it is surprisngly expensive. We had a tendency to vastly overbuild the hosting when for applications that have only hundreds or low thousands of users. What really gets us is the cost of the database.

This isn't a huge or highly performant database, but it is still costing us almost $400/month. For a small application with only a few hundred users this is a huge cost. Add on dev and test environments and the cost is even higher. Painful.

I wanted to understand if I was still recommending the right thing to people so I started looking at alternatives. My first stop was looking at a VPS provider. I wanted something in Canada becasue this is where I live and I wanted to support local businesses. I found a provider called OVH that had a VPS with 8 vCores, 24 Gig of memory and a 200 Gig SSD for $32 a month off contract.

Now obviously vCores aren't a standard measurement unit so it's hard to compares that with something from another offering like Azure but on the surface this feels similar to a D12 from Azure which is $380/month and that's in USD so closer to $500/month CAD. The OVH offering is about 1/15th the cost of Azure.

Yikes!

So I figured I'd sign up. Maybe the added cost here would be becasue it was really arduous to get signed up. I'll admit going through the process wasn't as polished as a cloud provier. Little things like this

Is that a last name field or a field for people who only have a single name? Is Madonna buying hosting a frequently? The 2FA setup, though, was probably the best I've ever done. It had the really nice feature of letting you record the name of the app you were using for 2FA. I have multiple apps for 2FA and this is a really nice feature.

I did spot this warning, though

Okay so there might be some latency in getting this server provisioned. That sucks, but perhaps I've gotten too used to the instant gratificaiton of demanding compute and getting it 5 minutes later. The email I got said that the server would be provisioned within the next 7 days. That's a long time to wait but whilte we do let's talk about why this VPS might not be a great idea. What are we losing by not using cloud hosting?

Well first off it's clear that scaling up and down is going to be so much harder. I can't scale the machine up instantly when there is a spike in traffic. Equally, if that spike subsites I can't scale down quickly. So I'm going to have to be comfortable knowing that I've got to pay for excess capacity. A lot of the line of business applications I work with have almost no traffic on the weekend and evenings but then predictable stable load during the day. I'm going to have to over provision to handle the maximum load. Maybe I can mine bitcoin or something on the weekends. Is that SETI@home project still a thing?

Next I'm losing the platform as a service benefits. I'm going to have to patch the operating system and database myself. I get to do backups, restores all that myself. So I'm going to trade my time for reduced hosting costs. Let's imagine that I'm setting up a database server and a web server which is kind of typical for a line of business application. This is going to run to $700/month in Azure(I'm going to use Candian dollars from here out). So that gives me $650 a month to play with for manual patching and updating and all that jazz. If I pay myself $200/hour (woo, massive pay raise) then I can spend 3 hours a month on that work and still come out ahead. I don't know if that scales well to dozens of servers but for a small application it might be worth it.

I'm likely going to have an increased cost for the initial set up of the server. I've got to figure out which web server to use, probably some default settings for PostgreSQL and tie it into a build pipeline. I suppose we'll be deploying via SFTP or something like that.

Finally, I'm losing the reliability and security of the cloud. I don't have the same level of redundancy and failover that I would get with a cloud provider. If my server goes down, it's on me to get it back up. If there is a security breach, it's on me to fix it. This is a big risk for any business, but especially for small businesses that may not have the resources to deal with these issues.

The other piece of the puzzle here is that the knowledge and expertise required to manage a VPS is higher than that required to use a cloud provider. I've been doing this computer stuff for year and I'm confident in my ability to teasily manage this but then I'm also still a bit sad about how Open Solaris failed to take off. I think that a lot of this gap can be closed using AI agents these days.

It's a calculated risk.

So, dear readers, I pulled the trigger and signed up for the VPS. Once it is provisioned then we'll return here and talk through setting up and deploying to it.

• ClearCase
• Perforce
• Subversion
• CVS
• Visual Source Safe
• Mercurial
• Git
• CCC/Harvest

I'm using Jujutsu at my day job now because it just layers transparently on top of git so I don't need to go seeking permission. I'm only a few days into using it and I'm not thoroughly convinced yet that it is better than git but I'm willing to keep trying.

Here are some of the commands I'm using so far:

Get the latest version of the code from a central repository locally

1

jj git fetch

Start new work from the latest mainline

1

jj new main@origin -m "Whatever I'm going to work on"

Bookmark the work with a name I'm going to use as a branch in git

1

jj bookmark my-feature-branch

Push my work up to Github

1

jj git push --allow-new

Create a new commit before my current one that I can squash into

1

jj new -B @ -m "Some description of the work"

Push individual files into the parent change

1

jj squash path/to/file1 path/to/file2

I'll keep expanding this document with new commands as I uncover them.

The first thing to do is to install the OpenAPI generator. It's written in Java which is obviously a decision I'm solidly against. As I run a Mac most of the time I prefer to use brew since it's easier that trying to figure out how to build stuff with Maven.

1

brew install openapi-generator

Now comes the fun part: figuring out the options to use. There are bunch of different generators for different languages and on top of that each generator has options. The C# generator is called csharp and the options are described in some detail here https://openapi-generator.tech/docs/generators/csharp. In my case I was looking to generator a client for a US Government API that they seem to be really snippy about people getting their hands on documentation about it without going through a heap of hoops so we'll just call it USGovAPI because this administration is not one I want to be on the wrong side of.

In addition I wanted to generate one for .NET 8 because the project is still running on the long term support version of .NET. So the command ended up being

1

openapi-generator generate -i swagger.json -g csharp -o out/usgoveapi --additional-properties=packageName=USGovAPI,targetFramework=net8.0

And with that we get a nice little C# client that we can use to call the USGovAPI. It even includes some unit tests to go along with it.

Fixing this is easy enough. I needed to add a couple of lines to my dependabot file to limit the sorts of updates it did to just minor and patch updates. Notice the update-types section.

1
2
3
4
5
6
7
8
9
10
11
12
13

updates:
  - package-ecosystem: "nuget"
    directory: "/."
    groups:
      all_packages:
        update-types:
          - "minor"
          - "patch"
        patterns:
          - "*"
    open-pull-requests-limit: 50
    schedule:
      interval: "weekly"

With this in place dependabot is only proposing minor and patch updates to my .NET packages. It does mean that if we see other major version updates to non-Microsoft packages we'll have to manually update them.

I tried overriding the entry point for the container and attaching to it to see what was going on but I couldn't see anything wrong. I was able to write to the directory without issue. Eventually I stumbled on a note in the RavenDB documentation which mentioned a change in the 6.x version of RavenDB which meant that Raven no longer ran as root inside the container.

K8S has the ability to change the ownership of the volume to the user that the container is running as. This is done by setting the fsGroup property in the pod spec. In this case Raven runs as UID 999. So I updated my tanka spec to include the fsGroup property and the problem was solved.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

...
deployment: deployment.new($._config.containers.ravendb.name) {
      metadata+: {
        namespace: 'wigglepiggle-' + $._config.environment,
        labels: {
          app: $._config.containers.ravendb.name,
        },
      },
      spec+: {
        replicas: 1,
        selector: {
          matchLabels: $._config.labels,
        },
        template: {
          metadata+: {
            labels: $._config.labels,
          },
          spec+: {
            securityContext: {
              fsGroup: 999,
              fsGroupChangePolicy: 'OnRootMismatch',
            },
            containers: [
              {
                name: $._config.containers.ravendb.name,
                image: $._config.containers.ravendb.image,
                ports: [{ containerPort: 8080 }, { containerPort: 38888 }],
                volumeMounts: [
                  {
                    name: 'data',
                    mountPath: '/var/lib/ravendb/data',
                  },
                ],
                env: [
                  {
                    name: 'RAVEN_Setup_Mode',
                    value: 'None',
                  },
                  {
                    name: 'RAVEN_License_Eula_Accepted',
                    value: 'true',
                  },
                  {
                    name: 'RAVEN_ARGS',
                    value: '--log-to-console',
                  },
                  {
                    name: 'RAVEN_Security_UnsecuredAccessAllowed',
                    value: 'PrivateNetwork',
                  },
                ],
              },
            ],
            volumes: [
              {
                name: 'data',
                persistentVolumeClaim: {
                  claimName: $._config.containers.ravendb.name,
                },
              },
            ],
          },
        },
      },
    },
...

This generated yml like

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: service-control-ravendb
  name: service-control-ravendb
  namespace: wigglepiggle-dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service-control
      environment: dev
  template:
    metadata:
      labels:
        app: service-control
        environment: dev
    spec:
      containers:
      - env:
        - name: RAVEN_Setup_Mode
          value: None
        - name: RAVEN_License_Eula_Accepted
          value: "true"
        - name: RAVEN_ARGS
          value: --log-to-console
        - name: RAVEN_Security_UnsecuredAccessAllowed
          value: PrivateNetwork
        image: ravendb/ravendb:6.0-latest
        name: service-control-ravendb
        ports:
        - containerPort: 8080
        - containerPort: 38888
        volumeMounts:
        - mountPath: /var/lib/ravendb/data
          name: data
      securityContext:
        fsGroup: 999
        fsGroupChangePolicy: OnRootMismatch
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: service-control-ravendb

1

docker logs -f --tail 10 5efd3ee0074a

There in the logs was the culprit No space left on device. Uh oh, what's going on here? Sure enough the disk was full.

1

df -h

1
2
3
4
5
6
7

Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              1.6G  1.9M  1.6G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   96G   93G     0 100% /
tmpfs                              7.8G     0  7.8G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
/dev/sda2                          2.0G  182M  1.7G  10% /boot
tmpfs                              1.6G   12K  1.6G   1% /run/user/1001

This stuff is always annoying because you get to a point where you can't run any commands because there is no space left. I started with cleaning up some small parts of docker

docker system prune -a

Then cleaned up docker logs

sudo truncate -s 0 /var/lib/docker/containers/**/*-json.log

This then gave me enough space to run docker system df and see where the space was being used. Containers were the culprit. So next was to run

docker ps --size

Which showed me the web scraper container had gone off the rails and was using over a 100GiB of space.

1

7cf14084c56a   webscraper:latest       "wsce start -v -y"       7 weeks ago   Up 20 minutes               0.0.0.0:7002->9924/tcp, [::]:7002->9924/tcp                                webscraper       123GB (virtual 125GB)

This thing is supposed to be stateless so I just killed an removed it.

1
2
3

docker kill 7cf14084c56a
docker rm 7cf14084c56a
docker compose up -d webscraper

After a few minutes these completed and all was good again. So we'll keep an eye on that service and perhaps reboot it every few months to keep it in check.

1
2
3
4
5
6
7
8
9

nodeLinker: node-modules

npmScopes:
  stimms:
    npmRegistryServer: "https://npm.pkg.github.com"
    
npmRegistries:
  "https://npm.pkg.github.com":
    npmAlwaysAuth: true

Now in the build I needed to add a step in to populate the GITHUB_TOKEN which can be used for authentication. I found quite a bit of documentation which suggested that the .yarnrc.yml file would be able to read the environment variable but I had no luck with that approach. Instead I added a step to the build to populate the GITHUB_TOKEN in the .npmrc file.

1
2
3
4
5
6
7
8
9

- name: Configure GitHub Packages Auth
    run: echo "//npm.pkg.github.com/:_authToken=${GITHUB_TOKEN}" > ~/.npmrc
    env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Build
    run: |
    yarn install
    yarn lint
    yarn build

The final thing to remember is that by default the GITHUB_TOKEN here doesn't have read permission over your packages. You'll need to go into the package settings and add the repository to the list of repositories which can use the package. You just need read access. If you don't do this step you're going to see an error like error Error: https://npm.pkg.github.com/@stimms%2fuicomponents: authentication token not provided

Install it with

1

dotnet tool install --global dotnet-outdated

Then you can run it in the root of your project to list the packages which will be updated with

1

dotnet outdated

Then, if you're happy, run it again with

1

dotnet outdated -u

to actually get everything updated.

• Kata 1 - Sending a message
• Kata 2 - Publishing a message
• Kata 3 - Switching transports
• Kata 4 - Long running processes
• Kata 5 - Timeouts
• Kata 6 - When things go wrong

NServiceBus is built to be reliable. It is designed to be able to handle errors and to be able to recover from them. The most common reason for an error to show up in NServiceBus is that a message fails to process. This could be because the message is malformed, a resource which is needed to process the message is unavailable or because the handler throws an exception. Throwing exceptions in handlers is actually the preferred way to handle errors in NServiceBus.

When a message fails to process NServiceBus will attempt to reprocess the message. NServiceBus defines two different retry mechanisms: immediate and delayed retry. Immediate retries will reprocess at once and by default happen 5 times. Delayed retries fire at 10, 20 and 30 second after the failed retries and they too will fire immediate retries. So by the time a message has fully failed it's been attempted 20 times. If all these attempts fail then the message is shuffled off to an error queue. This error queue is not typically watched by NService Bus and recovering a message form it is a manual process.

In NServiceBus installations I've worked on we monitor the error queues fairly closely and work to make the application resilient to errors such that the queues are only populated by messages which have failed due to resources being unavailable. Don't just ignore these messages as they are likely important user interactions or business processes which will cost you money if you don't address them.

The Kata

Let's make one of our messages fail to process and watch what happens. In this scenario let's fail to properly sanitize our inputs to the messages and have them take on a value which makes no sense. We have a message EatCake which has a property NumberOfCakes. Let's make it so that the NumberOfCakes is set to -1 and that we have a validation in place in our message handler to throw when we see this problem. Let's also log a message so we can see the retries happening.

My Solution

1. Modify the EatCake message hander to have a validation to make sure cakes are not eaten in the negative.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

public async Task Handle(EatCake message, IMessageHandlerContext context)
    {
        if (message.NumberOfCakes < 0)
        {
            Console.WriteLine("Negative cake? Really?");
            throw new Exception("Negative cake? Really?");
        }
        if (message.NumberOfCakes == 0)
        {
            Console.WriteLine("No cake to eat");
            throw new Exception("No cake to eat");
        }
        
        Console.WriteLine($"Cake eaten, NumberOfCakes = {message.NumberOfCakes}; Flavour = {message.Flavour}");
        await context.Publish(new CakeEaten
        {
            EatingFinishedAt = DateTime.Now
        });
    }

2. Modify the sender to send a message with a negative number of cakes

1
2
3
4
5
6
7
8
9
10

switch (line)
{
    case "send":
        await endpointInstance.Send("NServiceBusKataReceiver", new EatCake
        {
            Flavour = "Coconut",
            NumberOfCakes = -1
        });
        break;
        ...

3. You may need to configure the error queue in the receiver endpoint.

1

endpointConfiguration.SendFailedMessagesTo("NServiceBusKataReceiverError");

And build the error queue using the handy dandy rabbitmq-transport tool

1

rabbitmq-transport endpoint create NServiceBusKataAnotherReceiver -c "host=localhost" --errorQueue NServiceBusKataAnotherReceiverError

Things to try now

1. Run the sender and produce some messages with negative numbers of cakes
2. Observe the logs for the receiver and see the retries happening
3. Wait a bit for the retries to be exhausted and see the message end up in the error queue
4. Open up the rabbitmq management console and look at the error queue. Pay special attention to the headers of the message. You'll see the number of retries and the time the message was first sent as well as the exception message.

Things to think about

Looking at messages in the RabbitMQ management console is great for a single message but in the aggregate there is often a need for better tooling. Particular provide some tooling in the form of ServiceInsight which can be used to monitor the flow of messages through the system. This can be used to see where messages are failing and to help diagnose the problem.

• Kata 1 - Sending a message
• Kata 2 - Publishing a message
• Kata 3 - Switching transports
• Kata 4 - Long running processes
• Kata 5 - Timeouts
• Kata 6 - When things go wrong

I think we're pretty used to the idea that a timeout is something we want to avoid - it usually means that a server isn't available or that we've run a query that is taking too long. But there are lots of places in business processes where timeout are just part of the process we're modeling and aren't an error at all. As an example consider a shopping cart: if a user adds an item to the cart and then doesn't check out within a certain period of time we might want to send them a reminder and even offer them a discount on their purchases. If we've modeled the checkout process as a saga then this reminder can be set up as a timeout.

The Kata

We just implemented a cake order saga in the last kata. Let's extend that saga to include a timeout. If the cake isn't shipped within 2 minutes of the order being placed then we should send a message to the bakery to ask them to check on the order. Obviously 2 minutes is a pretty quick turn around on a cake but I don't imagine you want to hang around for hours writing this kata.

My Solution

1. Add a new message to the messages project

1
2
3
4
5
6
7
8

namespace messages;

using NServiceBus;

public class CakeOrderStalled : IMessage
{
    public Guid OrderId { get; set; }
}

2. Modify the saga to start a timeout when an order is placed. The CakeOrderPlaced handler becomes

1
2
3
4
5
6

public Task Handle(CakeOrderPlaced message, IMessageHandlerContext context)
{
    Console.WriteLine($"Order {message.OrderId} placed");
    
    return RequestTimeout<CakeOrderStalled>(context, TimeSpan.FromMinutes(2), new CakeOrderStalled{ OrderId = message.OrderId });
}

3. Add a handler for the timeout message

1
2
3
4
5

public Task Timeout(CakeOrderStalled state, IMessageHandlerContext context)
{
    Console.WriteLine($"Order {Data.OrderId} stalled!");
    return Task.CompletedTask;
}

4. Have the saga implement

1

IHandleTimeouts<CakeOrderStalled>

4. We missed it previously but we need to have canceling the order or completing the order mark the saga as complete. To do that add MarkAsComplete to the handlers for those messages

1
2
3
4
5
6

public Task Handle(CakeOrderCanceled message, IMessageHandlerContext context)
{
    Console.WriteLine($"Order {message.OrderId} canceled");
    MarkAsComplete();
    return Task.CompletedTask;
}

Things to try now

1. Run the applications and in the sender app try starting the saga with o to start an order. Wait a couple of minutes and see that the timeout fires.
2. Try starting the saga and then canceling it with o followed by c and see what happens when the timeout fires.

Things to think about

Once a timeout is registered there is no way to cancel it. How come? What would you do if you needed to cancel a timeout? If messages are delayed is it possible that timeouts could fire when an expected action has actually happened? How would you mitigate that risk?

• Kata 1 - Sending a message
• Kata 2 - Publishing a message
• Kata 3 - Switching transports
• Kata 4 - Long running processes
• Kata 5 - Timeouts
• Kata 6 - When things go wrong

Consider the bakery which is creating the cake for me to eat: when it starts making a new cake because I ate the last one it has a bunch of things it needs to coordinate. They need to preheat ovens, gather ingredients, mix ingredients, grease pans, fill pans, put pans in the oven, remember to take the cake out, cool it, ice it... the list goes on and on - no wonder cake is so expensive. Coordinating all these activities is complex in a distributed system, really in any system. There are a lot of corner cases that we usually fail to consider in a non-distributed system which become much more apparent when building out a process manager. What if we preheat the oven but then discover that we're all out of flour? In that monolithic system we might throw an exception and hope that somebody is monitoring for it in a log file somewhere. Realistically that's never going to happen. In the meantime nobody has shut the oven off and the bakery burns down.

A saga allows us to store the state of a process, to react to messages as they come in and send new messages. We use this to coordinate the activities of the bakery. In our example above we can probably call the process "BakeCakeSaga". When messages come in relating to the order then we need to be able to find a way to look up the state and make modifications to it. NServiceBus implements this through a method called ConfigureHowToFindSaga. This function will provide a mapping from every message that interacts with the saga to find the saga data. For our example we'd probably use something like an order id.

Let's build out a very simple saga which responds to just a few messages in our system so we can see how it works. Saga can get pretty complex but they are quite testable so that's nice.

The Kata

Create a saga which handles the messages CakeOrderPlaced, CakeOrderCanceled, CakeOrderShipped. Each of these messages will contain an OrderId, a GUID, which will be used to identify the saga as well as whatever information might be associated with those messages. For now just write out to the console when each of these messages is received - unless you want to bake me a cake which I will accept.

The Solution

1. Add some mechanism to handle the persistence of saga data. For now we'll just use the in learning persistence. In the various program.cs files add

1

var persistence = endpointConfiguration.UsePersistence<LearningPersistence>();

2. Create messages classes in the messages project

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

namespace messages;

using NServiceBus;

public class CakeOrderPlaced : IEvent
{
    public Guid OrderId { get; set; }
    public Guid CustomerId { get; set; }
    public Date OrderDate { get; set; }
}

public class CakeOrderCanceled : IEvent
{
    public Guid OrderId { get; set; }
    public string Reason { get; set; }
}

public class CakeOrderShipped : IEvent
{
    public Guid OrderId { get; set; }
    public string ShippingReferenceNumber { get; set; }
}

3. Create a saga class in the receiver project

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

using NServiceBus;
using messages;
public class CakeOrderSaga : Saga<CakeOrderSagaData>,
    IAmStartedByMessages<CakeOrderCanceled>,
    IAmStartedByMessages<CakeOrderPlaced>,
    IAmStartedByMessages<CakeOrderShipped>
{

    protected override void ConfigureHowToFindSaga(SagaPropertyMapper<CakeOrderSagaData> mapper)
    {
        mapper.MapSaga(sagaData => sagaData.OrderId)
            .ToMessage<CakeOrderCanceled>(x => x.OrderId)
            .ToMessage<CakeOrderPlaced>(x => x.OrderId)
            .ToMessage<CakeOrderShipped>(x => x.OrderId);
    }

    public Task Handle(CakeOrderPlaced message, IMessageHandlerContext context)
    {
        Console.WriteLine($"Order {message.OrderId} placed");
        return Task.CompletedTask;
    }

    public Task Handle(CakeOrderCanceled message, IMessageHandlerContext context)
    {
        Console.WriteLine($"Order {message.OrderId} canceled");
        Data.OrderCanceled = true;
        return Task.CompletedTask;
    }

    public Task Handle(CakeOrderShipped message, IMessageHandlerContext context)
    {
        Console.WriteLine($"Order {message.OrderId} shipped");
        Data.OrderShipped = true;
        return Task.CompletedTask;
    }

}

4. Add a saga data class to the receiver project

1
2
3
4
5
6
7
8

using NServiceBus;

public class CakeOrderSagaData : ContainSagaData
{
    public Guid OrderId { get; set; }
    public bool OrderCanceled { get; set; }
    public bool OrderShipped { get; set; }
}

5. Modify the sender project's program.cs to send the messages adding a loop which sends all the different messages involved in the saga.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

bool continueMessages = true;
Guid orderId = Guid.NewGuid();

while (continueMessages)
{
    var line = Console.ReadLine();
    switch (line)
    {
        case "p":
            await endpointInstance.Publish(new CakeOrderPlaced { OrderId = orderId });
            break;
        case "c":
            await endpointInstance.Publish(new CakeOrderCanceled { OrderId = orderId });
            break;
        case "s":
            await endpointInstance.Publish(new CakeOrderShipped { OrderId = orderId });
            break;
        case "q":
            continueMessages = false;
            break;
        default:
            break;
    }

}

Things to try now

1. Run the applications and in the sender app try pressing some keys like p or c or s to see the messages being handled by the saga.
2. Try starting the applications in different orders and see how the saga handles the messages.

Things to Notice

Notice that the Saga can be started by 3 different events. Why would a saga be started by a cancel message? You can't cancel an order which hasn't even been placed yet - right? Well it turns out you can. Without some serious hoop jumping through the order of message delivery it not guaranteed. So in fact orders can be canceled or shipped before we get the message telling us the order has been placed. It is sort of mind-blowing.

• Kata 1 - Sending a message
• Kata 2 - Publishing a message
• Kata 3 - Switching transports
• Kata 4 - Long running processes
• Kata 5 - Timeouts
• Kata 6 - When things go wrong

One of the really nice things about NServiceBus is that it abstracts away a lot of the mess of dealing with transports. You can spend less time fiddling with plumbing and more time getting to the nitty gritty of building business value. We have a lot of options for transport and the ability to write new ones if needed. The official supported transports are

• Azure Service Bus
• Azure Storage Queues
• Amazon SQS
• RabbitMQ
• SQL Server
• MSMQ

Each of these has advantages and disadvantages which you can read about in some detail in the documentation. In a production environment I'd push heavily towards Azure Service Bus but it has a major disadvantage: no local emulator. There is a github issue open to add an emulator which has been open since 2018. Normally I'd grumble a bit about how this is never going to be solved but actually the team has dedicated resources to building one out and think they'll have one ready by the end of 2024. Only you, dear readers from the future, know if this came to fruition.

For now we're going to take advantage of the interchangeability of transports and switch out the LearningTransport for the RabbitMQ transport. This can be run with a container on docker.

The Kata

Take the solution developed already and switch out the learning transport for the RabbitMQ transport running inside of a container. Everything should continue to work as it did before but now using RabbitMQ.

My Solution

1. Download and run the RabbitMQ container

1

docker run -d --hostname nsbkata --name nsbkata-rabbit -p 15672:15672 -p 5672:5672 rabbitmq:3-management

2. Install the RabbitMQ transport package in all the projects

1
2
3

dotnet add sender package NServiceBus.RabbitMQ
dotnet add receiver package NServiceBus.RabbitMQ
dotnet add anotherReceiver package NServiceBus.RabbitMQ

3. Modify the endpoint configurations across all the projects swapping

1

var transport = endpointConfiguration.UseTransport<LearningTransport>();

with

1
2
3

var transport = endpointConfiguration.UseTransport<RabbitMQTransport>();
transport.ConnectionString("host=localhost");
transport.UseConventionalRoutingTopology(QueueType.Quorum);

4. It is generally best that you take control of creating queues for NServiceBus but tooling is provided. Install that tooling

1

dotnet tool install -g NServiceBus.Transport.RabbitMQ.CommandLine

Next create the queues for the endpoints (and the delays queues)

1
2
3
4

rabbitmq-transport delays create  -c "host=localhost"
rabbitmq-transport endpoint create NServiceBusKataAnotherReceiver -c "host=localhost"
rabbitmq-transport endpoint create NServiceBusKataReceiver -c "host=localhost"
rabbitmq-transport endpoint create NServiceBusKataSender -c "host=localhost"

Things to try now

1. Run the sender, receiver and anotherReceiver projects. You should see the messages being sent and received as before.

2. Log into the RabbitMQ management console(running at http://localhost:15672/ with credentials guest/guest) you should see the queues created and in fact a message processed

• Kata 1 - Sending a message
• Kata 2 - Publishing a message
• Kata 3 - Switching transports
• Kata 4 - Long running processes
• Kata 5 - Timeouts
• Kata 6 - When things go wrong

The Problem

It is great being able to send a message from one system to another - you can instruct a remote system to take some action which you don't know how to do. For instance sending an email. In a large system lots of processes will likely result in an email but if you can centralize the logic around how to send an email it makes it very easy to do things like change email providers as that functionality is isolated and independent.

If you looked at my solution for the first Kata then you might have notice that I named my message EatCake this is named in the imperative form. This is a common pattern in messaging systems and is, I think, remarkably helpful to understand the command pattern. When you issue a command you know exactly who or what you're commanding and in the same way the command is sent to a known endpoint.

Sometimes, however, you don't have a particular target in mind and you just want to let other systems know that you have done something. In this case you can publish a message. Unlike a command you don't know who is going to act on it - it may be nobody or it may be multiple people. In a messaging system we call this an Event.

Let's extend our cake eating example to imagine some consequences of what might happen if I happily reported that I had eaten cake: CakeEaten. Finding out that the cake had been eaten the bakery might start baking more cake assuming that I'd want more (very likely). My gym might free up a treadmill knowing that my cake-induced guilt might drive me to run until I'd burned off the delicious butter-cream. It might also be of interest to my wife who can text me and see if I enjoyed the cake. I don't actually know who would be interested in the event so it is the responsibility of those who are interested to subscribe to the event.

The Kata

Using the solution to the first kata as a starting point extend the system to publish a CakeEaten message. Subscribe to the message in the sender and write out a message to the console. Also create another new project similar to the others and have it subscribe to the message.

My Solution

1. Create a new message in the messages assembly

1
2
3
4
5
6
7
8

namespace messages;

using NServiceBus;

public class CakeEaten : IEvent
{
    public DateTime EatingFinishedAt { get; set; }
}

2. Modify the Hander in the receiver to publish the new event. Notice that we have access to a IMessageHandlerContext which we can use to publish messages.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

using messages;

public class PlaceOrderHandler :
    IHandleMessages<EatCake>
{
    public async Task Handle(EatCake message, IMessageHandlerContext context)
    {
        Console.WriteLine($"Cake eaten, NumberOfCakes = {message.NumberOfCakes}; Flavour = {message.Flavour}");
        await context.Publish(new CakeEaten
        {
            EatingFinishedAt = DateTime.Now
        });
    }
}

3. Modify the sender to subscribe to the event by adding a Handler to the project.

1
2
3
4
5
6
7
8
9
10
11
12
13

using messages;

using NServiceBus;

public class CakeEatenHandler :
    IHandleMessages<CakeEaten>
{
    public Task Handle(CakeEaten message, IMessageHandlerContext context)
    {
        Console.WriteLine($"Uh oh, somebody ate a cake at {message.EatingFinishedAt}");
        return Task.CompletedTask;
    }
}

4. Optionally modify the sender to remain running so that it can receive the message. This is done by adding a Console.ReadLine() at the end of the Main method.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

using NServiceBus;

Console.Title = "NServiceBusKata - Sender";

var endpointConfiguration = new EndpointConfiguration("NServiceBusKataReceiver");

// Choose JSON to serialize and deserialize messages
endpointConfiguration.UseSerialization<SystemJsonSerializer>();

var transport = endpointConfiguration.UseTransport<LearningTransport>();

var endpointInstance = await Endpoint.Start(endpointConfiguration);

Console.WriteLine("Press Enter to exit...");
Console.ReadLine();

await endpointInstance.Stop();

Pause here and ensure that when running the sender and receiver that you see both a message sent and published.

5. Create a new project to act as a second subscriber or receiver

1
2
3

dotnet new console -o anotherReceiver
dotnet add anotherReceiver reference ../messages
dotnet add anotherReceiver package NServiceBus

6. Add a subscriber class to the project

1
2
3
4
5
6
7
8
9
10
11
12
13

using messages;

using NServiceBus;

public class CakeEatenHandler :
    IHandleMessages<CakeEaten>
{
    public Task Handle(CakeEaten message, IMessageHandlerContext context)
    {
        Console.WriteLine($"Awesome, somebody ate a cake at {message.EatingFinishedAt}. Time to bake another");
        return Task.CompletedTask;
    }
}

7. Update the Program.cs to initiate NServiceBus

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

using messages;
using NServiceBus;

Console.Title = "NServiceBusKata - AnotherPublisher";

var endpointConfiguration = new EndpointConfiguration("NServiceBusKataAnotherReceiver");

// Choose JSON to serialize and deserialize messages
endpointConfiguration.UseSerialization<SystemJsonSerializer>();

var transport = endpointConfiguration.UseTransport<LearningTransport>();

var endpointInstance = await Endpoint.Start(endpointConfiguration);


Console.ReadLine();

await endpointInstance.Stop();

Things to try now:

1. Start the receiver and the another receiver then the sender - ensure you see messages flowing correctly
2. Start services in different orders and see which messages might be lost

Sending and receiving messages and publishing messages should now be working great. But to get this example going we've made some simplifications which we'll have to address in the next kata.