• How do I see the exceptions that happened?
• How do I configure an Azure Function app to save a PDF to a Google Drive folder

This is from the perspective of someone who doesn't need their app to be highly available/scalable/reliable/tenable/affable/inscrutable/explicable, which the documentation for all cloud products seems to assume (and, it must be said, rightfully so).

I like crossword puzzles. And I like doing them on paper. I subscribe to a few services, including the New York Times, AVCX, Matt Gaffney, the Muller Monthly Music Meta, and a couple of others, both free and paid. For years, I've maintained a bash file that downloads PDFs from various services that allow it (and some that likely don't) and merges them into a single document for printing. It works fine but I wanted to try my hand at moving it to an Azure Function app that ran on a timer rather than on demand like I do with the bash file.

So the app in question is pretty simple and the code itself isn't super interesting. Here are the logistics:

• Runs nightly
• Downloads PDFs from various crossword providers (three so far)
• Saves the PDF to Google Drive

The last one is only because that's where I keep them after they're printed. I have an archive of the puzzles I've downloaded going back a number of years. I've no idea why except that I have the storage available. Maybe I'll run through them again in retirement but the more likely scenario is that my children will mass delete them in a housecleaning exercise after I die.

With that in mind, let's get to the two areas where I struggled.

App Insights

By most accounts, App Insights on Azure is a powerful tool and telemetry in general is a big business. But I've always shied away from the Diagnose/Investigate/Metrics tabs because they're just plain overwhelming for an aging hillbilly who's used to scrolling through IIS log files. So when I wake up in the morning and the latest Times puzzle isn't there, I need to figure out where to go.

My solution (which may not align with the solution) is pretty simple in the end:

• Navigate to the Function app in Azure
• Go to Monitoring | Logs
• Query the exceptions table with no other qualifers and with an appropriate time range

This shows the exceptions and the stack traces within the time period. Given this runs nightly, there likely won't be more than a couple in the last 24 hours.

Connect to a Google Drive folder

Ugh...

I've said this on more than one occasion: I hate all things security-related including, but not limited to: certificates, OAuth, face ID, fingerprint scanners, digital signatures, encryption, authentication providers, 2FA, MFA, SSL, VPN, CVE, IAM, JWT, SSO, AES, TSA, passkeys, passwords, passports, key fobs, car alarms, and bike locks.

Setting up an Azure Function App to connect to Google Drive touches on several of these pet peeves and adds a few more. Here's what I eventually did to get this to work:

• Create a project in my Google Developer Console
• Enable the Google Drive API (it wasn't enabled by default)
• Create a service account credential
• Under the new service account, create a key. This downloads a file to your computer.
• Upload the file to the Azure Storage account for the Azure Functions app in a File Share folder called secrets
• Create two environment variables for the Azure Function:
  • GoogleApiSecretsFileName: the name of the file (include the .json extension) I downloaded for the key
  • GoogleDriveFolderId: The ID of the folder where the puzzles should be saved. (Navigate to the folder in a browser. The ID is everything after folder/ in the URL.)
• Share the Google drive folder with the Google API service account (with Editor access). The email address is on the Credentials page in the Google Developer Console.

Oh, and also write the actual code.

The Azure documentation and ChatGPT suggest strongly that OAuth2 credentials (instead of a service account) should work if you set up Google as an authenticator on the Azure Functions app. I ran into problems with this and I think it's because I was running locally. I have the app set up to do everything through a console app instead of through the Azure Functions host when running in DEBUG mode and I suspect the problems are because my local app hasn't gone through the necessary authentication process. Either way, a service account explicitly says it's for unattended server-to-server scenarios, which this is, so I've justified it in my head, even if the Azure documentation suggests something else.

It's a fairly simple process to view all of these calendars in a single place. All the major calendar providers generously provide iCal links at the very least to let you combine everything in a single view and you can even turn individual calendars on and off.

I have two problems with this.

First, I often don't want to have all of these calendars visible in a single view on my screen. Screen-sharing is common and on more than one occasion, I've said, "let's schedule a meeting" and switched to the shared calendar view where all my personal Dudeism meetings are there for all to see.

Second (and more important than risking an argument with a nihilist), people booking meetings with me don't see the unavailable slots on the other calendars, leading to double-bookings. I didn't know how good I had it in high school when this was...let's just say, "not a problem".

So I went on the hunt for a solution that met these criteria:

1. The ability to book an event on one calendar and have the time blocked on the other two.
2. Have the time on other calendar show as a generic "busy" or "unavailable". I.e. don't broadcast the details of the events from other calendars.
3. Have a centralized view somewhere of all the events without duplicates. I.e. without a bunch of "busy" or "unavailable" events.
4. Ideally, works with both Google and Microsoft calendars

)

The easiest way to achieve this, of course, is to do it yourself. When you book an event on one calendar, you go through the motions of creating a generic event on the other two. This meets all the criteria but depending on how heavily your client leans into the "all meetings/all the time" project management style, this gets old real fast. But to be fair, it's served me reasonably well for many years.

This led naturally to a search for a technical solution, which yielded the following options:

• OneCal
• CalendarBridge
• Fantastical
• Calendly
• Reclaim.ai
• Spike
• Syncing calendars

I have OneCal at the top because it's (currently) my favorite option. But I'll skim through the rest and provide reasons why I didn't go with them. I'll use my tried-and-true practice of claiming that my choice is the One True Way™ and everyone else is wrong as a means of encouraging people to comment with their own options, if only to prove me wrong. Which I'm quite happy to be. But still, that's not the case here. I'm right and you're wrong. Comment below if you disagree.

Back to the evaluations. I'll start with some of the quicker ones.

Syncing calendars

I'll be honest, I just kind of skimmed this article. I have kind of a vague idea of what they're going for but this seems geared more toward people that are managing someone else's calendar. Plus I don't think it meets the criteria of actually booking the time off on the other calendar. Either way, it looks kind of cumbersome to set up and to manage.

Spike

I didn't give this much more than a glance. It does way more than I want it to and calendar sync doesn't appear to be among their more popular features. In any case, the banner says "email and chat app" and the byline talks about teams and partners. I don't think I'm their target audience.

Reclaim.ai

Also has many features I don't need nor want though at least it's focused on calendars. Google Calendar specifically. Outlook is coming soon. So while calendar sync is one of the highlighted features, the lack of Outlook, as well as the focus on collaboration and AI and "smart" also suggests it's overkill.

Calendly

I've used Calendly before to send people links to schedule calls with me. It's a lovely app and the free version is quite powerful. I'd recommend it even over Google Calendar's built-in "scheduling slots" or "appointment schedules" or whatever they're calling the feature now.

That said, based on what I read, I don't think they offer what I need. It looks like the paid plans will let you connect to multiple calendars so that you can give someone a link with a more comprehensive view of your availability. But I need something for internal people who have direct access to my calendar.

Fantastical

This is just plain a gorgeous app, if you'll forgive an awkwardly-worded almost-oxymoron. I keep threatening to use it every couple of years but can't really justify it. The major selling points are syncing across multiple devices, the natural language parsing, the UX, and integration with a lot of other productivity tools I don't use. Seriously, you should check this app out...

...except if all you need is to sync events across multiple calendars. I'm still in the midst of testing the app but the process to ensure an event appears as a block of unavailable time on multiple calendars seems to be to duplicate it. Which, to be fair, they make really easy. And if you keep the details the same in all the clones, the unified calendar view removes the duplicates. But that doesn't meet criteria #2 for me above. And if you change the name, it shows as a separate event which is a bit clumsy. For this feature specifically, it's not much better than what I can achieve in Google Calendar natively. But still, give it a whirl.

Final verdict: OneCal (CalendarBridge runner up)

That leaves two options: OneCal and CalendarBridge. Both do the same thing though their respective marketing teams might squabble over technicalities. Their primary purpose is to sync multiple calendars and avoid double-bookings. Perfect.

Both services sync one calendar to another by cloning events between them and monitoring the source calendar for new events. Both give you control over what information you want to copy over and have similar options like excluding events of a given colour or excluding events where you're marked free (useful if you track birthdays in your calendar). Once syncing was set up, my calendars looked pretty much exactly how they did when I was doing it manually.

Neither service has a mobile app from what I can tell which is a shame, especially in the case of OneCal because unlike CalendarBridge, they offer a unified calendar view of all your calendars with the option to hide the cloned events. This is a nice touch and while the view is mobile friendly, it's missing some features, like a three-day view or the ability to put a widget on your phone.

Pricing is virtually identical on both services in that the one I would need runs about $100/year to sync three calendars. OneCal has the ability to do two-way syncs between calendars or to broadcast a one-way sync from one calendar to multiple other calendars, something that I don't think CalendarBridge does. In CalendarBridge, a two-way sync is done with two one-way syncs and their pricing reflects that. The basic plan for both essentially covers two calendars so I'd be looking at premium for both.

Setting up the syncs seemed nicer in OneCal to me though I can't 100% say why. Both essentially walk you through a three-ish-step wizard to do it. Maybe OneCal worded things better or laid things out more intuitively for me. I found myself wondering if I was doing it right a couple of times in CalendarBridge. OneCal also felt zippier in general.

Both apps give you booking links (a la Calendly) which is nice if you need that. OneCal apparently integrates with Zoom to automatically add a meeting link to appointments booked online but I didn't test that out since I don't need it. Neither requires a credit card to sign up which is an underrated feature these days.

A couple of little UX things. First, OneCal has another underrated feature: a big ol' Delete My Account button at the bottom of the settings page. This is great for someone who wants to test things out, decide it's not for them, and move on. I'm not planning to use CalendarBridge and I've cancelled my free trial but the account remains. There's no obvious way to delete it so I guess I'm a CalendarBridge user indefinitely now. I suppose that's good for them to pump up their user numbers in case someone wants to buy them out but doesn't do me any good.

Second thing is CalendarBridge's login mechanism. To log in, I enter my email address, they email me a code, I enter the code. It's basically the standard 2FA mechanism from fifteen years ago. There's no password, no Google/Apple/Facebook/etc option. It's kinda weird. Like the developers have a personal agenda against traditional authentication and this is their manifesto.

So both apps do what I want and cost roughly the same but I give the edge to OneCal for their usability and their unified Calendar view. The price point is high enough that I'll give it a few days before pulling the trigger so I can decide if I want this more than I want to watch seasons 2 of Silo and Severance.

The alternative is to set up an auto-decline automation for all meetings which I haven't fully taken off the table yet.

For the longest time, the most commonly-used app on my phone by at least an order of magnitude was Crosswords Classic by Stand Alone, Inc. (They have a newer version that's probably better but every time I've tried it, there's a "who moved my cheese" factor I've never really cared to overcome.) When I had an Android device, it was Shortyz Crosswords. Both excellent and both worth whatever price I paid for them (I've long forgotten).

However, in my advancing years, I've taken to the printed page more and more and in the last year or so, I've been printing crosswords and doing them in pen and paper instead. This was a good move for two reasons: 1) My screen time dropped by more than half, and 2) it opened up several crossword providers that, to my knowledge, aren't available in the apps. For reasons unknown to me, many crossword providers are not making their work available to these apps (e.g. USA Today, Washington Post, and New York Times). In fact, several of them have set up their own payment systems to subscribe to them (e.g. American Values, Matt Gaffney, Elizabeth Gorski).

The upside is that now I have a whole new world of crosswords in an arguably more leisure form. The downside is why you're reading this post: now I have to hunt and peck my way through half a dozen providers, using different delivery mechanisms, rather than have an app just go out and get them. My workflow for this used to be:

1. As emails come in from email providers, save the PDFs in a special folder
2. For others:
   • Navigate to the website
   • Navigate to each day
   • Download the PDF
3. Once I have everything collected, merge the crosswords into a single PDF and print
4. Archive the ones I merged
5. Repeat when I run out of crosswords

Fun fact: Step 3 isn't as onerous as you might think. I spent a good half hour building an Automator action in MacOS to do this until I found out the functionality is built into the right-click menu.

As you can imagine, this virtually reeks of needing automation. So being a good developer, I designed in my head a system that would meet my needs:

• An ASP.NET Core-based app with a React front-end
• Authentication with multiple providers (Google, Facebook, etc.)
• Azure storage to store the PDFs
• Azure Logic Apps to parse emails that I send to a special address
• Azure function triggered nightly to download from non-email providers
• Some (hopefully free) PDF library to merge the PDFs
• A utility to parse XML into an image to be exported as a PDF

(That last one is mostly for USA Today but basically any puzzle provided by uclick.com. They have a special XML format that describes crosswords.)

Let's ignore the cost-benefit of building all this vs. the 10 minutes each week I spend on it manually and chalk it up to: because it's there

I'll skip ahead a bit so I can get to my point. I was a couple of days into this when I remembered a Reddit post I had read recently where someone had automated the download of New York Times crosswords (provided you had a subscription, of course) using a bash script. Then it occurred to me: Do I need all this infrastructure? Azure storage, hosting, logic apps, functions? Could I expand on this bash script from this random, helpful stranger instead?

Skipping ahead a bit further and the answers are, of course: no I don't, and yes I can. The new stack:

• bash
• Zapier
• Mailparser.io
• C# console app
• XML/XSLT/HTML/CSS Grid
• Google Chrome (headless)
• The built-in MacOS functionality I mentioned earlier

I'm using Zapier/Mailparser.io to handle the email providers, C#/XML/XSLT for the uclick providers to transform the XML into HTML (there's very little C#), Chrome to print that HTML to PDF, built-in MacOS to merge the PDFs, and bash to tie it all together.

Here's my new workflow:

1. Run the bash script
2. Print
3. Solve
4. Repeat when I run out

... and the only reason I haven't automated #2 is because I haven't looked into the lp command yet and I like to spot-check the NYT puzzles before I print them.

Now all this is not polished by any definition of the word and it is highly customized to my environment. But by gum, it works. And it still gave me the same rush I've always felt when I've been working at something and it eventually works. Figuring out Zapier's and MailParser's rule systems, re-learning XSLT, seeing HTML files disappearing and working PDFs taking their place. Getting into the minutiae of something that should be easy (like URI-decoding text in XSLT 1.0) but isn't. Not to say, "How can I learn Azure?" but to break a problem down, find a tool, and execute until it works. This is the same routine I used almost 40 years ago when I thought to myself, "I wonder if I can make it say, 'TAKE THAT LOSERS!!!' whenever I score a touchdown" on the old ASCII-based football game in my dad's office computer they let me sit at while waiting for a dentist appointment.

It's likely I'll be the only one who uses this thing, and to be honest, I hope I am because the idea of supporting it gives me the shakes. My original design was certainly more ambitious and might be easier to turn into a conference talk. But it's overkill. Maybe there's a market for applications that collect and manage crossword puzzle PDFs; I'm past the point where I want to build an application that meets that need first and my personal need second.

Kyle the Unsolveable

In the article, the author describes their Mr. Destiny moment where they wish they had gone into management rather than sticking with being "just" a programmer. Thus perpetuating the myth in our industry that you aren't worth anything unless you Change The World™.

To be fair, the author is careful not to be prescriptive. It's very much "these are my regrets and thoughts" and not "you should do this in the same situation". I'm thankful for that and I'll pay the same courtesy. These are my thoughts as they apply to me personally. If you can relate to it in any way, that's not my fault.

Twenty years ago, I suppose I was at my own crossroads though I didn't really recognize it as such until I thought about it just now. Ever since I was in grade 8, I was going to be an actuary. At the time, I was told "you have to be good at math and you'll make a lot of money". What else do you need to hear when you're in grade 8?

So I set about my goal and got a degree, a Bachelor of Commerce I believe. I wasn't really sure what to make of the Faculty of Management, what with their wine and cheeses and people coming to class in suits and "networking" but whatever, I did the work, finished my studies, and started interviewing like you were supposed to.

At some point in that first six months of interviewing, I clued in to something: programming, which I had been doing for about 10 years by that point as a hobby, was a career choice! Was it a lucrative one? Was it better than being an actuary? Who cares, it was fun and people got paid to do it! So I went back to school. That is, to my ageing memory, the sum total of the thought I put into it. (This will be a recurring theme throughout this post.)

So I did it. I went back to school, got the degree, and started working. My first job was in the corporate world, an oil and gas company. My next, with a startup where I interviewed badly and they hired me anyway. The third was with a consulting company. Since then: contracting, my own startup, and employeehood in some order. Along the way, I've learned, to varying degrees, VB6, classic ASP, .NET, RPG, Livelink, Sharepoint, JavaScript, Ruby, Java, SQL Server, Azure, Google Web Toolkit, Docker, CI, CD, CQRS, CORS, CSS and more acronymed software methodologies and techniques than I care to put thought into remembering admit. I've blogged (clearly), co-written a book, spoken at conferences, and created a user group that was the first of its kind in the country. (It lasted less than a year.)

While you're free to copy and paste all this into my obituary, there's a reason I list it out. I have never put much thought into my career and I don't have any intrinsic itch I've been meaning to scratch. How did I choose when and where to do all of these things? The opportunities came up and I said yes. Again, not much more thought goes into it than that.

I have no regrets (including the book). Certainly not on a macro level. Should I have pursued a career as an actuary which, in all likelihood, would have ended up more lucrative financially? Was it a waste of time learning Livelink? Should I have gone into a managerial role?

Does it matter?

I am where I am now because of the choices I've made. I've often joked that the main criteria I use to choose my contracts/positions is whether I think it'll be fun. Not whether it was in a hot technology or a dynamic industry. I'm too lazy to figure out what constitutes either of those.

The net result of those criteria is that I can look back favourably on a career (and I use that term loosely) as "just a programmer". One that I have no intention of leaving because why would I?

I am, and I can't emphasize this enough, having the time of my life.

Now to be fair, I've always had the power to see things through a pair of glasses a supernatural shade of rose. But consider the language from the post I mentioned earlier:

I can only imagine how in demand I could have been.

My sister has 10X the assets I have.

And my favourite one:

And today I am still just a programmer. Who’s the weenie now?

To me, this speaks of a chronic problem of chasing the wrong thing. What you're supposed to do versus what you want to do. And I've often wondered, if you had taken that other path, would you still have no regrets?

The world is filled with people who want to leave their mark, some of whom actually will. And I think it's great that these people exist. We need them. For my part, I see no point in feeling guilt or regret doing something I love doing in an industry that is privileged enough to provide me with a good living doing it.

Death to the Batch Job

In addition to the slides, you can find the source code for the presentation here. There's also a blog post that expands a little on the ideas.

Docker

The two sample apps I showed during the presentation are:

• AzureCodeCamp
• GenFu

For the first, I showed how to connect the app to PostgreSQL which can be running either locally or in Docker. For the second, I showed how to create a container for a .NET Core application running in Linux. If you want to play with this container, you can get it here on Docker Hub.

Pre-game Show

In addition to the presentations, Donald and I (but mostly me), with the help of the Winnipeg .NET User Group, hosted a fishbowl discussion and an evening of drinks and food the night before the conference courtesy of Particular Software. During the talk, we covered ways to build a solid team, the importance of company culture, and good ways to vet people. It was a great chat and it was nice to see so many people weigh in with their experience.

I'd like to thank...

It's hard to overstate the value this "big little conference" has for me. It has all the benefits of a larger conference while still being accessible to both the attendees and speakers. D'Arcy Lussier is a force to be reckoned with in the conference circuit.

Many thanks to the Winnipeg .NET User Group for their help with the fishbowl event.

Finally, a big thank you to those of you who came to the the conference itself and continue making it great.

I've talked on this subject before and it's something I often have in the back of my mind when reading comments. It's come to the forefront recently with some conversations we've had at Western Devs, which led to our most recent podcast. I wasn't able to attend so here I am.

There are certain phrases you see in comments that automatically seem to devolve a discussion. They include:

• "Why don't you just..."
• "Sorry but..."
• "Can't you just..."
• "It's amazing that..."

Ultimately, all of these phrases can be summarized as follows:

In my younger years, I could laugh this off amiably and say "Oh this wacky world we live in". But I'm turning 44 in a couple of days and it's time to start practicing my crotchety, even if it means complaining about people being crotchety.

So to that end: I'm asking, nay, begging you to avoid these and similar phrases. This is for your benefit as much as the reader's. These phrases don't make you sound smart. Once you use them, it's very unlikely anyone involved will feel better about themselves, let alone engage in any form of meaningful discussion. Even if you have a valid point, who wants to be talked down to like that? Have you completely forgot what it's like to learn?

Now I know (from experience) it's hard to fight one's innate sense of superiority and the overwhelming desire to make it rain down on the unwashed heathen. So take it in steps. After typing your comment, remove all instances of "just" (except when just means "recently" or "fair", of course). The same probably goes for "simply". It has more of a condescending tone than a dismissive one. "Actually" is borderline. Rule of thumb: Don't start a sentence with it.

Once you have that little nervous tic under control, it's time to remove the negatives. Here's a handy replacement guide to get you started:

See the difference? Instead of saying Sweet Zombie Jayzus, you must be the stupidest person on the planet for doing it this way, you've changed the tone to Have you considered this alternative? In both instances, you've made your superior knowledge known but in the second, it's more likely to get acknowledged. More importantly, you're less likely to look like an idiot when the response is: I did consider that avenue and here are legitimate reasons why I decided to go a different route.

To be fair, sometimes the author of the work you're commenting on needs to be knocked down a peg or two themselves. I have yet to meet one of these people who respond well to constructive criticism critique, let alone the destructive type I'm talking about here. Generally, I find they feel the need to cultivate an antagonistic personality but in my experience, they usually don't have the black turtlenecks to pull it off. Usually, it ends up backfiring and their dismissive comments become too easy to dismiss over time.

Kyle the Inclusive

Excerpts

Jekyll and Hexo handle excerpts very differently. In Jekyll, you can specify an excerpt in the front matter. If one isn't provided, it will use the first few characters of the post itself as the excerpt.

In Hexo, the excerpt is set in one and only one way. You need to add <!--more--> somewhere in the post. Everything before that is considered the excerpt. If you don't have that tag in your post, no excerpt.

Luckily, Amir had some foresight and added code to our Jekyll site to support <!--more-->. Unluckily, we rarely used it. Some of our posts had excerpts, particularly the podcasts. But mostly, we relied on Jekyll to parse the post for the excerpt.

In the end, we did two things to remedy this. First we added the https://github.com/lalunamel/hexo-front-matter-excerpt package. This fixed all the posts that already had an excerpt in the front matter. For the rest, we went through all the remaining posts and added <!--more--> after the first paragraph. Ever the optimists, we also submitted a pull request for the package to make it behave more like Jekyll.

Syntax highlighting

A code block in Jekyll looks like this

1
2
3

{% highlight html%}
<div>moo</div>
{% endhighlight %}

And in Hexo, like this:

1
2
3

{% codeblock lang:html %}
<div>moo</div>
{% endcodeblock %}

A series of "Find and replace in files" actions took care of this. We also found a syntax.styl file to our liking. Not sure if Jekyll has built-in support for syntax highlighting. I suspect not and that it was automatically included with the theme we chose for Jekyll.

CSS styles

Jekyll uses Kramdown for parsing Markdown. One of the features it adds is the ability to specify a class on your elements like so.

1
2

{: .pull-right }
![My image](http://my/image/reference)

This will add the pull-right CSS class to the image and, as the name suggests, this will float the image on the right.

As far as I know, there's no Kramdown support in NodeJS, likely because Kramdown is a Ruby gem. So we had to modify each instance of this syntax manually. For images, we converted to the Hexo img tag plugin like so:

1

{% img pull-right "http://my/image/reference" "My image"}

There are a few tag plugins in Hexo all ported from Octopress. Which likely means we could have used them in our Jekyll site but not much point in testing that out now...

For other instances of this kramdown-specific syntax that were applied to other elements, we just dropped down to native HTML in the Markdown:

1

<div class="notice">This is a notice</div>

Aliases

We changed the URL structure for our podcasts early on and to keep any links pointing to the existing ones valid, we used the jekyll-redirect-from gem to maintain the old link. We find a suitable replacement for it in Hexo. BUT that suitable replacement's official package on NPM doesn't support Hexo 3 at the time of writing so we had to reference it in our package.json file like so:

1

"hexo-generator-alias": "git+https://github.com/hexojs/hexo-generator-alias.git"

Bear in mind that this is a static site. So both of these plugins, the Jekyll and the Hexo one, handle redirects by generating an HTML page at the alias location with a redirect embedded in it:

1
2
3
4
5
6
7
8

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Redirecting...</title>
    <link rel="canonical" href="/podcasts/podcast-the-internet-of-things/"><meta http-equiv="refresh" content="0; url=/podcasts/podcast-the-internet-of-things/">
  </head>
</html>

This isn't quite ideal for SEO purposes but it is the recommended approach if you can't do a server-side redirect.

Empty categories

This issue was probably the source of most of our trouble. Our permalink URL is of the form /:category/:title. For example: /docker/yet-another-docker-intro/. The issue is that the vast majority of our posts are uncategorized. And Jekyll and Hexo each handle uncategorized posts very differently in their permalinks.

In the front matter for Hexo, you can define a default category which all posts will use if a category is not assigned. So if you set the default category to "moo", your permalink URL will be: /moo/a-discussion-on-knockout/.

In Jekyll, an uncategorized post appears at the root of the site. Like this: http://www.westerndevs.com/Migrating-from-Jekyll-to-Hexo-Part-2. But setting the default category to an empty string or to /, led to fully-qualified URLs that look like this: //Migrating-from-Jekyll-to-Hexo-Part-2. I believe this is a bug in the permalink generator. It's generating the permalink as /category/title and when there is no category, it's not going to treat it any differently.

Our initial solution to this was to stick with a default category of uncategorized but to alias each post to the root so that existing URLs would still work but would redirect to the new one. Alas, we have Disqus comments to deal with and those are tied to a specific URL. We could have migrated them but we also would have had to contend with sucky looking URLs that have uncategorized in them.

Our current solution: fork Hexo. In our fork, we add some handling when generating the URL for a post so that if it starts with a //, we trim one of them. Not the most elegant solution but workable for now. And if that's not hacky enough for you, check out the solution for...

Permalinks/Disqus Comments

After deployment, we discovered none of the existing Disqus comments were showing on our posts. The Disqus script was working because you could add new comments and it would show "Also on Western Devs" comments. Just no existing comments.

The culprit was, again, the permalink. Because of the leading //, Disqus thought the URL for our posts was (as an example): http://www.westerndevs.com//a-discussion-on-knockout. That double slash was enough to confuse it into not showing comments.

Our interim solution is to include this in our post template:

1

var disqus_url = '<%= page.permalink %>'.replace(".com//", ".com/");

That right there is some quality coding...

Feed/Sitemap/iTunes

This issue is more of a warning not to over-think things. Our feed is served up at http://www.westerndevs.com/feed. In Jekyll, we created a feed.xml file and assumed that it did some magic to generate a feed file. And we had a hell of a time trying to get Hexo to generate that same file without the .xml extension. Even to the point where we created our own package that still didn't quite work.

Then someone realized that Jekyll was not actually generating a feed file, just a feed.xml file. Whatever GitHub Pages runs on allows you to access XML files without the extension. Jekyll's development server knows this. Hexo's doesn't. So when we run locally, we can't access our various XML files (for our RSS feed, our iTunes feed, and our sitemap) without specifying the extension. But on the deployed site, they work fine.

Links in Headers

There was a bug in the default Markdown renderer that caused rendering issues when you had links within headers. The bug has since been fixed but I'm including it here mostly to reiterate that you aren't locked into anything with Hexo. While we were converting, the bug was still active and all we did to fix it was to switch to a different renderer.

One philosophical point: One of the reasons for this bug is that the default renderer will automatically add bookmarks for every heading. This is a design decision that I'm not convinced has a lot of value though it's not a position I'll defend strongly. The renderer we've switched to includes this as an option you have to explicitly turn on which I think is a better way to go.

Deployment

Since GitHub Pages runs Jekyll, it's understandable that deployment with Jekyll is relatively easy. We did need to wrap it in a Rake task to accommodate working locally with a different _config.yml file than what is used in production but that wasn't hard to do.

For Hexo, our process is still mostly manual and has to be done locally rather than on our CI server (Travis). We generate the site, copy the static files to a folder, then check those files into the gh-pages branch. I imagine it won't be hard to get our Travis process adapted to this but at present, we haven't got around to it yet. Just keep in mind, it won't be quite as easy as Jekyll.

So which one is "better"?

I forgot to answer this in the last post. As I mentioned, we're enjoying Hexo and it's nice having everyone excited about blogging again. For the Western Devs as a group, Hexo is the better choice.

But personally, I like Jekyll a little better. It feels more polished and doesn't require you to fork the product to get what you want. Typing this up, I was looking for a way to disable line numbers for individual codeblocks but I've come up short. Plus there's a larger and more comprehensive community behind it. That said, I am really enjoying the quick generation time in Hexo.

I believe that covers the major gotchas we encountered in our conversion. It glosses over a few things, like whether to stick with Stylus as the default CSS pre-processor or move to SASS. Or whether to use Jade as the templating language rather than the default, EJS. Those questions are quite a bit more subjective and I wanted to keep this discussion limited to the technical hurdles we encountered.

But if you run into an issue not mentioned here, add a comment and I will update the post.

Regardless, here we are and it's already been a positive move overall, not least because the move to Node means more of us are available to help with the maintenance of the site. But it wasn't without it's challenges. So I'm going to outline the major ones we faced here in the hopes that it will help you make your decision more informed than ours was.

To preface this, note that I'm new to Node and in fact, this is my first real project with it. That said, I'm no expert in Ruby either, which is what Jekyll is written in. And the short version of my first impressions is: Jekyll feels more like a real product but I had an easier time customizing Hexo once I dug into it. Here's the longer version

Documentation/Resources

You'll run into this very quickly. Documentation for Hexo is decent but incomplete. And once you start Googling, you'll discover many of the resources are in Chinese. I found very quickly that there is posts collection and that each post has a categories collection. But as to what these objects look like, I couldn't tell. They aren't arrays. And you can't JSON.stringify them because they have circular references in them. util.inspect works but it's not available everywhere.

Multi-author support

By default, Hexo doesn't support multiple authors. Neither does Jekyll, mind you, but we found a pretty complete theme that does. In Hexo, there's a decent package that gets you partway there. It lets you specify an author ID on a post and it will attach a bunch of information to it. But you can't, for example, get a full list of authors to list on a Who We Are page. So we created a separate data file for the authors. But we also haven't figured out how to use that file to generate a .json file to use for the Featured section on the home page. So at the moment, we have author information in three places. Our temporary solution is to disallow anyone from joining or leaving Western Devs.

Customization

If you go with Hexo and choose an existing themes, you won't run into the same issues we did. Out of the box, it has good support for posts, categories, pagination, even things like tags and aliases with the right plugins.

But we started from a design and were migrating from an existing site with existing URLs and had to make it work. I've mentioned the challenge of multiple authors already. Another one: maintaining our URLs. Most of our posts aren't categorized. In Jekyll, that means they show up at the root of the site. In Hexo, that's not possible. At least at the moment and I suspect this is a bug. We eventually had to fork Hexo itself to maintain our existing URLs.

Another challenge: excerpts. In Jekyll, excerpts work like this: Check the front matter for an excerpt. If one doesn't exist, take the first few characters from the post. In Hexo, excerpts are empty by default. If you add a <!--more--> tag in your post, everything before that is considered an excerpt. If you specify an excerpt in your front matter, it's ignored because there is already an excerpt property on your posts.

Luckily, there's a plugin to address the last point. But it still didn't address the issue of all our posts without an excerpt where we relied solely on the contents of the post.

So if you're looking to veer from the scripted path, be prepared. More on this later in the "good parts" section.

Overall feeling of rawness

This is more a culmination of the previous issues. It just feels like Hexo is a work-in-progress whereas Jekyll feels more like a finished product. There's a strong community behind Jekyll and plenty of help. Hexo still has bugs that suggest it's just not used much in the wild. Like rendering headers with links in them. It makes the learning process a bit challenging because with Jekyll, if something didn't work, I'd think I'm obviously doing something wrong. With Hexo, it's I might be doing something wrong or there might be a bug.

The good parts

I said earlier that the move to Hexo was positive overall and not just because I'm optimistic by nature. There are two key benefits we've gained just in the last two weeks.

Generation time

Hexo is fast, plain and simple. Our old Jekyll site took six seconds to generate. Doesn't sound like much but when you're working on a feature or tweaking a post, then saving, then refreshing, then rinsing, then repeating, that six seconds adds up fast. In Hexo, a full site generation takes three seconds. But more importantly, it is smart enough to do incremental updates while you're working on it. So if you run hexo server, then see a mistake in your post, you can save it, and the change will be reflected almost instantly. In fact, it's usually done by the time I've switched back to the browser.

Contributors

We had logistical challenges with Jekyll. To the point where we had two methods for Windows users that wanted to contribute (i.e. add a post). One involved a Docker image and the other Azure ARM. Neither was ideal as they took between seconds and minutes to refresh if you made changes. Granted, both methods furthered our collective knowledge in both Docker and Azure but they both kinda sucked for productivity.

That meant that realistically, only the Mac users really contributed to the maintenance of the site. And our Docker/Azure ARM processes were largely ignored as we would generally just test in production. I.e. create a post, check it in, wait for the site to deploy, make necessary changes, etc, etc.

With the switch to Hexo, we've had no fewer than five contributors to the site's maintenance already. Hexo just works on Windows. And on Mac. Best of both worlds.

Customization

This is listed under the challenges but ever the optimist, I'm including it here as well. We've had to make some customizations for our site, including forking Hexo itself. And for me personally, once I got past the why isn't this working the way I want? stage, it's been a ton of fun. It's crazy simple to muck around in the node modules to try stuff out. And just as simple to fork something and reference it in your project when the need arises. I mentioned an earlier issue rendering links in headers. No problem, we just swapped out the markdown renderer for another one. And if that doesn't work, we'll tweak something until it does.

I want to talk more on specific conversion issues we ran into as a guide to those following in our footsteps. But there are enough of them to warrant a follow up post without all this pre-amble. For now, we're all feeling the love for Hexo. So much so that no less than three other Western Devs are in the process of converting their personal blogs to it.

I was asked recently for some advice on generating data for use with integration and UI tests. I already have some ideas but asked the rest of the Western Devs for some elucidation. My tl;dr version is the same as what I mentioned in our discussion on UI testing: it's hard. But manageable. Probably.

The solution needs to balance a few factors:

• Each test must start from a predictable state
• Creating that predictable state should be fast as possible
• Developers should be able to figure out what is going on by reading the test

The two options we discussed both assume the first factor to be immutable. That means you either clean up after yourself when the test is finished or you wipe out the database and start from scratch with each test. Cleaning up after yourself might be faster but has more moving parts. Cleaning up might mean different things depending on which step you're in if the test fails.

So given that we will likely re-create the database from scratch before each and every test, there are two options. My current favourite solution is a hybrid of the two.

Maintain a database of known data

In this option, you have a pre-configured database. Maybe it's a SQL Server .bak file that you restore before each test. Maybe it's a GenerateDatabase method that you execute. I've done the latter on a Google App Engine project, and it works reasonably well from an implementation perspective. We had a class for each domain aggregate and used dependency injection. So adding a new test customer to accommodate a new scenario was fairly simple. There are a number of other ways you can do it, some of which Simon touched on in his post.

We also had it set up so that we could create only the customer we needed for that particular test if we needed to. That way, we could use a step like Given I'm logged into 'Christmas Town' and it would set up only that data.

There are some drawbacks to this approach. You still need to create a new class for a new customer if you need to do something out of the ordinary. And if you need to do something only slightly out of the ordinary, there's a strong tendency to use an existing customer and tweak its data ever so slightly to fit your test's needs, other tests be damned. With these tests falling firmly in the long-running category, you don't always find out the effects of this until much later.

Another drawback: it's not obvious in the test exactly what data you need for that specific test. You can accommodate this somewhat just with a naming convention. For example, Given I'm logged into a company from India, if you're testing how the app works with rupees. But that's not always practical. Which leads us to the second option.

Create an API to set up the data the way you want

Here, your API contains steps to fully configure your database exactly the way you want. For example:

1
2
3
4
5

Given I have a company named "Christmas Town" owned by "Jack Skellington"
And I have 5 product categories
And I have 30 products
And I have a customer
...

You can probably see the major drawback already. This can become very verbose. But on the other hand, you have the advantage of seeing exactly what data is included which is helpful when debugging. If your test data is wrong, you don't need to go mucking about in your source code to fix it. Just update the test and you're done.

Also note the lack of specifics in the steps. Whenever possible, I like to be very vague when setting up my test data. If you have a good framework for generating test data, this isn't hard to do. And it helps uncover issues you may not account for using hard-coded data (as anyone named D'Arcy O'Toole can probably tell you).

Loading up your data with a granular API isn't realistic which is why I like the hybrid solution. By default, you pre-load your database with some common data, like lookup tables with lists of countries, currencies, product categories, etc. Stuff that needs to be in place for the majority of your tests.

After that, your API doesn't need to be that granular. You can use something like Given I have a basic company which will create the company, add an owner and maybe some products and use that to test the process for creating an order. Under the hood, it will probably use the specific steps.

One reason I like this approach: it hides only the details you don't care about. When you say Given I have a basic company and I change the name to "Rick's Place", that tells me, "I don't care how the company is set up but the company name is important". Very useful to help narrow the focus of the test when you're reading it.

This approach will understandably lead to a whole bunch of different methods for creating data of various sizes and coarseness. And for that you'll need to...

Maintain test data

Regardless of your method, maintaining your test data will require constant vigilance. In my experience, there is a tremendous urge to take shortcuts when it comes to test data. You'll re-use a test company that doesn't quite fit your scenario. You'll alter your test to fit the data rather than the other way around. You'll duplicate a data setup step because your API isn't discoverable.

Make no mistake, maintaining test data is work. It should be treated with the same respect and care as the rest of your code. Possibly more so since the underlying code (in whatever form it takes) technically won't be tested. Shortcuts and bad practices should not be tolerated and let go because "it's just test data". Fight the urge to let things slide. Call it out as soon as you see it. Refactor mercilessly once you see opportunities to do so.

Don't be afraid to flip over a table or two to get your point across.

-- Kyle the Unmaintainable

Using the sample app with PostgreSQL

The sample application is just a plain ol' .NET application. It is meant to showcase different ways of doing things. One of those things is data access. You can configure the app to access the data from SQL storage, Azure table storage, or in-memory. By default, it uses the in-memory option so you can clone the app and launch it immediately just to see how it works.

Quick summary: Calgary, Alberta hosts an annual event called the Calgary Stampede. One of the highlights of the 10-ish day event is the pancake breakfast, whereby dozens/hundreds of businesses offer up pancakes to people who want to eat like the pioneers did, assuming the pioneers had pancake grills the size of an Olympic swimming pool.

The sample app gives you a way to enter these pancake breakfast events and each day, will show that day's breakfasts on a map. There's also a recipe section to share pancake recipes but we won't be using that here.

To work with Docker we need to set the app up to use a data access mechanism that will work on Docker. The sample app supports Postgres so that will be our database of choice. Our first step is to get the app up and running locally with Postgres without Docker. So, assuming you have Postgres installed, find the ContainerBuilder.cs file in the PancakeProwler.Web project. In this file, comment out the following near the top of the file:

1
2
3
4

// Uncomment for InMemory Storage
builder.RegisterAssemblyTypes(typeof(Data.InMemory.Repositories.RecipeRepository).Assembly)
       .AsImplementedInterfaces()
       .SingleInstance();

And uncomment the following later on:

1
2
3

// Uncomment for PostgreSQL storage
builder.RegisterAssemblyTypes(typeof(PancakeProwler.Data.Postgres.IPostgresRepository).Assembly)
    .AsImplementedInterfaces().InstancePerRequest().PropertiesAutowired();

This configures the application to use Postgres. You'll also need to do a couple of more tasks:

• Create a user in Postgres
• Create a Pancakes database in Postgres
• Update the Postgres connection string in the web project's web.config to match the username and database you created

The first two steps can be accomplished with the following script in Postgres:

1
2
3
4
5

CREATE DATABASE "Pancakes";

CREATE USER "Matt" WITH PASSWORD 'moo';

GRANT ALL PRIVILEGES ON DATABASE "Pancakes" TO "Matt";

Save this to a file. Change the username/password if you like but be aware that the sample app has these values hard-wired into the connection string. Then execute the following from the command line:

psql -U postgres -a -f "C:\path\to\sqlfile.sql"

At this point, you can launch the application and create events that will show up on the map. If you changed the username and/or password, you'll need to update the Postgres connection string first.

You might have noticed that you didn't create any tables yet but the app still works. The sample is helpful in this regard because all you need is a database. If the tables aren't there yet, they will be created the first time you launch the app.

Note: recipes rely on having a search provider configured. We won't cover that here but I hope to come back to it in the future.

Next, we'll switch things up so you can run this against Postgres running in a Docker container.

Switching to Docker

I'm going to give away the ending here and say that there is no magic. Literally, all we're doing in this section is installing Postgres on another "machine" and connecting to it. The commands to execute this are just a little less click-y and more type-y.

The first step, of course, is installing Docker. At the time of writing, this means installing Docker Machine.

With Docker Machine installed, launch the Docker Quickstart Terminal and wait until you see an ASCII whale:

If this is your first time running Docker, just know that a lightweight Linux virtual machine has been launched in VirtualBox on your machine. Check your Start screen and you'll see VirtualBox if you want to investigate it but the docker-machine command will let you interact with it for many things. For example:

docker-machine ip default

This will give you the IP address of the default virtual machine, which is the one created when you first launched the Docker terminal. Make a note of this IP address and update the Postgres connection string in your web.config to point to it. You can leave the username and password the same:

1

<add name="Postgres" connectionString="server=192.168.99.100;user id=Matt;password=moo;database=Pancakes" providerName="Npgsql" />

Now we're ready to launch the container:

docker run --name my-postgres -e POSTGRES_PASSWORD=moo -p 5432:5432 -d postgres`

Breaking this down:

If this is the first time you've launched Postgres in Docker, it will take a few seconds at least, possibly even a few minutes. It's downloading the Postgres image from Docker Hub and storing it locally. This happens only the first time for a particular image. Every subsequent postgres container you create will use this local image.

Now we have a Postgres container running. Just like with the local version, we need to create a user and a database. We can use the same script as above and a similar command:

psql -h 192.168.99.100 -U postgres -a -f "C:\path\to\sqlfile.sql"

The only difference is the addition of -h 192.168.99.100. You should use whatever IP address you got above from the docker-machine ip default command here. For me, the IP address was 192.168.99.100.

With the database and user created, and your web.config updated, we'll need to stop the application in Visual Studio and re-run it. The reason for this is that the application won't recognize that we've changed database so we need to "reboot" it to trigger the process for creating the initial table structure.

Once the application has been restarted, you can now create pancake breakfast events and they will be stored in your Docker container rather than locally. You can even launch pgAdmin (the Postgres admin tool) and connect to the database in your Docker container and work with it like you would any other remote database.

Next steps

From here, where you go is up to you. The sample application can be configured to use Elastic Search for the recipes. You could start an Elastic Search container and configure the app to search against that container. The principle is the same as with Postgres. Make sure you open both ports 9200 and 9300 and update the ElasticSearchBaseUri entry in web.config. The command I used in the presentation was:

docker run --name elastic -p 9200:9200 -p 9300:9300 -d elasticsearch

I also highly recommend Nigel Poulton's Docker Deep Dive course on Pluralsight. You'll need access to Linux either natively or in a VM but it's a great course.

There are also a number of posts right here on Western Devs, including an intro to Docker for OSX, tips on running Docker on Windows 10, and a summary or two on a discussion we had on it internally.

Other than that, Docker is great for experimentation. Postgres and Elastic Search are both available pre-configured in Docker on Azure. If you have access to Azure, you could spin up a Linux VM with either of them and try to use that with your application. Or look into Docker Compose and try to create a container with both.

For my part, I'm hoping to convert the sample application to ASP.NET 5 and see if I can get it running in a Windows Server Container. I've been saying that for a couple of months but I'm putting it on the internet in an effort to make it true.

After posting giddily on Docker in the Windows world recently, Microsoft released Windows Server 2016 Technical Preview 3 with container support. I've had a chance to play with it a little so let's see where this goes...

It's a preview

Like movie previews, this is equal parts exciting and frustrating. Exciting because you get a teaser of things to come. Frustrating because you just want it to work now. And extra frustration points for various technical issues I've run into that, I hope, are due to the "technical preview" label.

For example, installing container support into an existing VM is mind-numbingly slow. Kudos to the team for making it easy to install but at the point where you run ContainerSetup.ps1, be prepared to wait for, by my watch, at least 45 minutes without any visual indication that something is happening. The only reason I knew something was happening is because I saw the size of the VM go up (slowly) on my host hard drive. This is on a 70Mbps internet connection so I don't think this can be attributed to "island problems" either.

I've heard tell of issues setting up container support in a Hyper-V VM as well. That's second-hand info as I'm using Fusion on a Mac rather than Hyper-V. If you run into problems setting it up on Hyper-V, consider switching to the instructions for setting up containers on non-Hyper-V VMs instead (no longer available).

There's also the Azure option. Microsoft was gracious enough to provide an Azure image for Windows Server 2016 pre-configured with container support. This works well if you're on Azure and I was able to run the nginx tutorial on it with no issues. I had less success with the IIS 10 tutorial even locally. I could get it running but was not able to create a new image based on the container I had.

It's also a start

Technical issues aside, I haven't been this excited about technology in Windows since...ASP.NET MVC, I guess, if my tag cloud is to be believed. And since this is a technical preview designed to garner feedback, here's what I want to see in the Windows container world

Docker client and PowerShell support

I love that I can use the Docker client to work with Windows containers. I can leverage what I've already learned with Docker in Linux. But I also love that I can spin up containers with PowerShell so I don't need to mix technologies in a continuous integration/continuous deployment environment if I already have PowerShell scripts set up for other aspects of my process.

Support for legacy .NET applications

I can't take credit for this. I've been talking with Gabriel Schenker about containers a lot lately and it was he who suggested they need to have support for .NET 4, .NET 3.5, and even .NET 2.0. It makes sense though. There are a lot of .NET apps out there and it would be a shame if they couldn't take advantage of containers.

Smooth local development

Docker Machine is great for getting up and running fast on a local Windows VM. To fully take advantage of containers, devs need to be able to work with them locally with no friction, whether that means a Windows Container version of Docker Machine or the ability to work with containers natively in Windows 10.

ARM support

At Western Devs, we have a PowerShell script that will spin up a new Azure Linux virtual machine, install docker, create a container, and run our website on it. It goes without saying (even though I'm saying it) that I'd like to do the same with Windows containers.

Lots of images out of the gate

I'd like to wean myself off VMs a little. I picture a world where I have one base VM and I use various containers for the different pieces of the app I'm working on. E.g. A SQL Server container, an IIS container, an ElasticSearch container, possibly even a Visual Studio container. I pick and choose which containers I need to build up my dev environment and use just one (or a small handful) of VMs.

In the meantime, I'm excited enough about Windows containers that I hope to incorporate a small demo with them in my talk at MeasureUP in a few scant weeks so if you're in the Austin area, come on by to see it.

It is a glorious world ahead in this space and it puts a smile on this hillbilly's face to see it unfold.

Kyle the Barely Contained

Western Devs uses Jekyll. Someone suggested it, I tried it, it worked well, decision was done. Except that it doesn't work well on Windows. It's not officially supported on the platform and while there's a good guide on getting it running, we haven't been able to do so ourselves. Some issue with a gem we're using and Nokogiri and lib2xml and some such nonsense.

So in an effort to streamline things, Amir Barylko create a Docker image. It's based on the Ruby base image (version 2.2). After grabbing the base image, it will:

• Install some packages for building Ruby
• Install the bundler gem
• Clone the source code into the /root/jekyll folder
• Run bundle install
• Expose port 4000, the default port for running Jekyll

With this in place, Windows users can run the website locally without having to install Ruby, Python, or Jekyll. The command to launch the container is:

docker run -t -p 4000:4000 -v //c/path/to/code:/root/jekyll abarylko/western-devs:v1 sh -c 'bundle install && rake serve'

This will:

• create a container based on the abarylko/western-devs:v1 image
• export port 4000 to the host VM
• map the path to the source code on your machine to /root/jekyll in the container
• run bundle install && rake serve to update gems and launch Jekyll in the container

To make this work 100%, you also need to expose port 4000 in VirtualBox so that it's visible from the VM to the host. Also, I've had trouble getting a container working with my local source located anywhere except C:\Users\mysuername. There's a permission issue somewhere in there where the container appears to successfully map the drive but can't actually see the contents of the folder. This manifests itself in an error message that says Gemfile not found.

Now, Windows users can navigate to localhost:4000 and see the site running locally. Furthermore, they can add and make changes to their posts, save them, and the changes will get reflected in the browser. Eventually, that is. I've noticed a 10-15 second delay between the time you press Save to the time when the changes actually get reflected. Haven't determined a root cause for this yet. Maybe we just need to soup up the VM.

So far, this has been working reasonably well for us. To the point, where fellow Western Dev, Dylan Smith has automated the deployment of the image to Azure via a Powershell script. That will be the subject of a separate post. Which will give me time to figure out how the thing works.

I created a VM in Fusion for the server which went pretty smoothly. When presented with a list of operating systems, I selected Windows Server 2012 and it installed fine from the ISO file. After that, I started on the documentation and at the step where you run the ContainerSetup.ps1 powershell script, I got hit with an error:

New-NetNat : No matching interface was found for prefix (null).At C:\ContainerSetup.ps1:247 char:5New-NetNat -Name ContainerNAT -InternalIPInterfaceAddressPrefix $ ...CategoryInfo : NotSpecified (MSFT_NetNat:root/StandardCimv2/MSFT_NetNat) [New-NetNat], CimExceptionFullyQualifiedErrorId : Windows System Error 1169,New-NetNat

This tripped me up for a while for a few reasons:

1. I'm not crazy familiar with PowerShell
2. Nothing came up for the error message in the Bingoogle
3. InternalIPInterfaceAddressPrefix doesn't even appear in the documentation for New-NetNat

The answer, which came to me in a rubber ducking episode as I typed on a question on the forums, was in the VM's network configuration in VMWare Fusion:


By default, VMs get created with the "Share with my Mac" setting. Changing this to Autodetect allowed the script to continue.

By that time, I had discovered that an Azure VM image already exists for Server 2016 Tech Preview 3 and even with the Containers Feature already enabled. So that's as far as I ventured with the VM.

In the meantime, I have to temper my enthusiasm for the time being because Docker is still a Linux-only concern. Yes, you can run Docker containers on Windows but only Linux-based ones. So no SQL Server and no IIS.

But you can't stop a hillbilly from dreaming of a world of containers. So with a grand assumption that you know what Docker is roughly all about, here's what this coder of the earth meditates on, Docker-wise, before going to sleep.

Microservices

Microservices are a hot topic these days. We've talked about them at Western Devs already and Donald Belcham has a good and active list of resources. Docker is an eerily natural fit for microservices so much so that one might think it was created specifically to facilitate the architecture. You can package your entire service into a container and deploy it as a single package to your production server.

I don't think you can understate the importance of a technology like Docker when it comes to microservices. Containers are so lightweight and portable, you just naturally gravitate to the pattern through normal use of containers. I can see a time in the near future where it's almost negligent not to use microservices with Docker. At least in the Windows world. This might already be the case in Linux.

Works On My Machine

Ah, the crutch of the developer and the bane of DevOps. You set it up so nicely on your machine, with all your undocumented config entries and custom permissions and the fladnoogles and the whaztrubbets and everything else required to get everything perfectly balanced. Then you get your first bug from QA: can't log in.

But what if you could test your deployment on the exact same image that you deployed to? Furthermore, what if, when a bug came in that you can't reproduce locally, you could download the exact container where it was occurring? NO MORE EXCUSES, THAT'S WHAT!

Continuous Integration Build Agents

On one project, we had a suite of UI tests which took nigh-on eight hours in TeamCity. We optimized as much as we could and got it down to just over hours. Parallelizing them would have been a lot of effort to set up the appropriate shared resources and configurations. Eventually, we set up multiple virtual machines so that the entire parallel test run could finish in about an hour and a half. But the total test time of all those runs sequentially is now almost ten hours and my working theory is that it's due to the overhead of the VMs on the host machine.

Offloading services

What I mean here is kind of like microservices applied to the various components of your application. You have an application that needs a database, a queue, a search components, and a cache. You could spin up a VM and install all those pieces. Or you could run a Postgres container, a RabbitMQ container, an ElasticSearch container, and a Redis container and leave your machine solely for the code.

When it comes right down to it, Docker containers are basically practical virtual machines. I've used VMs for many years. When I first started out, it was VMWare WorkStation on Windows. People that are smarter than me (including those that would notice that I should have said, "smarter than I") told me to use them. "One VM per client" they would say. To the point that their host was limited to checking email and Twitter clients.

I tried that and didn't like it. I didn't like waiting for the boot process on both the host and each client and I didn't like not taking full advantage of my host's hardware on the specific client I happened to be working on at that moment.

But containers are lightweight. Purposefully so. Delightfully so. As I speak, the overworked USB drive that houses my VMs is down to 20 GB of free space. I cringe at the idea of having to spin up another one. But the idea of a dozen containers I can pick and choose from, all under a GB? That's a development environment I can get behind.

Alas, this is mostly a future world I'm discussing. Docker is Linux only and I'm in the .NET space. So I have to wait until either: a) ASP.NET is ported over to Linux, or b) Docker supports Windows-based containers. And it's a big part of my excitement that BOTH of those conditions will likely be met within a year.

In the meantime, who's waiting? Earlier, I mentioned Postgres, Redis, ElasticSearch, and RabbitMQ. Those all work with Windows regardless of where they're actually running. Furthermore, Azure already has pre-built containers with all of these.

Much of this will be the basis of my talk at the upcoming MeasureUP conference next month. So...uhhh....don't read this until after that.

In the meantime, I have been talking about them quite a bit recently. We've been using them on a project at Clear Measure and I'd like to think it's been successful but it's too soon to tell. I feel good about what we've done which, historically, has always been a good metric for me.

The topic has been covered at a technical and architectural level pretty well by Martin Fowler, so much so that he's even collected his discussions into a nice little Microservices Resource Guide. In it, he and other ThoughtWorkians define them (to the extent that anything in software containing the word "services" can be defined), point out pros and cons compared to monolithic applications, describe testing strategies, and cover off the major success stories in the space.

That doesn't leave much ground for me to cover which, from a marketing standpoint, is almost surely the point. But I would like to add my voice if for no other reason than to plug the podcast on the subject.

One of the more interesting links on Fowler's Resource Guide is tucked away at the bottom. It's a series of posts on how SoundCloud is migrating from a monolith to microservices. Part 1 discusses how they stopped working on the monolith and performed all new work in new microservices and part 2 is on how they split the monolith up into microservices. There were challenges in both cases, leading to other architectural decisions like event sourcing.

The arguments for and against are, predictably, passionate and academic. "Overkill!" you say. "Clean boundaries!" sez I. "But...DevOps!" you counter. "Yes...DevOps!" I respond. But SoundCloud's experience, to me, is the real selling point of microservices. Unlike Netflix and Amazon, it's a scale that is still relatable to many of us. We can picture ourselves in the offices there making the same decisions they went through and running up against the same problems. These guys have BEEN THERE, man! Not moving to microservices because they have to but because they had a real problem and needed a solution.

Now if you read the posts, there's a certain finality to them. "We ran into this problem so we solved it by doing X." What's missing from the narrative is doubt. When they ran into problems that required access to an internal API, did anyone ask if maybe they defined the boundaries incorrectly? Once event sourcing was introduced, was there a question of whether they were going too far down a rabbit hole?

That's not really the point of these posts, which is merely to relay the decision factors to see if it's similar enough to your situation to warrant an investigation into microservices. All the same, I think this aspect is important for something still in its relative infancy, because there are plenty of people waiting to tell you "I told you so" as soon as you hit your first snag. Knowing SoundCloud ran into the same doubt can be reassuring. Maybe I'm just waiting for Microservices: The Documentary.

Regardless, there are already plenty of counter-arguments (or more accurately, counter-assumptions) to anecdotal evidence. Maybe the situation isn't the same. They have infrastructure. They have money and time to rewrite. They have confident, "talented" developers who always know how to solve architectural problems the right away.

So now I've more or less done what I always do when I talk microservices, which is talk myself into a corner. Am I for 'em or agin 'em? And more importantly, should you, reader, use them?

The answer is: absolutely, of course, and yes. On your current project? That's a little murkier. The experience is there and microservices have been done successfully. It's still a bit of a wild west which can be exciting if you ain't much for book learnin'. But "exciting" isn't always the best reason to decide on an architecture if someone else is paying the bills. As with any architectural shift, you have to factor in the human variables in your particular project.

For my limited experience, I like them. They solve one set of problems nicely and introduce a new set of problems that are not only tractable, but fun, in this hillbilly's opinion.

And why else did you get into the industry if not to have fun?

The side project is Western Devs, a fairly tight-knit community of developers of similar temperament but only vaguely similar backgrounds. It's a fun group to hang out with online and in person and at one point, someone thought "Wouldn't it be nice to build ourselves a website and have Kyle manage it while we lob increasingly ridiculous feature requests at him from afar?"

Alas, I suffer from an unfortunate condition I inherited from my grandfather on my mother's side called "Good Idea At The Time Syndrome" wherein one sees a community in need and charges in to make things right and damn the consequences on your social life because dammit, these people need help! The disease is common among condo association members and school bus drivers. Regardless, I liked the idea and we're currently trying to pull it off.

The first question: what do we build it in? WordPress was an option we came up with early so we could throw it away as fast as possible. Despite some dabbling, we're all more or less entrenched in .NET so an obvious choice was one of the numerous blog engines in that space. Personally, I'd consider Miniblog only because of its author.

Then someone suggested Jekyll hosted on GitHub pages due to its simplicity. This wasn't a word I usually assocated with hosting a blog, especially one in .NET, so I decided to give it a shot.

Cut to about a month later, and the stack consists of:

• Jekyll
• GitHub Pages
• Markdown
• SASS
• Slim
• Rake
• Travis
• Octopress
• Minimal Mistakes Jekyll theme

Of these, the one and only technology I had any experience with was Rake, which I used to automate UI tests at BookedIN. The rest, including Markdown, were foreign to me.

And Lord Tunderin' Jayzus I can not believe how quickly stuff came together. With GitHub Pages and Jekyll, infrastructure is all but non-existent. Octopress means no database, just file copying. Markdown, Slim and SASS have allowed me to scan and edit content files easier than with plain HTML and CSS. The Minimal Mistakes theme added so much built-in polish that I'm still finding new features in it today.

The most recent addition, and the one the prompted this post, was Travis. I'm a TeamCity guy and have been for years. I managed the TeamCity server for CodeBetter for many moons and on a recent project, had 6 agents running a suite of UI tests in parallel. So when I finally got fed up enough with our deploy process (one can type git pull origin source && rake site:publish only so many times), TeamCity was the first hammer^* I reached for.

One thing to note: I've been doing all my development so far on a MacBook. My TeamCity server is on Windows. I've done Rake and Ruby stuff on the CI server before without too much trouble but I still cringe inwardly whenever I have to set up builds involving technology where the readme says "Technically, it works on Windows". As it is, I have an older version of Ruby on the server that is still required for another project and on Windows, Jekyll requires Python but not the latest version, and I need to install a later version of DevKit, etc, etc, and so on and so forth.

A couple of hours later, I had a build created and running with no infrastructure errors. Except that it hung somewhere. No indication why in the build logs and at that moment, my 5-year-old said, "Dad, let's play hockey" which sounded less frustrating than having to set up a local Windows environment to debug this problem.

After a rousing game where I schooled the kid 34-0, I left him with his mother to deal with the tears and I sat down to tackle the CI build again. At this point, it occurred to me I could try something non-Windows-based. That's where Travis came in (on a suggestion from Dave Paquette who I also want to say is the one that suggested Jekyll but I might be wrong).

Fifteen minutes. That's how long it took to get my first (admittedly failing) build to run. It was frighteningly easy. I just had to hand over complete access to my GitHub repo, add a config file, and it virtually did the rest for me.

Twenty minutes later, I had my first passing build which only built the website. Less than an hour later and our dream of continuous deployment is done. No mucking with gems, no installing frameworks over RDP. I updated a grand total of four files: .travis.yml, _config.yml, Gemfile, and rakefile. And now, whenever someone checks into the source branch, I am officially out of the loop. I had to do virtually nothing on the CI server itself, including setting up the Slack notifications.

This is a long-winded contradiction of my post of four years ago where my uncertainty with Java drove me to the comfort of .NET. And to keep perspective, this isn't exactly a mission critical, LOB application. All the same, for someone with 15-odd years of .NET experience under his obi, I'd be lying if I said I wasn't amazed at how quickly one can put together a functional website for multiple authors with non-Microsoft technology you barely have passing knowledge of.

To be clear, I'm fully aware of what people say about these things. I know Ruby is a fun language and I feel good about myself whenever I do anything substantial with it. And I know Markdown is all the rage with the kids these days. It's not really one technology on its own that made me approach epiphaniness. It's the way all the tools and libraries intermingle so well. Which has this optimistic hillbilly feeling like his personal life and professional life are starting to mirror each other.

Is there a lesson in here for others? I hope so as it would justify me typing all this out and clicking publish committing to the repository. But mostly, like everything else, I'm just happy to be here. As I've always said, if you've learned anything, that's your fault, not mine.

Kyle the Coalescent

_{^{* With credit to Brendan Enrick's and Steve Smith's Software Craftsmanship Calendar 2016}}

I don't recall all the participants but it was a good number of the people on this list. Here, I'm going to attempt to summarize the salient points but given my memory, it'll more likely be a dissertation of my own thoughts. Which is just as well as I recall doing more talking than I should have.

Should you UI test?

This was a common thread throughout. Anyone who has done a significant amount of UI testing has asked a variant of this question. Usually in the form, "Why the &*%$ am I doing this?"

Let it not be said that UI testing is a "set it and forget it" affair. Computers are finicky things, UI's seemingly more so. Sometimes things can take just that one extra second to render and all of a sudden your test starts acting out a Woody Allen scene: Where's the button? There's supposed to be a button. YOU TOLD ME THERE WOULD BE A BUTTON!!!

Eventually, we more or less agreed that they are probably worth the pain. From my own experience, working on a small team with no QA department, they saved us on several occasions. Yes, there are the obvious cases where they catch a potential bug. But there was also a time when we had to re-write a large section of functionality with no change to the UI. I felt really good about having the tests then.

One counter-argument was whether you could just have a comprehensive suite of integration tests. But there's something to be said for having a test that:

1. Searches for a product
2. Adds it to the shopping cart
3. Browses more products
4. Checks out
5. Goes to PayPal and pays
6. Verifies that you got an email

This kind of integration test is hard to do, especially when you want to verify all the little UI things in between, like whether a success message showed up or whether the number of items in the shopping cart incremented by 1.

We also had the opposite debate: If you have a comprehensive suite of UI tests and are practicing BDD, do you still need TDD and unit tests? That was an interesting side discussion that warrants a separate post.

Maintenance

...is ongoing. There's no getting around that. No matter how bullet-proof you make your tests, the real world will always get in the way. Especially if you integrate with third-party services (<cough>PayPal<cough>). If you plan to introduce UI tests, know that your tests will be needy at times. They'll fail for reasons unknown for several consecutive runs, then mysteriously pass again. They'll fail only at certain times of the day, when Daylight Savings Time kicks in, or only on days when Taylor Swift is playing an outdoor venue in the western hemisphere. There will be no rhyme or reason to the failures and you will never, ever be able to reproduce them locally.

You'll add sleep calls out of frustration and check in with only a vague hope that it will work. Your pull requests will be riddled with variations of "I swear I wouldn't normally do this" and "I HAVE NO IDEA WHAT'S GOING ON". You'll replace elegant CSS selectors with XPath so grotesque that Alan Turing will rise from his grave only to have his rotting eyeballs burst into flames at the sight of it.

This doesn't really jibe with the "probably worth it" statement earlier. It depends on how often you have to revisit them and how much effort goes into it. From my experience, early on the answer to that is often and a lot. As you learn the tricks, it dwindles significantly.

One of those tricks is the PageObject pattern. There was universal agreement that it is required when dealing with UI tests. I'll admit I hadn't heard of the pattern before the discussion but at the risk of sounding condescending, it sounds more like common sense than an actual pattern. It's something that, even if you don't implement it right away, you'll move toward naturally as you work with your UI tests.

Data setup

...is hard, too. At least in the .NET world. Tools like Tarantino can help by creating scripts to prime and tear down a database. You can also create an endpoint (on a web app) that will clear and reset your database with known data.

The issue with these approaches is that the "known" data has to actually be known when you're writing your tests. If you change anything in it, Odin knows what ramifications that will have.

You can mitigate this a little depending on your technology. If you use SpecFlow, then you may have direct access to the code necessary to prime your database. Otherwise, maybe you can create a utility or API endpoints that allow you to populate your data in a more transparent manner. This is the sort of thing that a ReST endpoint can probably do pretty well.

Mobile

Consensus for UI testing on mobile devices is that it sucks more than that time after the family dinner when our cousin, Toothless Maggie, cornered---...umm... it's pretty bad...

We would love to be proven wrong but to our collective knowledge, there are no decent ways to test a mobile UI in an automated fashion. From what I gather, it's no picnic doing it in a manual fashion. Emulators are laughably bad. And there are more than a few different types and versions of mobile device so you have to use these laughably bad options about a dozen different ways.

Outsourcing

What about companies that will run through all your test scripts on multiple browsers and multiple devices? You could save some development pain that way. But I personally wouldn't feel comfortable unless the test scripts were extremely prescriptive. And if you're going to that length, you could argue that it's not a large effort to take those prescriptive steps and automate them.

That said, you might get some quick bang for your buck going this route. I've talked to a couple of them and they are always eager to help you. Some of them will even record their test sessions which I would consider a must-have if you decide to use a company for this.

Tooling

I ain't gonna lie. I like Cucumber and Capybara. I've tried SpecFlow and it's probably as good as you can get in C#, which is decent enough. But it's hard to beat fill_in 'Email', :with => 'hill@billy.edu' for conciseness and readability. That said, do not underestimate the effort it takes to introduce Ruby to a .NET shop. There is a certain discipline required to maintain your tests and if everyone is scared to dive into your rakefile, you're already mixing stripes with plaid.

We also discussed Canopy and there was a general appreciation for how it looks though Amir is the only one who has actually used it. Seems to balance the readability of Capybara with the "it's still .NET" aspect of companies that fear anything non-Microsoft.

Of course, there's Selenium both the IDE and the driver. We mentioned it mostly because you're supposed to.

Some version of Visual Studio also provided support for UI tests, both recorded and coded. The CodedUI tests are supposed to have a pretty nice fluent interface and we generally agreed that coded tests are the way to go instead of recorded ones (as if that were ever in doubt).

Ed. note: Shout out to Protractor as well. We didn’t discuss it directly but as Dave Paquette pointed out later, it helps avoid random Sleep calls in your tests because it knows how to wait until binding is done. Downside is that it’s specific to Angular.

Also: jasmine and PhantomJS got passing mentions, both favorable.

Continuous Integration

This is about as close as we got to disagreement. There was a claim that UI tests shouldn't be included in CI due to the length of time it takes to run them. Or if they are included, run them on a schedule (i.e. once a night) rather than on every "check in" (by which we mean, every feature).

To me, this is a question of money. If you have a single server and a single build agent, this is probably a valid argument. But if you want to get full value from your UI tests, get a second agent (preferably more) and run only the UI tests on it. If it's not interfering with your main build, it can run as often as you like. Yes, you may not get the feedback right away but you get it sooner than if you run the UI tests on a schedule.

The main takeaway we drew from the discussion, which you may have gleaned from this summary, is: damn, we should have recorded this. That's a mistake we hope to rectify for future discussions.