This is a series of articles chronicling my learning journey as I was asked to build an IdentityServer4-based authentication system for one of my clients. This story included details about my adoption of Kubernetes, Azure Kubernetes Service, and all the things that I had to do to stand-up this client's new IdentityServer4 authentication implementation.

As with most learning experiences, I've made mistakes, arrived at working applications, adjusted my implementations, and continued to grow. My hope is that this series will continue to grow and evolve and be a bit of a living series of documents. I'll actively change articles when I discover something new or a better way to describe things. I'll add new articles or maybe alternate paths through the series as new topics present themselves. And I'm certainly not an authority on all of these topics, so as I get feedback from friends and peers, I'll certainly be making adjustments.

I have two primary goals with this series. Documenting the learning journey is the first one, but the second and almost as important goal was to provide someone (you) with a complete set of steps to get a Kubernetes cluster up and running with an IdentityServer4 implementation running inside of it. Once someone has this platform up and running, they can continue to learn and grow in the same manner that I will but hopefully they're path getting to this point was a little smoother than mine was.

As always, comments and feedback are encouraged and very welcome.

Now, on to the articles!

Series Table of Contents

1. Business problems
2. Initial Assumptions, Technologies, Learning resources
3. Tools in Use
4. Building an ASP.NET Core IdentityServer Implementation
5. Getting Started with Kubernetes - Minikube
   • Part A - Getting Started with Kubernetes - Minikube
   • Part B - Getting Started with Kubernetes - Minikube
6. Pause to reflect
7. Moving to Azure Kubernetes Service
   • Part A - Moving to Azure Kubernetes Service
   • Part B - Moving to Azure Kubernetes Service
8. Making Your Kubernetes Cluster Publicly Accessible
9. Adding Cluster Logging (fluentd)
10. Tuning resource usage

Approach to this series

I've decided to break this series into discrete posts to make this easier to write and consume. I'll have specific topics for a post that are aligned with the overall vision, and you'll be able to read the parts that are important to you.

This series is going to strive to demonstrate work that is completely done. Every bit of code in each post should work and be complete. I'll point out bits of knowledge and wisdom I've learned along the way, but the intention is to give you a working system that you can then alter/re-create and learn from that experience.

All the code, projects, manifests, etc. are (or will be) in Github here.

Enjoy

Next up:Business problems

The Business Problem

My client has been in business since 1993. As you can imagine, software has been an integral part of their ability to deliver services to their customers. About 10 years ago, they had a re-platforming initiative that was the start of their current monolithic, critical LOB system. It is an ASP.NET MVC 5.x application that has evolved over time and is certainly reaching the limits of what its architecture can provide. It isn't "cloud-native" and it basically continues to be enhanced/evolved based on assumptions that were made 10 years ago.

For this project, the important part to understand is that it used forms authentication and has a user store internally that manages all of the user accounts and authorization rules. It currently only uses cookies to store authenticated user details on the client and doesn't support any modern sort of token-based activities that are needed for modern applications.

It is also important to understand that the platform is growing as users and customers demand more modern and specific user experiences. New web applications (React), mobile devices (native), IoT devices, and system-to-system integrations are all being added to the platform and all these new applications need to participate in the authentication scheme. In some cases, new applications have been created but because the current LOB system doesn't support modern authentication techniques, they cloned the existing user store and used the data to implement the appropriate authentication techniques for their specific use case. This has led to a bunch of applications re-implementing their own authentication, with everyone sharing the user store (or a copy of it) of the core LOB system.

So, the current system has demonstrated a few key problems that the new system needs to address.

Security is hard and incredibly important

One of the plagues of modern business is bad actors attacking various systems trying to gain access and cause lots of troubles. This is an ever-present problem for most corporations. Security needs to be a first-class citizen in all projects. Using modern authentication platforms that are proven and community-reviewed is seen as a requirement.

Speaking of security, in this series, you will find a lot of usernames, passwords, and internal details about the systems that is being built. I would ask that you please take a great deal of care when building your systems! I'm not worried about presenting these details as I'm tearing all of this down all the time and it won't be left running on my Azure accounts. I will change passwords, usernames, and all kinds of other details to make it harder for anyone to break into a running system. I strongly encourage you to do the same.

Decentralized identity management sucks

The platform is currently a monolithic LOB system that provides all functional aspects for the various business groups. The platform is also growing a collection of loosely related applications that basically represent specific implementations that are tailored for each individual area of the business. All the systems are re-implementing authentication while using, in most cases, a copy of the LOB user store. This basically means that we have multiple user stores that all must be managed independently. There are SSIS workflows trying to keep the right data in sync while not clobbering specific customizations to the user store data.

This is all complicated, complex, and error prone. We generally don't have a lot of troubles with authentication, but it is expensive and time-consuming to setup and maintain and as new applications are added, it adds more things to manage and has more things that can go wrong.

New applications require modern Identity technologies

In 2020, business that used to be well-served by a monolithic web application, are finding that this is no longer the case. Native applications, new web applications based on new technologies (SPA), IoT devices and System-to-System integrations all require more modern authentication technologies and these applications are being added to platform ecosystems at an increasing pace.

Reduce the costs of running applications

One of the overall initiatives for this client is to become more cloud-native and reduce the running costs of applications. There is already an initiative to move all the platform servers to Azure, but we didn't want any new applications continuing the pattern of "lift and shift" used for the existing apps. There is a strong desire to leverage containers (docker) and orchestrators (Kubernetes) at an increasing rate for all the applications in order to reduce overall run costs.

Being able to scale up is important

As with all businesses, we expect growth. My client is heavily reliant on devices for their current business needs and will be adding customers, native apps, and IoT devices at an ever-increasing rate. This means that having the ability to scale vertically or horizontally needs to be present from the start. Growth was anticipated (pre-COVID19) to be > 30% year over year for the next 3-5 years so there were going to be lots of people and devices needing to use the system in the future.

Summary (TL;DR;)

Security is hard! De-centralized (read: many) authentication systems are expensive. Old Application are ... well... old! Modern applications need support! Moving to the cloud is happening! And businesses grow!!

I hope that this gives some context that helps you understand some of the decisions that I'll be sharing in the next section!

Next up:Initial Assumptions, Technologies, Learning resources

Previously:Business problems

Initial Assumptions, Technologies, Learning Resources

There are some assumptions that were made before my involvement with this company that made some of this decision making easy. Those assumptions were driven by the business problems we discussed previously.

Some of these decisions were made by me throughout this project. It was important for me to understand the organization's business, goals, and culture when making these decisions so they weren't as easy to make. The good thing is that I've got a great group of leaders and developers in this organization that helped me and trusted me to make better decisions on their behalf.

Pre-defined Assumptions

So, there are a bunch of things that we're decided before me arriving on the scene! These decisions governed my initial decision making to just get the project started.

We're going to the cloud

I'm not sure this needs a lot of elaboration. Everyone is going to the cloud. Owning and operating data centres is hard. Companies like Microsoft, Amazon, and Google have commoditized hardware setup, management, and maintenance so much so that it almost doesn't makes sense to own your own data centre. So, anything I did should strive to be cloud-native.

Azure

Azure was chosen as the vendor for all our cloud services prior to my involvement. This certainly plays to my strength as a Microsoft-based technologist and an avid Azure user for all my own projects.

ASP.NET Core (C#)

This company is a Microsoft shop. All the developers are C# developers who are familiar with Microsoft technologies, so it only makes sense that we are going to leverage the existing people and skills that are present in the organization. Using .NET Core and ASP.NET Core 3+ in all projects going forward was made a requirement (and is probably simple common sense) by the assumption of moving to the cloud.

I couldn't build something that only I could maintain or would require the company to go and try to find new developers or acquire new skills in their current developers. There is going to be a lot of learning being done by everyone here, but we don't need to add to the pile of new stuff to learn.

HTTPS

One of the things that seems obvious, but turns out isn't always obvious, is the fact that in today's world, we should be using HTTPS everywhere. It has become easier and easier to do this, and part of this project's mandate was to ensure that we accelerated the adoption of HTTPS throughout the deployed platform and make it easier to own and maintain. Simply doing an Identity implementation will drive this.

Decisions I made

During the planning of this project, I had the flexibility to make a lot of decisions. I didn't make them in isolation, but I did get to make the final decisions.

Containers and Orchestration

Going to the cloud can be as simple as lift-and-shift. Pick up your server, virtualize it if necessary, and place it in the cloud. In some cases, we can make this even simpler by provisioning a VM with all the pre-installed parts we need and simply deploying our application into that new VM. This is certainly "going to the cloud" but it isn't considered cloud-native.

Docker

I'm not an expert on what cloud-native entirely means, but I've discerned that one thing you need to do is containers. So today, that means Docker. Being a Microsoft shop, going to Azure, using Visual Studio and the az cli as tools, Docker was a no-brainer for a container engine with all the great tool support in Visual Studio, Azure DevOps Server, and the other tools, so I just rolled with it.

Kubernetes

When using Azure and running containers in the cloud, you have several options. Two easy ones are Azure Container Instances (server-less container hosting) and Azure Kubernetes Services (aka AKS). Since our environment was going to be far more complicated than running a bunch of single containers in the cloud, we needed an orchestration platform. There are two options for that today: Kubernetes and DockerCompose and we chose Kubernetes. The community support for Kubernetes is incredible, the momentum it has in the marketplace can't be denied, and it is natively supported by Azure via the AKS product, so it was another easy decision.

Cloud-Hosted

I sort of covered this one off, but there is an option to stand up your own Kubernetes cluster manually in Azure simply using Virtual Machines. AKS effectively took this option off the table. In my experience so far, AKS is a polished product, easy to use, and the control plane components are included for free. You simply pay for all the other Azure products that will make up your cluster. These include VMs, Storage, Networking, and any other products you leverage in your implementation.

Azure Kubernetes Service (AKS)

AKS also makes some of our ability to scale requirements much easier to address. You can scale vertically by dynamically increasing the size of the VMs in the cluster and we can scale horizontally by dynamically adding (or removing) nodes from the cluster.

Identity Implementation

While this project has many objectives, they key objective for all this work was to deliver a new Identity management implementation to the platform that would enable a secure, modern way for all users, applications, and devices to authenticate with each other.

One of the underlying cultural desires at this client that I should state is a strong desire to minimize subscriptions and dependencies on 3rd party vendors. This desire precluded us from selecting any of the current Identity providers that exist on the marketplace today. AuthO and Okta are two such companies and while I'm sure they have great products, wedecided to build our own Identity system.

IdentityServer4

Now, we didn't want to build all the identity system from scratch! That doesn't make sense. We'd have to become experts in OAuth, OpenID Connect, flows, encryption, tokens, and all those technologies that are implemented by great communities out there already. This OSS frameworks are also scrutinized and vetted by the community, so you know that a lot of people care that they are done right. Basically, we wanted to take the best implementation of a security framework in the Microsoft open-source community, host it in our infrastructure, and be able to customize it to our specific use-cases. This led us to IdentityServer4, which is the core technical component of our Identity implementation.

Skoruba IdentityServer4 Admin

One of the things about identity implementations is that there is a lot of configuration and user data to manage. To be honest, client configuration and user data is one of the biggest PITA about doing Identity work. Once you get it, it isn't so bad but so much of a successful roll-out with an identity platform hinges on getting all this data right. And to do that, it helps to have a good administrative platform.

While I'd love to have had the time to write an application that gives a good user experience for the administration of this data, I didn't have that time. So, I went looking for something in the community that used IdentityServer4 as a foundational component and added the user experience I wanted. And after several pilots, I found and selected Skoruba IdentityServer4 Admin. This is an ASP.NET Core 3.x web application that provides the Security Token Service (STS), and Administrative Application, and an Administrative API, all in one easy to use package! I've been really happy that I found it.

Data Persistence

The Skoruba templates allows for the selection of one from three different built-in persistence providers via Entity Framework Core. SqlServer, MySql and Postgres.

Postgres

Postgres was the database provider the I selected. This reduced our licensing costs, provides the appropriate level of performance and functionality, and with us being in Azure, we always have the option of moving to Azure Database for PostgreSQL which has single instance SKU and an HA/Hyperscale SKU if that eventually becomes a needed capability. There is also a ton of community support around the product, which means libraries, documentation, and examples are plentiful.

Logging

I cannot under-state how important logging is when developing, maintaining, and owning a multi-container environment, spread across multiple servers (nodes) that are hosted in the cloud. This isn't the first and hopefully won't be the last blog post to emphasize this point. The community building cloud-based applications already blogs about this, you've almost certainly read about it somewhere before here, but it still may not have sunk in that, perhaps, this needs to be the first thing you figure out in your new container-based platform. I paid some attention at the beginning, but not enough.

Seq - Log Ingestion and Analytics

When I first arrived at my client, they were still relying on file-based logging, spread over all the servers in the farm. They had a lot of logging in the application, but it wasn't easily accessible, and it wasn't easy at all to find out what happened across a workflow, sometimes broken, that traversed a bunch of servers.

The other thing that wasn't possible was querying and basic analytics of the log data that was being generated. It was inconsistent, unstructured, and just barely helping them solve problems.

At a previous engagement, I had used Splunk and really came to love it! The tool was great. Powerful queries, visualization, dashboard, alerts, integrations! Splunk has it all. There was only one problem. Splunk is expensive. That isn't a problem if your needs justify the spend and you have the required expertise in owning and operating Splunk, but that wasn't the case here, so I needed to find something easier to bring into the ecosystem. This was when I found Seq!

Seq is a great entry-level tool (that keeps on getting better) for organizations that are just getting started on getting logs into a central product and using that log data to analyze problems and operational aspects of the platform. Seq is built by Datalust.co, who also makes Serilog the .NET library that we use for logging internally in the application. This combination has turned out to be a cost-effective, easy to learn, setup and maintain logging infrastructure for our apps, so it was quite easy for me to bring this into the Identity project and the Kubernetes (k8s) infrastructure.

Fluentd

In addition to the identity applications that are in the k8s cluster that will be logging to Seq, the infrastructure applications in the cluster itself will generate an incredible amount of information about its operations and problems. This is a non-trivial log ingestion problem that has been made very approachable with a product called fluentd. Built to live inside of k8s clusters, fluentd is basically an event processing pipeline implementation that takes events from log files that are written all over the k8s cluster. It takes care of finding the files, processing them as they get additional entries, and sending those events "somewhere". In this case, I used an fluentd docker image that emits logs to graylog consumers, which thankfully, Seq is with an addon. So, with Serilog in the applications, fluentd in the cluster using a graylog variant and Seq running in the cluster to ingest and aggregate all the log information, we have a very compact, useful logging strategy to get us started.

I don't know how far this logging implementation will grow with the cluster into the future, but it is designed to have pieces replaced as we outgrow their capabilities.

Learning Resources

This adventure required an incredible amount of learning on my part. The way that I learn best is by reading/learning enough to get started, and then building like crazy, discovering deficiencies in what I know, resolving the deficiency, and then finding the next thing I don't know.

This series of blog posts will hopefully capture some of that learning experience, but I wanted to make sure I shared the places that I started.

Pluralsight

There are a lot of courses on Pluralsight. I think I've enjoyed all them, with some courses being better than others for me at that point in my learning. For this project, I didn't need any help with C# or ASP.NET Core or any of that part. I needed to get familiar with Kubernetes and I needed to become familiar with authentication flows using OAuth2 and OpenID Connect.

The k8s courses were from Nigel Poulton were great places to get started with k8s. The ones that I watched are:

• Getting Started with Kubernetes
• Docker and Kubernetes: The Big Picture

When getting more details on OAuth2 and OpenId Connect, Kevin Dockx and Micah Silverman had two nice courses to review.

• Securing ASP.NET Core 2 with OAuth2 and OpenID Connect
• THAT Conference '19: OAuth 2.0 and OpenID Connect (In Plain English)

There is a new version of Kevin Dockx course available using ASP.NET Core 3 here

Kubernetes.io

Based on what and how I was learning k8s from Nigel on Pluralsight, Kubernetes.io became an invaluable resource for me to learn about k8s and its ecosystem. Nigel's courses encourage you to work with manifests, building your k8s cluster in a declarative manner, and kubernetes.io supported that approach well. I would highly recommend working with manifests and understanding them and how they work when learning k8s. We'll discover later that I don't work directly with manifests when provisioning my cluster resources, but you have to know manifests because all the examples in the communities are in manifests!

Community blogs

This would have been much harder without the incredibly vibrant community of bloggers in the world who are sharing what they are doing, the problems, and how they are solving those problems. I'm hoping that my addition in writing this blog fills a gap and makes it a bit easier for you or someone else to put this all together, but this would have been incredibly difficult without this vibrant community.

Next up:Tools in Use

Links

• Azure
• Azure Kubernetes Services
• Kubernetes
• IdentityServer4
• Skoruba IdentityServe4 Administration
• Pulumi
• Seq Log Ingestion
• Fluentd
• Postgres
• pgAdmin4
• CertManager
• Nginx
• Traefix

Previously:Initial Assumptions, Technologies, Learning resources

Tools in Use

Tooling is an incredibly important aspect of a developer's daily life. IDEs, CLIs, Automations, Visualizations; the list goes on and on. Sifting through the set of tools that are available and finding out if they work for you can take a lot of time. As a former Microsoft MVP (Dev Tools), I'm always interested in tooling because I know the importance it can make in developer productivity and the overall productivity and quality for an organization! I'm hoping that by sharing my base set of tools, and what I use them for, you'll get a bit of tool-curation time savings back to spend on learning other things.

These are the tools I use every day on this project.

Visual Studio

Visual Studio has come a long, long way since the first time I ever used is as Microsoft Visual Studio 97 (5.x). Wow! This IDE has been my constant companion for the entirety of my development career, and I have to say that the current VS 2019 version of the tool is a pleasure to work with. Given the complex nature of building projects and supporting all of the ecosystems that we build projects for, VS 2019 does a fantastic job of supporting my needs.

With this project, I obviously appreciate all of the normal C# coding functions that are present, but the docker/docker-compose support is particularly important and the ability to debug applications running in docker containers is great.

I still have ReSharper and I still have many extensions (Thanks Mads!) running in Visual Studio, but this IDE is the workhorse of my day-to-day activities when I'm working in C#.

I don't recommend that you download and try to use Microsoft Visual Studio 97!

VS Code

Visual Studio Code is a fantastic, light-weight text editor with an incredible extensibility feature that the community has taken full advantage of! Out of the box, it is very good at one thing. Editing text files. With all the extensions being written and placed on the marketplace, it has become some people's full-time IDE. The great thing about VS Code is that it runs on macOS, Linux, and Windows, so you take it wherever you go! And, it's free! It has mostly replaced all my other text editors, with the exception of Notepad. Still use that one from time to time.

I use VS Code for editing all my manifests, Pulumi Typescript applications, and markdown files for documentation. This blog post was written in Markdown using VS Code! With an integrated terminal window using PS Core, I can do pretty much all my k8s deployment and resource work without leaving VS Code.

Pulumi

I've mentioned that while I find manifests a great way to learn k8s, I've adopted a different approach for deploying k8s. That approach is from a company called Pulumi! What they've basically done is built a platform and multiple SDKs in various languages that allow us to do Infrastructure as Code in a programming language of our choice!! It's a great way to define and manage your cloud resources. Once you have written your application that knows what you want to do, you simply tell the pulumi cli to up or destroy your infrastructure! pulumi up and it will create or update your infrastructure resources as needed, and pulumi destroy tears it all down and cleans everything up!

Git

Git is the most popular and arguably the defacto source control system in the world today. All my manifests and pulumi typescript files are version controlled in git, as are all the Identity application projects that we'll be building later on.

Azure DevOps Services (Server 2019)

Azure DevOps (Server) is our DevOps management software. Source control (Git and TFVC), Work Item Management, Builds, and Deployments are all provided by this service. It provides us a tremendous amount of automation, and it gives us a place to make our knowledge about how to build and deploy our applications concrete! If you haven't tried Azure DevOps recently, you really should give it another go. It's been a tremendously valuable addition to the development process.

Kubernetes Web UI (Dashboard)

Kubernetes Web UI (Dashboard) is a great dashboard that is built into your k8s cluster that provides a User experience to help you visualize and manage your k8s cluster.

Octant

Octant is a dashboard for your k8s cluster, similar to the Kubernetes Dashboard, but it doesn't require you to port-forward to your local machine to get at the dashboard, it simply runs on your local development machine and uses the kubectl current context to access the cluster! This makes is quite easy to spin up. It provides port-forwarding capabilities in addition to many of the capabilities present in the Kubernetes Web UI. I find myself using them both quite often throughout the day.

K9s

K9s is a console-based user experience for managing your k8s cluster! As with Octant, it uses kubectl and your current context to determine which k8s cluster it is managing. I quite like it! I also find it really interesting how I have three different tools for managing the k8s cluster and depending on what I'm doing or even my mood, I can pick from any of the three tools.

Honorable Mention - Docker Desktop for Windows

Docker Desktop for Windows is the easiest way to get docker containers running on a windows desktop. You need a CPU that supports virtualization and you have to have Windows 10 Pro for the HyperV/WSL2 support. There are other options if you aren't running Windows 10 Pro and up, but I'll leave that as an exercise for you to figure out. But during initial development of the Skoruba project, it was immensely helpful to be able to build, deploy, and debug the application without having to deal with k8s or minikube. There are additional complexities involved with that that you will want to defer for a while.

Next up:Building an ASP.NET Core IdentityServer Implementation

Links

• Visual Studio
• VS Code
• Pulumi
• Git
• Github
  • Git Simulator - Github
• Azure DevOps (Server)
• Kubernetes Dashboard
• Octant - Github
• K9s - Github
• Useful Interactive Terminal And Graphical UI Tools For Kubernetes - Virtually Ghetto

Previously:Tools in Use

Building an ASP.NET Core IdentityServer Implementation

This is where the articles start to get a bit longer. I've broken a couple up into a-b parts. We won't be talking about tasks in k8s for a bit, so if you want to skip on to that, you can go here.

There are, in my opinion, 2 parts that need to be considered when building up this IdentityServer4 implementation, and the 2nd part can arguably be split into 2 smaller parts, which is how this implementation is built.

The first part of the system needs to be the Security Token Services (STS) implementation. This part of the system is simply responsible for managing tokens. This means validating log in credentials, creating tokens, refreshing tokens, and sending along the appropriate metadata and claims. I would want this broken into a separate service because it will probably have different operational characteristics then the other parts of the application, and so we can scale is separately. If there are 1000s of request per minute to generate/validate/refresh tokens and send along metadata, that is different than administrating the data that the system relies on. Additionally, it makes it easier to secure the administrative access if it is a separate application.

The second part of the system is the Administrative (Admin) implementation. The application that provides a user experience that will help identity administrators do the right things.

In our implementation, which is based on the Skoruba project, there is actually a third part to the administrative side of the system and that is a Admin WebAPI that does all of the actual CRUD on the database. I'm happy with this setup and I'm glad the Skoruba project went in this direction.

Let's get into the details of what I did.

Start Learning with the IdentityServer4 Quickstart

If you are new to IdentityServer4 or even newer to OAuth2 and OpenID Connect, your first stop needs to be the IdentityServer4 Quickstarts. These quickstart modules will walk you through all of the steps required to incrementally build an IdentityServer4 server. They are all self-contained, building your knowledge from one to the next, and by the end of the quickstarts, you've built a functional IdentityServer4 implementation. I'm not going to re-produce any of these quickstarts or even how to get you started on them, other than providing the link and some encouragement. Go do the quickstarts, then come back here and continue reading!

It was during the quickstarts that I side-stepped and did the Pluralsight courses on OAuth2 and OpenID Connect.

Starting your project with Skoruba

So, you've done the quickstarts, or you're already experienced with IdentityServer4 and the real reason you are here is because the quickstarts left you wanting more! You wanted an Administrative application!! You wanted databases to persist all of this configuration and the quickstarts don't give you any of that! You're in the right place! I was in that position just a little while ago myself.

After doing the quickstarts, I realized I had to build/find an administrative experience for our platform. It isn't hard to find a couple when you google IdentityServer4 Administration. There are a number of Github repos that you'll find, and perhaps a couple products that you can purchase. Again, because of the desire to build/own vs. buy/subscribe, we continued looking until we found the Skoruba IdentityServer4 Admin project on Github. This is a great project and I've been very happy that I found it. It is put together well and is super easy to get started with.

In order to use the project, you'll need to have the latest ASP.NET Core 3.x SDK and development tools.

The first thing I did was get the awesome template that the Skoruba team build to get you started using their project. This command tells the dotnet tooling to go off to nuget.org and get the template.

dotnet new -i Skoruba.IdentityServer4.Admin.Templates::1.0.0-rc1-update2

Once you have the template, you are ready to start creating your solution. You can start by using the template.

1
2
3
4

dotnet new skoruba.is4admin --name MyProject --title MyProject `
--adminemail "admin@codingwithdave.xyz" --adminpassword "P@ssw0rd!" `
--adminrole IdentityAdminRole --adminclientid MyClientId `
--adminclientsecret MyClientSecret --dockersupport true

A couple notes about that command that make sense once you have done the ID4 Quickstarts:

• -adminrole IdentityAdminRole is the User Role that will be used in the Admin application and the AdminAPI to know if a user is an administrator of the Identity system. This role will be assigned to your admin user account.
• -adminclientid MyClientId is the ClientId (username of your application) for the Admin application in the STS. Your Admin application needs this id to be allowed to access the STS.
• -adminclientsecret MyClientSecret is the ClientSecret (password of your application) for the Admin application in the STS. It is used with ClientId.

Since we are planning to run this all in Docker locally and eventually deploying this to a k8s cluster, Docker support is required. All of the yaml examples in this post are in the docker-compose.yml file.

Once that command has run, you should have a project that looks something like this:

Select your DB Platform

One of the cool things about the Skoruba template is that it comes out of the box with 3 different database persistence mechanism already waiting for you. You can just select the one you want to use, and off it goes. I don't need to provide the ability to switch mechanisms, so I'm simply going to remove the templated MySql and SqlServer options.

If you are planning to use SqlServer in your k8s cluster, you can skip these instructions and the next Postgres/pgAdmin4 instructions. You could delete everything except for SqlServer implementation details. I understand why the template spits it out, but your running application probably won't need persistence choices.

So you selected PostgreSQL

PostgreSQL (Postgres) is an open-source database that is more than sufficient for our use-cases. It is free to use/run and has great support in the community. In order to minimize costs, I choose to use PostgresSQL.

The first thing I did was delete the MySql and SqlServer projects. I wasn't planning to use them and as such, just deleting them is the right course. They can always be put back in if the need ever arose to migrate to a different database platform. This is going to immediately break your solution. You can just go and delete/fix all of the broken code. You're just removing using statements and adjusting/removing if {} blocks. I also deleted all of the database types/selection code and configuration settings.

That should leave you with a solution that builds and that only contains these projects.

You probably also want to delete the db entry in the docker-compose.yml file. The SqlServer image is a large image and you don't necessarily need to download it.

1
2
3
4
5
6
7
8
9
10
11

# remove this entry
  db:
    image: "mcr.microsoft.com/mssql/server"
    ports:
      - 1433:1433
    container_name: skoruba-identityserver4-DB
    environment:
      SA_PASSWORD: "${DB_PASSWORD:-Password_123}"
      ACCEPT_EULA: "Y"
    volumes:
      - dbdata:/var/opt/mssql

In case you didn't know, Docker Desktop is the local image registry for all things Docker on your Windows workstation. All images will be downloaded and cached here. Docker Desktop is not the local image registry for k8s (minikube).

Backend Containers - Postgres, pgAdmin4 & Seq

Now, your solution is building and ready to run, but our local infrastructure isn't quite there yet. We're going to need a Postgres database instance on our local machine to run this in Visual Studio, so we can do that in the docker-compose.yml file.

To bring a Postgres database container into Docker Desktop, add the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

postgresdb:
  image: postgres:alpine
  hostname: postgres
  ports:
    - 5432:5432
  container_name: postgresdb
  environment:
    - "POSTGRES_USER=admin"
    - "POSTGRES_PASSWORD=P@ssw0rd!"
    - "POSTGRES_DB=identity" # this is the DB name that will be generated by EFCore
  volumes:
    - postgresdata:/var/lib/postgresql/data
  networks:
    default:
      aliases:
        - postgres

What this does is get the latest version of the Postgres container image from DockerHub. It exposes the default Postgres port of 5432, gives this container an alias in the DNS of postgres, and hooks the container up to a persistent volume, mapped to a volume on the local host, to store its identity data. We need to add that mapping near the bottom of the existing docker-compose.yml file. We will add a mapping for pgAdmin4 while we are at it.

1
2
3
4
5

volumes:
  postgresdata:
    driver: local
  pgdata:
    driver: local

There is a lot of YAML in k8s and docker. You are going to have to become familiar with it. I'm going to assume that you'll work through any yaml syntax errors that may come out of working through the articles.

After adding in Postgre, we can now add in our pgAdmin4 container instance. pgAdmin4 is a database management tool built as a web application. If you have another Postgre management tool, you don't need to follow these steps.

1
2
3
4
5
6
7
8
9
10
11

pgAdmin4:
  image: dpage/pgadmin4:4.20
  ports:
    - 5050:80
  container_name: pgAdmin4
  environment:
    - "PGADMIN_DEFAULT_EMAIL=admin@codingwithdave.xyz"
    - "PGADMIN_DEFAULT_PASSWORD=P@ssw0rd!"
    - "PGDATA=/mnt/data/pgdata"
  volumes:
    - pgdata:/mnt/data/pgdata

The pgAdmin4 container doesn't need an alias. We gave the Postgre instance a DNS alias so that all of the running containers in the cluster can simply reference the database (in the connection string) by the DNS entry of postgres. The IdentityServer4 apps and pgAdmin4 don't really need this kind of mapping so we'll leave them as accessible at the http://127.0.0.1.xip.io:port addresses.

If you want to explore more options in the pgAdmin4 configuration that are available to you, you can go to the pgAdmin4 Container Configuration page for more details.

We're also going to add a Seq instance into our docker-compose.yml file.

1
2
3
4
5
6
7

seq:
  container_name: seq
  image: datalust/seq:preview
  ports:
    - "5341:80"
  environment:
    - "ACCEPT_EULA=Y"

When I added the Seq container, I did not give it a persistent volume on the host to store log data. In the case of Seq, I'm using it for transient purposes in docker so I don't really care if the history of log entries goes away when the container is re-started. If you do want to keep your log files, you can give it a persistent volume on the host, in the same way as the databases.

Persistent Volumes do not survive a Docker Desktop data purge.

We are going to add one more little container that I want to introduce now before it really appeared in my story, but I wish that I had found it earlier. It is a small little container from Google that helps me do DNS diagnostics and debugging in the containers/k8s network. You can quickly and easily add this container at just about any time, do your diagnostics, and then remove it from the cluster. It automatically stops after 1 hour as well.

1
2
3
4

dnsutil:
  container_name: dnsutils
  image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
  command: "sleep 3600"

Configuration Changes

We aren't quite ready to run things yet. We need to make some more configuration adjustments.

You should be prepared to make a lot of configuration adjustments in your IdentityServer4 and k8s career!

Here are the things I've done to make this easier/cleaner for running locally.

1. Change all usernames to admin
2. Change all admin passwords to P@ssw0rd!
3. Change all Role entries to IdentityAdminRole
4. Delete all of the unnecessary launchSettings profiles
5. Tweak the URL configuration values
6. Change all of the database connection strings to point at our local Postgres instance
7. Tweak the code to always run migrations and seeding data on startup
8. Configure Serilog to use Seq and the console

Change Credentials and Role

You will have lots of chances once you've explored and learned about all of these configurations to change usernames, potentially move to Azure Managed Identities, or use something like Azure KeyVault to store your credentials. For the time being, we want this to be quite easy to do, so we're going to simplify this and make all the admin accounts the same.

• Postgres - If you revisit the Postgre yaml snippet, you see the default Postgre power user is admin and P@ssword!.
• pgAdmin4 - You'll see that the pgAdmin4 wants an email address, so in the yaml snippet we see the username admin@codingwithdave.xyz and P@ssw0rd!
• Seq - currently requires no authentication

There are three .json files in a file in the root of the solution called shared. These files hold seed data for the applications. They are joined to the docker containers and are used by the running instances of the applications when they are in docker. You need to edit the copies of these files.

I added a solution folder to the solution and added these files to it.

I changed the following:

• identitydata.json
  • change the initial username credentials to admin and P@ssw0rd!
  • change the Roles entry to IdentityAdminRole.

In the three projects, we need to make a few tweaks in the appsettings.json files. I did this for a little added clarity.

• appsettings.json in MyProject.Admin
• appsettings.json in MyProject.Admin.Api
• appsettings.json in MyProject.STS.Identity
  • change the AdminApiConfiguration:AdministrationRole to IdentityAdminRole

Here is an example from the STS appsettings.json file.

1
2
3
4
5
6
7

"AdminConfiguration": {
    "PageTitle": "Skoruba IdentityServer4",
    "HomePageLogoUri": "/images/skoruba-icon.png",
    "FaviconUri": "/favicon.ico",
    "IdentityAdminBaseUrl": "http://127.0.0.1.xip.io:9000",
    "AdministrationRole": "IdentityAdminRole"
},

Delete Unnecessary LaunchSettings Profiles

This one is fairly simple and just a cleaning exercise. I can debug with Docker Desktop and the DockerCompose project, so good riddance to all the other clutter! I don't plan to run this in IIS or single Docker containers, so I can remove those profiles and settings. I will leave the Kestrel settings since I can envision someone doing that. Otherwise, I'm only planning to run ecosystem this via DockerCompose or sending it to a k8s cluster.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

{
  "iisSettings": {
    "windowsAuthentication": false,
    "anonymousAuthentication": true,
    "iisExpress": {
      "applicationUrl": "http://127.0.0.1.xip.io:5000",
      "sslPort": 0
    }
  },
  "profiles": {
    "IIS Express": {
      "commandName": "IISExpress",
      "launchBrowser": true,
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
    },
    "MyProject.AspNetIdentity": {
      "commandName": "Project",
      "launchBrowser": true,
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      },
      "applicationUrl": "http://127.0.0.1.xip.io:5000"
    },
    "Docker": {
      "commandName": "Docker",
      "launchBrowser": true,
      "launchUrl": "{Scheme}://{ServiceHost}:{ServicePort}",
      "environmentVariables": {},
      "httpPort": 10000
    }
  }
}

becomes

1
2
3
4
5
6
7
8
9
10
11
12

{
  "profiles": {
    "MyProject.AspNetIdentity": {
      "commandName": "Project",
      "launchBrowser": true,
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      },
      "applicationUrl": "http://127.0.0.1.xip.io:5000"
    }
  }
}

Repeat for all of the launchSettings.json files.

The URL configuration values

Identity implementations require a lot of information about people and applications in order to work. One of those pieces of information is the URL of the caller. The Skoruba template spits out all of the URLs in the form of 127.0.0.1.xip.io. These URLs are using the xip.io service provided by the makers of Basecamp. This DNS server basically takes the DNS entry request, pulls the IP address out of it, and send that IP address base as the resolved IP. You can basically get a public DNS to resolve to an IP address that is your local machine.

One thing I found while looking over the project was a blend of localhost configurations and 127.0.0.1.xip.io. Because of the way that xip.io works, we can get rid of all of the localhost references. So, you can do a Replace in Files and replace all of the localhost with 127.0.0.1.xip.io.

We could also replace all of the 127.0.0.1.xip.io with lvh.me which is a public DNS entry that resolves to 127.0.0.1 as well. This link describes lvh.me and some other good utilities.

These URLs are in the launchSettings.json file and the appsettings.json. Remember that in ASP.NET Core applications, by convention the environmental variable configurations are loaded last and overwrite anything in the appsettings.json files.

One thing about configuration in this adventure is making sure that your configuration supports anywhere you run. That is why you see this crazy overlap of appsettings.json files or environmental variables managed all over the place. You have to develop a good understanding of how the configuration systems work in the runtime environments and take advantage of them.

While we are doing this work, I also adjusted the ports that everything lives at. There are some mistakes in the Skoruba template that state that the Admin API is at 5001. So in this exercise I've set:

• STS to port 80 (no port)
• Admin is at port 9000
• Admin API is at port 5000

These go along with pgAdmin4 already being at port 5050 and Seq being at port 5341.

Database Connection strings

The Skoruba uses 5 different DbContext to do its work. This in theory allows you to maintain smaller, more specific entity sets and could also spread some of this persistence work across different servers, but in practice, we'll only use one server. You can replace all connection strings with this one connection string.

1

Server=postgres; User Id=admin; Database=identity; Port=5432; Password=P@ssw0rd!; SSL Mode=Prefer; Trust Server Certificate=true;

Notice that we can reference the Postgre DB at its DNS alias of postgres.

Tweak Code to Run Migrations Always

You can see that the Skoruba team already had this idea in mind when they created the template. You just need to make the switch in Program.cs around line 32 and the MyProject.Admin project.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

// Uncomment this to seed upon startup, alternatively pass in `dotnet run /seed` to seed using CLI
await DbMigrationHelpers.EnsureSeedData<IdentityServerConfigurationDbContext,
                                        AdminIdentityDbContext,
                                        IdentityServerPersistedGrantDbContext,
                                        AdminLogDbContext,
                                        AdminAuditLogDbContext,
                                        UserIdentity,
                                        UserIdentityRole>(host);
//if (seed)
//{
//    await DbMigrationHelpers
//        .EnsureSeedData<IdentityServerConfigurationDbContext, AdminIdentityDbContext,
//            IdentityServerPersistedGrantDbContext, AdminLogDbContext, AdminAuditLogDbContext,
//            UserIdentity, UserIdentityRole>(host);
//}

Configure Serilog to use Seq

This is going to require a little bit of work in Visual Studio. The Skoruba template already makes use of Serilog as the logging engine, but it doesn't have the Seq sink that will be used to send all of the structured log messages to Seq. We need to add that bit to the Identity applications.

1. Add the Serilog.Sinks.Seq package to all three Identity projects
2. Change the configuration in the /shared/serilog.json file that will be used by the docker containers.

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

   { "Serilog": { "Using": [ "Serilog.Sinks.Console" ], "MinimumLevel": { "Default": "Debug", "Override": { "Microsoft": "Information", "System": "Error" } }, "WriteTo": [ { "Name": "Console", "Args": { "outputTemplate": "[{Timestamp:o}][{Level:u4}][{SourceContext}] {Message}{NewLine}{Exception}" } },{ "Name": "Seq", "Args": { "serverUrl": "http://seq:5341" } } ], "Enrich": [ "FromLogContext", "WithMachineName" ], "Properties": { "Product": "IdentityServer4", "Platform": "Docker" } } }

3. Optional Change the individual serilog.json files in each project to approximately match. Remember that the docker containers all use the shared/serilog.json file, not the individual serilog.json file via the volumes: directive

   1 2 3 4

   volumes: - "./shared/serilog.json:/app/serilog.json" - "./shared/identitydata.json:/app/identitydata.json" - "./shared/identityserverdata.json:/app/identityserverdata.json"

Ready to run

Alright! We've scaffolded the IdentityServer4/Skoruba applications, we've put all of our infrastructure in place, and we've adjusted all of our configuration! We should be ready to run this application!

1. Select the docker-compose project in the solution explorer!
2. You should see only the docker-compose option in the Visual Studio Run button
3. Run your docker-compose orchestration!

Now we can start to explore what we've got running in the container network.

1. pgAdmin4
2. Seq
3. STS
4. Admin
5. AdminApi

pgAdmin4

You should be able now to navigate to http://127.0.0.1.xip.io:5050 in order to see the pgAdmin4 login screen.

Now we enter our login credentials that we set in the pgAdmin4 section in the docker-compose.yml file.

1
2
3
4
5
6
7
8
9
10
11

pgAdmin4:
  image: dpage/pgadmin4:4.20
  ports:
    - "5050:80"
  container_name: pgAdmin4
  environment:
    - "PGADMIN_DEFAULT_EMAIL=admin@codingwithdave.xyz"
    - "PGADMIN_DEFAULT_PASSWORD=P@ssw0rd!"
    - "PGDATA=/mnt/data/pgdata"
  volumes:
    - pgdata:/mnt/data/pgdata

Once we've logged in! We need to add a connection to the Postgre database into pgAdmin4.

1. Create Server Listing Entry
2. Enter server location (DNS)
3. Enter Postgre server admin credentials

   1 2 3 4 5 6 7 8 9 10

   postgresdb: image: postgres:alpine hostname: postgres ports: - "5432:5432" container_name: postgresdb environment: - "POSTGRES_USER=admin" - "POSTGRES_PASSWORD=P@ssw0rd!" - "POSTGRES_DB=identity" # this is the DB name that will be generated by EFCore

   

And voila! We can administer our Postgre-based Identity store that is used by the 3 applications in the platform!

We can now run Postgre queries and sql commands to look at or manipulate our data.

Seq

Looking at Seq is a little easier! We just need to go to the http://127.0.0.1.xip.io:5341 URL. Since we are in a single-user license and with no authentication turned on or hooked up, we'll simply land on the query screen!

Now, we can start to just get a taste of the benefit of a embedded log-ingestion application. We can see that the apps are all logging into the Seq platform.

And now we can start to look at basic loads across all three applications in the same timeframe.

I'll leave a much deeper exploration of what Seq can do for you as homework! I know your probably already at home (or work?!?!) but I'm not going to directly explore Seq's capabilities in this blog post! I'll save that for another series. But you should definitely go check out Seq by DataLust.co.

I am not affiliated with Datalust or Seq in anyway. I get no money for this. I just really like the product.

STS

Database. Check!Database Admin. Check!Log Ingestion. Check!

Now we get to see the applications (finally) that is the reason we are doing all of the rest of this work.

Ensure the DockerCompose project is running in Visual Studio.

If you navigate to http://127.0.0.1.xip.io you will land on the STS login page.

If you have another web server running and serving pages on port 80 you may have a conflict here.

We can log into this application using the account that was created for us! We saw the admin account in the pgAdmin4 query of the Users table!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

{
  "IdentityData": {
    "Roles": [
      {
        "Name": "IdentityAdminRole"
      }
    ],
    "Users": [
      {
        "Username": "admin",
        "Password": "P@ssw0rd!",
        "Email": "admin@codingwithdave.xyz",
        "Roles": [
          "IdentityAdminRole"
        ],
        "Claims": [
          {
            "Type": "name",
            "Value": "admin"
          }
        ]
      }
    ]
  }
}

Once we are logged into the STS, we can see that users can manage their details. They can look at the applications they've granted permissions to, they can look at their profile data. They can delete their personal data, turn on MFA, and change their password! Phew!

Again, I'll leave a deeper exploration of the STS for homework. You should have enough familiarity after working through the IdentityServer4 Quickstarts that most of this will seem familiar. Also, this is only the user profile information, so it isn't exciting.

Admin

Next, we'll open a tab to the Admin application. If you are already logged into the STS, you won't be challenged for a username/password but have no doubt, you were authenticated!

Navigate to http://127.0.0.1.xip.io:9000 you will land on the Admin application landing page.

If you were not logged in, you would have been re-directed to the STS to enter your credentials, prove who you were, and then returned to the Admin page. Give it a try. Log out of the STS and try going to the admin URL again!

There is a lot to learn about administering an IdentityServer4 implementation. I'm going to cop out again and leave you to explore this as homework.

Admin Api

Last but not least, the Swashbuckle Api Explorer application (swagger) that is embedded in the AdminApi project!

Navigate to http://127.0.0.1.xip.io:5000/swagger you will land on the Admin WebApi ApiExplorer page.

The Api Explorer is already setup to leverage token-based authentication, so if you hit the Authorize button, you'll be directed to log in (authenticate) with the STS (if you haven't already). If you are already authenticated, you may be asked by the Api Explorer app for your consent to use your profile details, but you won't have to type in your username/password again. And if you allow your consent to be remembered, you won't be challenged for it again.

If you execute any of the API methods before you are authorized, you'll get the dreaded 401 Unauthorized HTTP status code!

Now get authenticated!

And this time, we are authorized and our API call succeeds!

Summary

Now that is it! Wow! That was a lot of work! But we have been rewarded by a fully functioning IdentityServer4 running on our machine with a Postgres DB as persistence, pgAdmin4 for database administration, and Seq for log ingestion!

One thing I didn't do for demonstration purposes is do any sort of custom-branding on the Skoruba applications, but they are just ASP.NET Core 3.1 web applications that you already know how to modify and enhance. The Skoruba template has a little bit of built-in branding configuration, but you can certainly charge ahead and make this look however you'd like.

So, there we go. We have a fully functional system running in Docker, but that isn't going to help us where we are going. Time for the next step and we move our platform to Kubernetes!!

Next up:Getting Started with Kubernetes - Minikube

Previously:Building an ASP.NET Core IdentityServer Implementation

Getting Started with Kubernetes - Minikube - Part A

We're finally getting to the part of the series where we have a group of applications and we want to have them live in Kubernetes (k8s). In order to do that, we need to have k8s on our local development machine and we are going to use Minikube to do that.

Important Caveat

I'm going to make an assumption that at a minimum, you've reviewed various types of k8s resources on kubernetes.io and in a best case scenario, you've watched Nigel Poulton's Pluralsight course Getting Started with Kubernetes. If you haven't, my discussions about these topics may be harder to understand because they do not cover these basics.

Also, my understanding of Kubernetes is certainly not as extensive as I'd like. I'm not sure I'd hazard calling myself an expert. I've got a working cluster and applications in that cluster but I am not going to make the statement that I've done it all right or with the current best practices in place. This is a learning exercise for me (and you) and while I want to get you with a cluster up and running as soon as possible, I expect you to learn/challenge/grow your k8s cluster knowledge as well.

Getting started

Thankfully, I don't have to create a huge blog post on this! There is already some great documentation at kubernetes.io that gets you setup with a learning environment based on minikube so I'll just let you go there and read the installation guide! I'm using v1.9.2 of minikube during the creation of this series of articles.

A Powershell Script

It is certainly easy enough to remember some commands when standing up your environment, but as we work through getting everything into minikube using the command-line and manifests, it can become quite a list of commands and so I'd recommend creating a powershell script that you can capture your commands in the order that you are likely to execute them.

The way that I've been structuring my source files is:

1
2
3
4

project_root
      |- infra
      |- manifests
      |- src

The src folder is where the ASP.NET Core applications are. You can ignore the infra directory for a while, but for this part of the journey, we'll be using the manifests folder to store all of our k8s declarative manifest files. This is also where I store my stand-up.ps1 powershell script file!

The first line in your stand-up.ps1 file should probably be (if your on windows):

1

minikube start --vm-driver=hyperv --cpus=4 --memory=16g --extra-config=apiserver.service-node-port-range=80-33000

This is saying:

• start minikube
  • use the hyperv driver
  • with 4 logical CPUS (4 of the n you have in Task Manager - Performance Tab)
  • with 16 gb of memory
  • 20 gb of disk space (default value)
  • and expanding the range of usable ports in minikube

You're going to want to adjust these numbers to what make sense for your workstation. I have a i9-9900k with 64 gbs of memory, so these numbers makes sense for me. You can certainly run all of this on 1 CPU and 4 gb of RAM with no problems. As you build k8s clusters with more hosted pods, you'll probably need to increase the values when starting minikube.

It is important to understand that this command creates a virtual machine, running linux, in hyperv, on your local workstation. If you want to change these parameters, you'll need to destroy your current minikube instance and re-create a new one.

Out of the box, k8s (minikube) limits the port range of containers in the cluster to 30000-33000. I've expanded this range because I want to use the same port values that we used in docker. This expanded range allows k8s in minikube to use more ports, but it doesn't claim all of them.

After the minikube VM has been created (which may take a few minutes), we can then invoke our next command to make sure it is up and running!

1
2
3

# Start a tunnel from your local machine into minikube and the kubernetes dashboard
# 127.0.0.1 should open in a browser window for you and it should be the Kubernetes Web UI in minikube
Start-Process -NoNewWindow minikube dashboard

Using this powershell command, the process creating the tunnel is started and control of the shell is returned to you, but the process remains running. When you close down this shell, the running process will also be closed and the tunnel will close.

Hopefully, the tunnel started, a browser window opened and you can see the Kubernetes Web UI for your new minikube k8s cluster! Let's leave this browser window open and we can keep our eyes on it.

Putting your backend into the cluster

Now that k8s is running on your local machine, we can start to install our backend services into the cluster. We'll do this activity first, with easy to configure pods, to get used to working with manifests. Installing our IdentityServer4-based applications will require some additional resources and automation.

In this section, we will start the process of converting our docker-compose.yml into a bunch of k8s manifest files. We could do this as a monolithic manifest file, but I prefer smaller manifest files. They are easier to think about and just as easy to use.

Postgres manifests

If we look at the postgres section of our docker-compose.yml file, we will see a couple of things.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

postgresdb:
  image: postgres:alpine
  hostname: postgres
  ports:
    - "5432:5432"
  container_name: postgresdb
  environment:
    - "POSTGRES_USER=admin"
    - "POSTGRES_PASSWORD=P@ssw0rd!"
    - "POSTGRES_DB=identity" # this is the db name that will be generated by EFCore
  volumes:
    - postgresdata:/var/lib/postgresql/data
  networks:
    default:
      aliases:
        - postgres

• an image from DockerHub
• container name
• ports that are exposed
• environmental variable declarations
• attached persistent volumes
• network alias

In k8s, those concepts are separated into different types of resources that we will want to provision. Generally speaking, the container-based resources will be described in a Deployment manifest and the network-based resources will be described in a Service manifest.

I tend to clump container-based concerns into a manifest file. So in this case, with postgres, I will also describe the PersistentVolume resources in with the Deployment resources.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  hostPath:
    path: /var/lib/postgresql/data
# divider to separate resource declarations in a yaml file
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-dep
  labels:
    app: postgres
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      volumes:
        - name: postgres-pv-storage
          persistentVolumeClaim:
            claimName: postgres-pv-claim
      containers:
        - name: postgres
          image: postgres:alpine
          env:
            - name: POSTGRES_USER
              value: "admin"
            - name: POSTGRES_PASSWORD
              value: "P@ssw0rd!"
            - name: POSTGRES_DB
              value: "identity"
          ports:
            - containerPort: 5432
          volumeMounts:
            - mountPath: "/var/lib/postgresql/data"
              name: postgres-pv-storage

Let's decompose that Deployment manifest in more detail.

PersistenVolume

From kubernetes.io...

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. It is a resource in the cluster just like a node is a cluster resource.

So we need to have some disk space made available for us to use in the cluster.

PersistentVolumes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-pv-volume
  labels:
    type: local
    app: postgres
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  hostPath:
    path: /var/lib/postgresql/data

The TL;DR; (too long; didn't read it) description of this resource description is:

• Create a volume on the linux host (k8s node)
  • at the path described
  • make it 1Gi large
  • only let one node (VM/host) connect to it

PersistentVolumeClaim

From kubernetes.io...

A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory).

We need to claim some of the disk resources from the cluster (the PersistentVolume) in the same way that we would request some CPU or memory capacity from the cluster. It is important to explore the relationship between PV and PVC. This blog article provides some insights as does this Stack Overflow Question/Answer.

As I understand it, we will need to do this PV+PVC technique because we are using minikube, not AKS, and the storageClassName that we are using (manual) doesn't allow for dynamic provisioning of the storage resource. This will change when we move to AKS.

PersistentVolumeClaims

1
2
3
4
5
6
7
8
9
10
11
12
13

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pv-claim
  labels:
    app: postgres
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Deployment

So we have the persistent storage that we want for our database setup. Now we need to get postgres deployed into our cluster.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-dep
  labels:
    app: postgres
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      volumes: #this pod is going to claim this PVC and call it postgres-pv-storage
        - name: postgres-pv-storage
          persistentVolumeClaim:
            claimName: postgres-pv-claim
      containers:
        - name: postgres
          image: postgres:alpine
          env:
            - name: POSTGRES_USER
              value: "admin"
            - name: POSTGRES_PASSWORD
              value: "P@ssw0rd!"
            - name: POSTGRES_DB
              value: "identity"
          ports:
            - containerPort: 5432
          volumeMounts: # this container is going to mount the volume that is the claimed PVC
            - mountPath: "/var/lib/postgresql/data"
              name: postgres-pv-storage

There is a lot that goes into a Deployment manifest. You can see more details here in the event that you need a refresher.

We've created this manifest that will stand-up Postgres in the k8s cluster. Now we can use kubectl to run this manifest against our cluster.

First, let's make sure our kubectl is configured to point at minikube.

kubectl config current-context should report minikube.

Next, let's run the postgre-dep.yml file through kubectl.

kubectl create -f .\postgres-dep.yml

1
2
3
4
5
6
7

PS D:\temp\testidentity\MyProject\manifests> kubectl create -f .\postgres-dep.yml

persistentvolume/postgres-pv-volume created
persistentvolumeclaim/postgres-pv-claim created
deployment.apps/postgres-dep created

PS D:\temp\testidentity\MyProject\manifests>

Now go to the browser window that is showing you the Kubernetes Web UI and you should see all of the new resources in your cluster!

PersistentVolume In the Cluster

Notice how this PV is a resource in the cluster and not related to any particular pod.

Postgres-based Resources

If you'd like to see only the Postgres resources,you can use the Search bar to filter everything else out.

And now we can see all of the resources we just deployed, and explore them in more detail individually.

So that completes putting our Postgres database into the cluster! Now we have to expose it.

Postgres Service Manifest

From kubernetes.io...

An abstract way to expose an application running on a set of Pods as a network service.With Kubernetes you don’t need to modify your application to use an unfamiliar service discovery mechanism. Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods, and can load-balance across them.

Services

Generally, you want to present as little surface area for security reasons as possible, so I have no desire to expose the database outside of the cluster. We still need a service resource created so that the container gets a DNS name within the cluster, and as containers/pods come and go, the cluster will ensure that the DNS entry always points to the right place.

So in this service manifest, we are basically telling k8s to map the DNS entry postgres-svc to whatever pod spins up with the label app:postgres.

1
2
3
4
5
6
7
8
9
10

apiVersion: v1
kind: Service
metadata:
  name: postgres-svc
spec:
  selector:
    app: postgres
  ports:
    - protocol: TCP
      port: 5432

pgAdmin4 Deployment manifests

The pgAdmin4 deployment manifest is very similar to the postgres-dep.yml file we've created already. So I'll just post it in here so you can take a look at it. The only things that are really different are the environmental variables and the names of things.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pgadmin4-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  hostPath:
    path: /var/lib/pgadmin4/data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pgadmin4-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pgadmin4-dep
  labels:
    app: pgadmin4
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: pgadmin4
  template:
    metadata:
      labels:
        app: pgadmin4
    spec:
      volumes:
        - name: pgadmin4-pv-storage
          persistentVolumeClaim:
            claimName: pgadmin4-pv-claim
      containers:
        - name: pgadmin4
          image: dpage/pgadmin4
          env:
            - name: PGADMIN_DEFAULT_EMAIL
              value: "admin@codingwithdave.xyz"
            - name: PGADMIN_DEFAULT_PASSWORD
              value: "P@ssw0rd!"
          ports:
            - containerPort: 80
          volumeMounts:
            - mountPath: "/var/lib/pgadmin/data"
              name: pgadmin-pv-storage

pgAdmin4 Service manifests

Since I do want to be able to access the pgAdmin4 application from outside of the cluster, we need to make a service manifest that exposes the pgAdmin4 pod.

1
2
3
4
5
6
7
8
9
10
11
12

apiVersion: v1
kind: Service
metadata:
  name: pgadmin4-svc
spec:
  selector:
    app: pgadmin4
  type: NodePort
  ports:
    - protocol: TCP
      port: 80
      nodePort: 5050

Simply speaking, this manifest says to create a NodePort service resource for this pod and expose it on port 5050. Notice that the spec: selector: is saying that this service will be applied to pods with the same label of pgadmin4. Remember, in k8s, labels are usually very important. And because of the type: NodePort on this service, we will be able to access this pod from outside of the cluster.

If you want to expose your Postgres resource outside of the cluster, you can change that manifest to create a type: NodePort service exposing the postgres pod on port 5432.

Now we need to use kubectl to create this resource in the cluster.

kubectl create -f .\pgadmin4-svc.yml

You should see the new pgAdmin4 resources in your cluster.

We can also now use another feature of minikube which allows you to expose services in your cluster to the outside world on minikube's IP address. In order to expose pgAdmin4, simply type:

minikube service pgadmin4-svc

Minikube should then create a tunnel into the cluster for you, and open a web browser to the URL that was generated!

You can also use minikube service list at any time to see a list of the exposed services in your cluster.

1
2
3
4
5
6
7
8
9
10
11
12

PS D:\temp\testidentity\MyProject\manifests> minikube service list  

|----------------------|---------------------------|--------------|----------------------------|
|      NAMESPACE       |           NAME            | TARGET PORT  |            URL             |
|----------------------|---------------------------|--------------|----------------------------|
| default              | kubernetes                | No node port |                            |
| default              | pgadmin4-svc              |           80 | http://172.28.129.202:5050 |
| default              | postgres-svc              | No node port |                            |
| kube-system          | kube-dns                  | No node port |                            |
| kubernetes-dashboard | dashboard-metrics-scraper | No node port |                            |
| kubernetes-dashboard | kubernetes-dashboard      | No node port |                            |
|----------------------|---------------------------|--------------|----------------------------|

You should be able to log into pgAdmin4 with the credentials we've come to know and love (user: admin@codingwithdave.xyz pwd: P@ssw0rd!). Once in there, you will be able to re-create your server list entry, but this time, the location of the server is the name of the postgres service, as described in the metadata: element of the yaml.

1
2

metadata:
  name: postgres-svc

Seq Deployment manifests

The Seq deployment manifest is very similar to the others we've created already. So take a look, and then put Seq into your minikube cluster.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

apiVersion: v1
kind: PersistentVolume
metadata:
  name: seq-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 3Gi
  hostPath:
    path: /mnt/data/seqv6/
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: seq-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: seq-dep
  labels:
    app: seq
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: seq
  template:
    metadata:
      labels:
        app: seq 
    spec:
      volumes:
        - name: seq-pv-storage
          persistentVolumeClaim:
            claimName: seq-pv-claim
      containers:
      - name: seq 
        image: datalust/seq:preview
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: "/data"
          name: seq-pv-storage
        env:
        - name: ACCEPT_EULA
          value: "Y"

kubectl create -f .\seq-dep.yml will get your resources created!

Seq Service manifests

We want to see Seq outside of the cluster, so we will also want to expose it with a NodePort service.

1
2
3
4
5
6
7
8
9
10
11
12

apiVersion: v1
kind: Service
metadata:
  name: seq-svc
spec:
  selector:
    app: seq
  type: NodePort
  ports:
    - protocol: TCP
      port: 80
      nodePort: 5341

kubectl create -f .\seq-dep.yml will get your service resource created!

Then expose your service via:

minikube service seq-svc

And voila! A browser window opens and you'll see Seq!! Nothing is logging there yet, but it's there! Woo hoo!!

Summary

Hopefully now, you've shifted all of the backend services required for our IdentityServer4 applications into minikube! They are all up and running, and you've been able to access pgAdmin4 (connected to the postgres db) and you've been able to access Seq, even if nothing is logging to it!

This has been really long post, so I'm going to take a break, let you have a break, and continue on to Part B of putting our system in Minikube!

Next up:Getting Started with Kubernetes - Minikube - Part B

Previously:Getting Started with Kubernetes - Minikube - Part A

Getting Started with Kubernetes - Minikube - Part B

We're finally getting to the part of the series where we have a group of applications and we want to have them live in Kubernetes (k8s). In order to do that, we need to have k8s on our local development machine and we are going to use Minikube to do that.

Important Caveat

I'm going to make an assumption that at a minimum, you've reviewed various types of k8s resources on kubernetes.io and in a best case scenario, you've watched Nigel Poulton's Pluralsight course Getting Started with Kubernetes. If you haven't, some of the stuff I discuss will not have sufficient detail in this post to close the gap.

Also, my understanding of Kubernetes is certainly not as extensive as I'd like. I'm not sure I'd hazzard calling myself an expert. I've got a working cluster and applications in that cluster but I am not going to make the statement that I've done it all right or with the current best practices in place. This is a learning exercise for me (and you) and while I want to get you with a cluster up and running as soon as possible, I expect you to learn/challenge/grow your k8s cluster knowledge as well.

Continuing on...

So we've now deployed all of our backend services into minikube. There is a little detail that probably went un-noticed until now. Where did minikube get the container images from? In this case, it pulled the postgres, pgAdmin4, and Seq container images from DockerHub, a public container registry. So if you want to publish your custom-built Skoruba templates to DockerHub, everything will keep on working without any changes. But chances are, your going to want to publish those images to a private container registry such a DockerHub (private) or in my case, an Azure Container Registry hosted in our Azure subscription.

What about my local docker repository?

You may be asking, you've built your app locally and deployed it to Docker Desktop registry, why can't I just get it from there?

Minikube (k8s in the VM) has it's own docker daemon instance running, so now you have two docker daemons running on your machine. One in Docker Desktop and one in minikube. And the one in minikube didn't build/deploy your applications, the Docker Desktop one did. But minikube doesn't use that one. It uses its own internal docker container registry or a public/private one from the internet. I've read that you can use minikube's docker daemon for builds, but I don't want to make any adjustments to your environment that might break the Docker Desktop experience so we'll just continue on with creating a private container registry. We need to get there eventually no matter what, so let's keep going!

Azure Container Registry (ACR)

For this part of the journey, we are going to leave Visual Studio and the comfort of C# and yaml and journey to the Azure Subscription that we want to host our private container registry in!

At this point, you'll need to have access to a Azure subscription. Microsoft has made it very easy to create them (they are free) and use them (there are lots of free services)! The ACR service is not free, but it is pretty inexpensive for the Basic SKU which holds up to 10gb of container images.

If an Azure ACR is not in the cards for you, you can continue (without my help for the moment) with a free DockerHub public registry, or a private container registry that you may already have access to.

For this part of the project, I'll be using my Depth Consulting Azure Subscription and Azure DevOps Service instance. My client, who has the identity management business problem, also has Azure DevOps Server 2019 and an Azure subscription that holds their private ACR.

Creating the ACR

Creating a ACR is very easy.

1. Log into the Azure Portal
2. Create a new ACR Resource
   
3. Follow the wizard and complete the creation of your ACR
   • pick the Basic SKU. It is more than enough for now and you can upgrade later.
   • The Basic ACR costs about $0.17 per day for 10GiB of storage.

Tada! ACR created!

I could create the ACR via Pulumi but it is a one-and-done kind of activity, so I didn't do that. It's just easier to create it in the portal.

Secrets! Secrets! Secrets!

In order to access a private container registry, you need credentials! And more specifically, your k8s cluster needs the credentials for your private container registry. And it isn't just k8s that needs those credentials, your DevOps pipeline will need those credentials as well in order to publish container images into the registry.

Our ACR credentials are available in the Azure Portal. You can find the ones we'll need there.

Remember where they are. When we get back to manifest land, we'll need them.

Azure DevOps Service (Server 2019)

Now that we have a private container registry, we need to build our IdentityServer4 applications and publish those container images to our ACR. If you do not have an Azure DevOps Service instance, you can create one for free.

I will try to create blog post sometime doing something like this from GitHub, using GitHub Actions with a public DockerHub Registry.

Assumptions

I'm progressing on the assumption that your IdentityServer4 implementation projects have been checked into a git repository somewhere. If this is in an Azure DevOps Services instance, that's great, but it can be GitHub or any other repository that Azure DevOps Pipelines can monitor for changes.

Build and Publish in one Pipeline

The next step is to build and publish our applications. For this, we're going to leverage Azure DevOps Pipelines that contain a couple DockerCompose tasks. Thankfully, this is very easy to setup in thanks to the pipeline templates and the integration between Azure Portal and Azure DevOps Services.

1. Go to your Pipelines in Azure DevOps Service instance

2. Create a new Pipeline

3. Connect the pipeline to your repository

4. On the Configure your pipeline wizard step, select Docker - Build and Push an Image to Azure Container Registry

5. Select your Azure subscription with the ACR in it

6. Select your ACR that you want the images in

   • don't worry about the image name or Dockerfile parameters. They'll be deleted in a moment

7. Create the new pipeline

8. Delete the whole Docker@2 task

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

   stages: - stage: Build displayName: Build and push stage jobs: - job: Build displayName: Build pool: vmImage: $(vmImageName) steps: - task: Docker@2 # <-- remove this whole task displayName: Build and push an image to container registry inputs: command: buildAndPush repository: $(imageRepository) dockerfile: $(dockerfilePath) containerRegistry: $(dockerRegistryServiceConnection) tags: | $(tag)

9. Delete the Docker@2 task variables

   1 2 3 4 5 6 7

   variables: # Container registry service connection established during pipeline creation dockerRegistryServiceConnection: '<your service connection Id>' imageRepository: 'identityserver' # <-- delete containerRegistry: 'depthconsulting.azurecr.io' # <-- delete dockerfilePath: '$(Build.SourcesDirectory)/src/DepthConsulting.Identity.Admin.Api/Dockerfile' # <-- delete tag: '$(Build.BuildId)'

10. With your cursor under the steps yaml entry, Add a Docker Compose task

    • Work through the wizard and the Command is build

11. With your cursor under the last DockerCompose@0 task, Add another Docker Compose task

    • Work through the wizard again and the Command is push

12. Save the pipeline and run it!

    • You may need to give your pipeline permission to use a service principal to connect to the ACR. If you see that your pipeline is queued and waiting for persmission, go ahead and give it permission.

Your final pipeline yaml should look something like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

# Docker
# Build and push an image to Azure Container Registry
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
- master

resources:
- repo: self

variables:
  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: '<your service connection guid here>'
  tag: '$(Build.BuildId)'
  
  # Agent VM image name
  vmImageName: 'ubuntu-latest'

stages:
- stage: Build
  displayName: Build and push stage
  jobs:  
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: DockerCompose@0
      inputs:
        containerregistrytype: 'Azure Container Registry'
        azureSubscription: '<your subscription id here>'
        azureContainerRegistry: '{"loginServer":"depthconsulting.azurecr.io", "id" : "/subscriptions/<your subscription id here>/resourceGroups/<your resource group here>/providers/Microsoft.ContainerRegistry/registries/depthconsulting"}'
        dockerComposeFile: '**/docker-compose.yml'
        action: 'Run a Docker Compose command'
        dockerComposeCommand: 'build'
    - task: DockerCompose@0
      inputs:
        containerregistrytype: 'Azure Container Registry'
        azureSubscription: '<your subscription id here>'
        azureContainerRegistry: '{"loginServer":"depthconsulting.azurecr.io", "id" : "/subscriptions/<your subscription id here>/resourceGroups/<your resource group here>/providers/Microsoft.ContainerRegistry/registries/depthconsulting"}'
        dockerComposeFile: '**/docker-compose.yml'
        action: 'Run a Docker Compose command'
        dockerComposeCommand: 'push'

After the build has run, we should now see our IdentityServer4 container images in the ACR!

You should navigate the ACR repositories and look at what images are in each repository. Also, all of our images were tagged as latest so every time the build runs, we'll have new a latest image that will be pulled down as needed by the k8s cluster. I'll leave container image tag management to another blog post. We're just trying to get to k8s today.

Adding our ACR Secrets to minikube

Now that we have our IdentityServer4 applications published to a container registery, we need to give our k8s cluster the credentials that will allow it to pull those images down from the ACR into the cluster. To do that, we will create a k8s Secret resource.

Secrets

For this step in the process, the easiest way to do this is to use kubectl to create a secret in the cluster.

1
2
3
4
5

kubectl --namespace default create secret docker-registry docker-credentials `
--docker-server=<acr url> `
--docker-username=<acr username> `
--docker-password=<acr password> `
--docker-email=<your@email.address.com>

Put this command in your start-up.ps1 file for now

This will create a Secret resource in the k8s cluster that it's docker daemon will use to access your private container registry. You can look at the secret that was created using kubectl.

kubectl get secrets

1
2
3

NAME                  TYPE                                  DATA   AGE
default-token-7n4h4   kubernetes.io/service-account-token   3      26h
docker-credentials    kubernetes.io/dockerconfigjson        1      17s

And you can look at the specific secret in yaml with kubectl as well.

kubectl get secret docker-credentials -o yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

apiVersion: v1
data:
  .dockerconfigjson: <snipped>
kind: Secret
metadata:
  creationTimestamp: "2020-04-30T16:15:02Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:.dockerconfigjson: {}
      f:type: {}
    manager: kubectl.exe
    operation: Update
    time: "2020-04-30T16:15:02Z"
  name: docker-credentials
  namespace: default
  resourceVersion: "207328"
  selfLink: /api/v1/namespaces/default/secrets/docker-credentials
  uid: 419c553b-bcaa-49e5-831e-1388c9fad5fe
type: kubernetes.io/dockerconfigjson

You could take this yaml and create a docker-secret.yml manifest file and declaratively create your secret. I'll leave that to you as homework.

Now our k8s cluster should be able to get the container images. We'll prove that in the next section!

Deploying our IdentityServer4 applications to minikube

Now that we have our applications in our container registry, we can build up the manifest files for these applications and we can reasonablly expect our k8s to be able to fulfill the Deployments that we have in those manifest files.

STS

Here is the deployment and service manifests for the STS.

Deployment Manifest

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

apiVersion: apps/v1
kind: Deployment
metadata:
  name: identity-sts-dep
  labels:
    app: identity-sts
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: identity-sts
  template:
    metadata:
      labels:
        app: identity-sts
    spec:
      containers:
      - name: identity-sts
        image: depthconsulting.azurecr.io/depthconsulting.identity.sts:latest
        env:
        - name: ASPNETCORE_ENVIRONMENT
          value: "Development"
        - name: SEQ_URL
          value: "http://seq-svc"
        - name: AdminConfiguration__IdentityAdminBaseUrl
          value: http://127.0.0.1.xip.io:9000
        - name: ASPNETCORE_URLS
          value: "http://+:80"
        ports:
        - containerPort: 80
      imagePullSecrets:
      - name: docker-credentials

Interesting entries in this manifest are:

• Labels are important. They match a Service to a Deployment
• We are telling the deployment where to get the container images from
• our Seq url is the DNS entry created for the Seq post in the cluster
• we are telling k8s to use the docker-credentials secret when trying to pull down the container images from our ACR
  • this allows us to pull images from multiple private repositories if needed

Service Manifest

1
2
3
4
5
6
7
8
9
10
11
12

apiVersion: v1
kind: Service
metadata:
  name: identity-sts-svc
spec:
  selector:
    app: identity-sts
  type: NodePort
  ports:
    - protocol: TCP
      port: 80
      nodePort: 80

A simple Service manifest telling us that:

• use the type: NodePort to expose the STS service to the external network on port 80
• the label will hook this Service up with the pods with the matching label

Admin

Deployment Manifest

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

apiVersion: apps/v1
kind: Deployment
metadata:
  name: identity-admin-dep
  labels:
    app: identity-admin
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: identity-admin
  template:
    metadata:
      labels:
        app: identity-admin
    spec:
      containers:
      - name: identity-admin
        image: depthconsulting.azurecr.io/depthconsulting.identity.admin:latest
        env:
        - name: ASPNETCORE_ENVIRONMENT
          value: "Development"
        - name: SEQ_URL
          value: "http://seq-svc"
        - name: AdminConfiguration__IdentityServerBaseUrl
          value: http://127.0.0.1.xip.io
        - name: AdminConfiguration__IdentityAdminRedirectUri
          value: http://127.0.0.1.xip.io:9000/signin-oidc
        - name: ASPNETCORE_URLS
          value: "http://+:9000"
        ports:
        - containerPort: 80
      imagePullSecrets:
      - name: docker-credentials

Service Manifest

1
2
3
4
5
6
7
8
9
10
11
12

apiVersion: v1
kind: Service
metadata:
  name: identity-admin-svc
spec:
  selector:
    app: identity-admin
  type: NodePort
  ports:
    - protocol: TCP
      port: 80
      nodePort: 9000

Admin Api

Deployment Manifest

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

apiVersion: apps/v1
kind: Deployment
metadata:
  name: identity-admin-api-dep
  labels:
    app: identity-admin-api
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: identity-admin-api
  template:
    metadata:
      labels:
        app: identity-admin-api
    spec:
      containers:
      - name: identity-admin-api
        image: depthconsulting.azurecr.io/depthconsulting.identity.adminapi:latest
        env:
        - name: ASPNETCORE_ENVIRONMENT
          value: "Development"
        - name: SEQ_URL
          value: "http://seq-svc"
        - name: AdminApiConfiguration__IdentityServerBaseUrl
          value: http://127.0.0.1.xip.io
        - name: AdminApiConfiguration__ApiBaseUrl
          value: http://127.0.0.1.xip.io:5000
        - name: ASPNETCORE_URLS
          value: "http://+:80"
        ports:
        - containerPort: 80
      imagePullSecrets:
      - name: docker-credentials

Service manifest

1
2
3
4
5
6
7
8
9
10
11
12

apiVersion: v1
kind: Service
metadata:
  name:  identity-admin-api-svc
spec:
  selector:
    app:  identity-admin-api
  type: NodePort
  ports:
    - protocol: TCP
      port: 80
      nodePort: 5002

Now we can kubectl all of these manifests into our k8s cluster and we should see them all appear in the Web UI.

• kubectl create -f .\sts-dep.yml

• kubectl create -f .\sts-svc.yml

• kubectl create -f .\admin-dep.yml

• kubectl create -f .\admin-svc.yml

• kubectl create -f .\adminapi-dep.yml

• kubectl create -f .\adminapi-svc.yml

Feel free to put put these into a single manifest file or any other combination that makes sense to you.

Now expose all of the services from minikube.

1
2
3
4

# expose the services from minikube
minikube service identity-sts-svc
minikube service identity-admin-svc
minikube service identity-admin-api-svc

Inspect The Pods In Kubernetes

Now is a good time, before we go try to use these applications, to see if the Pods are up and healthy in your k8s cluster in minikube. If you haven't yes, you can use the minikube dashboard command to get to the Kubernetes Web UI or you can go to a command prompt and type octant to start that tool if you've installed it.

Octant

We're taking a look at the state of the pods now because with our approach to putting pod resources into k8s, we are taking an indirect path to getting actual pods running. Our approach is to:

• create a Deployment resource
  • which creates a ReplicateSet resource
    • which creates Pod resource(s) as necessary

In our previous steps, the backend infrastructure resources that we wanted running came from DockerHub (a public repository) and they are pretty easy to get started, so we were unlikely to see any Pod start-up failures. With our own Skoruba-based applications in a private repository, there is a greater chance that our pods will fail to start up, and we'd like to see that now.

Hopefully, everything worked and what you see in Octant is this:

What we see in this image is all the pods having fulfilled their ReplicaSet replicas: pod count and that they are all running.

When I did this the first time, my Pods did not all end up running. I encountered for the first time the dreaded ImagePullBackoff error condition! This is a condition where the k8es control plane is having troubles acquiring an image, for any of a number of reasons, and is now slowing down the rate that it tries to acquire the image.

In this case, I've simply changed the image that the Deployment is trying to pull to create this error condition. If the image name is incorrect, this error condition will occur.

Another reason that it occurred for me was because I had my private container registry credentials wrong. If you are missing the docker-secret or you have the wrong username/password in that secret, you will also get an ImagePullBackoff error condition.

I'd encourage you to give this a try! You only need to kubectl apply -f .\identity-sts-dep.yml with an incorrect image name and you'll see this error. Change it back and apply again and you'll see it fixed. You could also remove the docker-secret to see how that affects things.

It should go without saying, that if you have this problem, you have to resolve this prior to continuing.

We need DNS entries for IdentityServer4

The combination of hosting our k8s cluster on minikube and running this all on our local machine and the fact that this is an bunch of Identity applications where the hostname is really important, we have to do a little extra work. You're going to want to put this into your start-up.ps1 file.

Update your Windows host file

We need to be able to use hostnames for our web applications, so we are going to have to alter our host file in order to make that happen. Here is a powershell script (that you should 100% review yourself!) that will udpate your host file to make this work with the 127.0.0.1.xip.io DNS/hostname. This should not damage your existing host file entries, but please back-up your hosts file first.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# edit your workstations host file to add minikube ip and DNS alias
$hostFileLocation = "C:\Windows\System32\drivers\etc\hosts"
$ip = minikube ip # get the minikube IP adddress put the address in a variable call $ip
$localDnsEntry = "127.0.0.1.xip.io" # our fake DNS entry/hostname

Write-Host "We are going to set the minikube IP address $ip to DNS entry of $localDnsEntry"
$dump =  Get-Content -Path $hostFileLocation
$hostFile = [System.Collections.Generic.List[string]] $dump
$entriesCount = $hostFile.Count
Write-host "Found $entriesCount entries in the hosts file"
if($hostFile.Contains("# minikube section"))
{
    Write-Host "Found a minikube section... replacing it."
    $hostFile.Remove("# minikube section") | Out-Null
    $hostFile.FindAll({param($line) $line.Contains($localDnsEntry)}) | % { $hostFile.Remove($_) | Out-Null }
    $hostFile.Remove("# minikube end section") | Out-Null
}
$hostFile.Add("# minikube section")
$hostFile.Add("$ip`t$localDnsEntry")
$hostFile.Add("# minikube end section")

if ($hostFile.Count -ge $entriesCount) # test to ensure we don't put fewer entries back in host file
{
  Write-Host "Updating our hosts file with $($hostFile.Count) entries."
  Set-Content -path C:\Windows\System32\drivers\etc\hosts $hostFile
}
Write-Host "Done"

This powershell script will change the way that the DNS system works on your Windows workstation. The hosts file takes precedence over a DNS query to the internet, so once we do this, our Windows machine will resolve the IP address we've stated and not the one that would have been return by xip.io. This will break our ability to use DockerCompose. Comment out the entry to use DockerCompose again. The alternative is to create separate domains/configurations for each environment (Docker & minikube) on your local workstation. This is a bit cumbersome with the way seed data is handled today, but it is a problem that could be solved.

You should now be able to visit the following URLs and only be challenged for your username/password once and you will be asked for consent from each set that you visit.

• http://127.0.0.1.xip.io - STS Landing page
• http://127.0.0.1.xip.io:9000 - Admin Application
• http://127.0.0.1.xip.io:5000/swagger - Admin API Swagger Api Explorer

With the Admin Application, if you have not already logged in, you should be re-directed to the STS login page, hence the single-sign on aspect of our authentication ecosystem!

Victory!!

So! I'm crossing my fingers (virtually and into perpetuity) that you've arrived at a place that you can test the IdentityServer4 applications (all three), running in a minikube-based K8S cluster! If you can't, you'll have to work at it a bit and if you really get stuck, try reaching out to me on Twitter! I'm usually paying attention to notifications there!

I certainly encourage you to explore and experiment with k8s and IdentityServer4 in this environment. With these instructions, your manifests, and a little bit of luck, you should be able to always get back to a known working state, even if that means a little bit of lost data.

You can always start from scratch very easily from a minikube k8s cluster perspective. The following commands will tear down and re-build you minikube cluster. This destroys everything and gives you a totally clean slate.

1
2
3

minikube stop
minikube destroy
minikube start --vm-driver=hyperv --cpus=4 --memory=16g --extra-config=apiserver.service-node-port-range=80-33000

Our next stop will be a bit of a reflection point, and then, on to AKS!!

Next up:Pause to reflect

Previously:Getting Started with Kubernetes - Minikube - Part B

Pause to reflect

There was a point in this development effort that things started to click in my head, and I wanted to put this chapter in here as a more concrete reflection point and not leave it to luck that you'll have yours as well. I also wanted to share my epiphany(s) so this spot makes as good a place as any.

There were a bunch of things that clicked for me after moving this from docker to k8s.

Kubernetes is not hard

Kubernetes, all by itself, is very approachable with the tools and the self-guided training I started. With minikube, kubernetes.io, and Pluralsight, I was able to stand-up a cluster and just do simple things like put resources into the cluster! Sure there are some things that turned out to be a bit of a pain in the butt that we'll discuss later on, but Kubernetes itself isn't hard.

Getting a running system in Kubernetes IS hard

So, as easy as kubernetes was, getting an actual working business solution running in kubernetes was much harder! This exercise has really adjusted my perspective about the scope of DevOps and it has adjusted my perspective of what "full-stack" could really mean. Let's step back for a second and look at what I had to create and stand-up in my k8s cluster for things to work.

• a database (Postgres)
  • needs Postgres-specific configuration
  • needs networking configuration
  • needs persistent volumes
  • needs secrets
• a database administration application (pgAdmin4)
  • needs pgAdmin-specific configuration
  • needs networking configuration
  • needs persistent volumes
  • needs secrets
• log ingestion (Seq)
  • needs Seq-specific configuration (minimal)
  • needs networking configuration
  • needs persistent volumes
• raw log consumption (fluentd/graylog/sqelf)
  • needs fluentd-specific configuration
  • needs rule pipelines designed
  • needs networking configuration
• the identityserver4 group (STS, Admin, AdminAPI)
  • build the applications (ASP.NET Core, Web Apps, etc)
  • OAuth2, OpenID Connect is all about configuration!
  • needs networking configuration
• Ingress Controller (nginx and/or traefik)
  • needs nginx-specific configuration
  • needs traefik-specific configuration
  • Azure-specific annotations
• DNS rules (CoreDNS)
  • forwarding and rewrite rule configuration
• certificate management (CertManager)

The interesting thing about all of these applications is that they are all completely independent pieces of software/products that you need to become proficient, if not an expert, in configuring, running, and maintaining. Kubernetes makes it pretty easy to put all of these things in the cluster. Making it all work, in a manner than you can own and operate in an enterprise production setting, is non-trivial and I'll just come out and say it, Hard!

I don't want it to be a doom and gloom story though. There is a light at the end of the tunnel. These are modern pieces of software that are really good to work with. But you still should plan to work with them if you are doing this yourself. If you are on a bigger team with lots of support, then make sure you delegate tasks to those with skills and or experience to get that part of it done. Then automate it, and share the knowledge you've learned.

Getting it running in the cloud is not bad

The good thing about getting it working in the cloud is, your probably going to focus on one cloud. Could you get it working in multiple clouds? You bet! The k8s resources are all the same, it would just be the cloud provisioning application(s) that would be different. Getting to be really familiar with what services your cloud provider offers and how to use those services will be key. But, again, you will need to be very proficient if not an expert in your cloud providers services to own and operate your k8s cluster in the cloud.

The Azure services that we are using and need to understand are:

• Azure Kubernetes Service
• Public IP Address
• Load Balancer
• Azure Storage Accounts
  • azureFile
  • azureDisk
• Virtual Network
• Virutal Machine Scale Set
• Route Table
• Network Security Group

Thankfully, programmatically approaching my infrastructure with Pulumi has made this much easier, especially with intellisense support, so don't worry about this too much. I'll show you what I did shortly.

You do not need to master ALL runtime environments

One thing I discovered quickly is I don't need to keep my web applications running in:

1. Windows Native - IIS
2. Kestrel
3. Docker
4. AKS

Trying to do this will lead to a lot of work that just isn't going to be needed. It is potentially a valuable learning experience so I'm not saying don't do it, but once you decide on your runtime environment(s), do what is best to make sure it is configured and runnable there.

In this case, I felt I did want to keep it running in Docker for the local developer debugging experience. But my primary runtime target was going to be AKS so I didn't spend any time keeping this running in Kestrel or IIS via Visual Studio from a configuration perspective, and Docker did not get the full-fidelity experience that eventually AKS did. I do keep the DockerCompose experience working so I can debug the IdentityServer4 applications on my local machine.

Cluster Logging from the Start

When I started this, I already knew I wanted to do logging in my applications, but I really didn't think about logging across the cluster. I knew it happened and I did solve some problems by looking at the logs in various pods, I should have explored and found fluentd sooner in my journey. Logs are so important in this environment because we have many applications and kubernetes itself spread across multiple nodes and who knows how many pods.

IaC and Automation are Awesome

I am really glad that I started down the path of Infrastructure as Code right from the very start of the AKS journey, and I'm going to get my Pulumi IaC applications running against minikube as well. Being able to capture what you've learned about your infrastructure as application code is a fantastic benefit, in addition to the actual automation that you can invoke with a couple key strokes, the push of a button, or a commit/push. If I was going to give myself some advice, I'd tell myself I should really make sure I pay attention to the next post in the series.

Be Prepared to need a LOT of Permissions

At the beginning of your project, if you aren't in full control of the whole ecosystem, you should start the process of getting the required permissions. This whole series generally assumes you have full-control when we start working on the AKS components, but you'll also need access to DNS registrations, and you may need permissions to acquire and use some of the tooling.

Summary

This was intended to be a brief stop. A chance to pause, catch your breath, and reflect on what you might of learned based on what I did learn. Don't be discouraged if all of this is hard or tricky or taking longer than you expected. This has been an incredible amount of learning if you started from scratch like me with k8s and you should be proud of where you've gotten too.

Next up:Moving to Azure Kubernetes Service - Part A

Previously:Pause to reflect

Moving to Azure Kubernetes Service - Part A

It would be great if we could get this all working in minikube and call it done, but we're not quite that lucky! We're probably going to need a platform with a bit more breathing room and additional capabilities to run our production workloads, so we'll have to figure out a way to move all of this into that platform. In our case, that platform is going to be Azure and the Azure Kubernetes Services (AKS).

With the desire to move our resources into a new k8s cluster in the cloud, there are a lot of moving parts in the infrastructure as compared to what minikube has. Here is a picture of the basic resources we'll have in Azure after we stand up this k8s cluster.

I also had to consider managing the k8s resources (apps in manifests). I want that to be a part of any automation as well.

With all of this in mind, I knew I was going to want something more than a collection of PowerShell scripts to manage the AKS resources and the k8s resources in our cluster. Thankfully, a new product called Pulumi had recently joined the market that looked like it would fit the bill as far as ease of use, community support, and a full IaC ecosystem for me to work with.

This part of the series is mostly going to be about Pulumi, with side discussions about the specific Azure resources that we will instantiate with Pulumi.

Important Assumption

Now that we are moving our activites off of our development machines and into the cloud, it is very important that you have all of the required permissions to act (or for Pulumi to act on your behalf) in your Azure subscription. We will be creating many resources in Azure and you must have permission to create these resources.

Pulumi - Getting Started

Pulumi is a platform that includes:

1. A cloud-platform that stores data about your preferences, your settings for projects (stacks), and the results of your executions.
2. Multiple language-specific SDKs (see languages below) that allow you to create a Pulumi application that will run and deploy your infrastructure. You can choose the language you are most comfortable with to write your application.
3. The Pulumi CLI tool that will allow you to manage your infrastructure and run your application to stand-up, tear down, or manage your project (stack).

In addition to the actual tooling, there is a tremendous amount of documentation and community support. I've generally been happy with the documentation even though I think it is still lacking in a couple places, but the community support has been really good. Pulumi has a Slack that anyone can join; it has logical channels that will generally meet your needs, and the Pulumi team have been very responsive in this slack whenever I encountered a problem.

I don't know about you, but when I have a programming problem, I skip all of the "conceptual" stuff, dive in, and thrash around a bunch. But, if you are inclined to understand the core Pulumi architecture and concepts, you should start reading here.

Creating an Account

Pulumi is a platform and a part of that platform are cloud-based services, associated to an account, that stores your settings, secrets, and outcomes from deployments. Pulumi has 4 pricing tiers, the first of which is Community and is free! This is the one I'm currently using. In the Community edition, your user is basically mapped one to one with an Organization and this organziation can have stacks which are (sort of) the Pulumi term for a deployment target. These stacks are associated with deployment projects so a project can have n stacks in it. The Community SKU of Pulumi is free and so far, it has been everything I needed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Pulumi Organization
  |- User Account(s)
  |- Project A
  |   |- Pulumi application
  |   |- stack (dev)
  |      |- config, history, etc
  |   |- stack (prod)
  |      |- config, history, etc
  |
  |-Project B
      |_ Pulumi application
      |- stack (dev)
         |- config, history, etc
      |- stack (prod)
         |- config, history, etc

You may already know that you need to have more than one person working on this or you may be concerned that you'll outgrow the Community edition, but you shouldn't be concerned. It looks like Pulumi has a seamless upgrade path (that I haven't used) and Pulumi also has a feature that allows you to transfer a stack to another account, so you aren't going to be stuck as you grow with the platform. Additionally, everything that you create to use Pulumi (apps and scripts) is yours and can be version controlled, shared, and re-used as you see fit. It would be quite easy to re-create a stack in a new organization as needed.

So, unless you know you are going to have multiple people involved in the IaC part of the project, you can just create a Community-based account and start deploying!

Install the Tools

Pulumi has a great set of tutorials here for getting started with Azure. I'm going to repeat some of it, but you should definitely check out their learning resources.

Now that you've created an account, it is time to start building your application! First, you'll need to install the Pulumi CLI in your development environment and sign into your cloud account.

Pulumi CLI

You have a couple choices to get the Pulumi CLI!

1
2
3
4

choco install pulumi # requires chocolatey
# -or-
# plain powershell
iex ((New-Object System.Net.WebClient).DownloadString('https://get.pulumi.com/install.ps1'))

Once you have the CLI, you can login via a username/password redirect to a browser or you can use an access token that you've created in the web admin pages for your Pulumi account.

You can use the pulumi whoami to see if you are currently logged in (or who you are logged in as) as well.

Language-specific SDKs

Next, you'll need to consider what language you are going to use when creating your Pulumi application. There are many choices. Typescript/JavaScript, Go, .NET Core (C#, F#, VB), and Python. Pick whichever one your organization has the most skills in. I like Typescript so that is what I picked and what my code will be written in. If you want, you can write an SDK in your favorite language. This is all open-source.

Project Structure

Once you've selected a language, you can use the Pulumi CLI to create your first deployment project and stack. I treat a stack as a deployment that I want to put in a specific environment. For example, I would have 2 stacks for my k8s infrastructure. One is the dev infrastructure in our Development Azure subscription, and the other is the production infrastructure which would be in the Production Azure subscription. These 2 stacks stand-up all of the Azure AKS resources. I would also have 2 stacks for the k8s resources that go into those k8s clusters.

Projects are mostly containers for stacks (configuration, history) and an application. Stacks have specific configuration settings and histories that are important. The application that you run for the project can use each stack for specific deployment details.

A monolithic stack with a single app is a good way to learn and this is how most of the tutorials work. However, I found that it wasn't how I wanted to manage my deployments.

I originally created a single monolithic project with a single stack and one application but have since changed this to two projects with a single application and a dev/prod stack each. The first project for the AKS infrastructure was less volatile and I didn't need to tear it all down all the time. It takes about 18 minutes to stand-up our cluster and 10 minutes to tear it all down. The second project for the k8s resources changed much more frequently and I would often want to clear out the k8s cluster and start from a clean slate. k8s resources can be added or removed from the cluster quickly and frequently. This series will only show the multi-project approach.

First, using the Pulumi CLI, we are going to create a deployment application, in the Typescript language, for the Azure cloud, for only the AKS infrastructure. We'll do our k8s resource deployment application later.

Let's create a folder in our infrastructure folder for the AKS deployment stack.

Once that is done, we can use the Pulumi CLI to build our new project with its initial stack.

pulumi new azure-typescript --secrets-provider=passphrase

This will kick off the workflow to acquire some details before it creates the stack. In my case, I answered the workflow questions with:

1. project name (aks) <-- hit enter and accepted default
2. project description: Deploy our kubernetes infrastructure
3. stack name: (dev) <-- hit enter and accepted default
4. Enter your passphrase to protect config/secrets: P@ssw0rd!
5. azure:environment: (public) <-- hit enter and accept default
6. azure:location: (WestUS) WestUS

After answering those questions, the CLI will finish off by:

• creating your project and first stack
  • saving them in the cloud - this happens automatically
• scaffolding out the initial application files locally
  • pulling down all of the correct npm modules based on your cloud provider selection and language choices.

We can also look in the web portal for our Pulumi account and see the new stack is available there!

You can click the the stack to see what information has been published to the Pulumi cloud. There isn't much there yet, but there are some instructions on how to get more information there. We'll see that shortly.

A Note about --secrets-provider

You should have noticed that I've used the --secrets-provider parameter in the pulumi CLI invocation. If you are going to be building a single stack like many of the pulumi examples you'll find, you will not use or see this parameter. By default, each stack as a unique secrets provider and stacks cannot read each other's secrets. I already plan to have multiple stacks that I want to be able to share secrets between so I need to use this parameter in order to create secrets providers that can reach each other's secrets.

Passphrase is the simplest to use and get working so I'm using that for this article, but you can also use external 3rd party secrets providers. Support providers include:

• awskms: AWS Key Management Service (KMS)
• azurekeyvault: Azure Key Vault
• gcpkms: Google Cloud Key Management Service (KMS)
• hashivault: HashiCorp Vault Transit Secrets Engine

More details about how to use these encryption providers can be found here -- Alternate Secrets Encryption.

We will re-visit secrets in a little while. Now back to our new project.

Scaffolded Files

If we inspect the scaffolded application in the aks folder, we'll see the following:

More Tooling - azure-cli

So, we have the Pulumi CLI, maybe git CLI, and now we need to make sure we have one more tool in place. We need the azure-cli command line tool. Pulumi will use the azure-cli to actually do all of the work in the correct subscription.

You'll need to install the azure-cli with instructions here. I like the little PowerShell script that does it for you myself.

Once the azure-cli is installed, you'll need to log into your Azure subscription that you want to work with.

az login will open a browser window and help you log into your subscription.

az account list will list all of the available subscriptions (if there is more than one)

az account set <subscription name> will set the current context to the desired subscription

If you have multiple subscriptions, you'll probably spend a bit of time switching back and forth. One thing I would suggest is to be careful when working with multiple subscriptions. Pulumi, via the current azure-cli context, will happily deploy or tear-down your infrastructure when asked. There are some safe-guards in place with regard to tear-down or changing, but I've found Pulumi is always happy to stand new things up into a subscription! I accidentally installed a minecraft server into my client's development subscription this way once! Ok, maybe twice!

With Pulumi ready and azure-cli ready, we should be ready to start coding! If you haven't done this already, it's time to open VS Code or your favorite text editor!

Your first Pulumi Application

I like to open VS Code right away for a couple reasons. It is a nice text editor with excellent TypeScript support and it also has a built-in terminal window that I can set to use PowerShell Core and I can leave the directory set to the one that holds the files I'm working in.

Open your index.ts file and take a look at what the Pulumi CLI scaffolded.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

import * as pulumi from "@pulumi/pulumi"; // Add the Pulumi core SDK module to your application
import * as azure from "@pulumi/azure";   // Add the Azure core SDK module to your application

// Create an Azure Resource Group
const resourceGroup = new azure.core.ResourceGroup("resourceGroup");

// Create an Azure resource (Storage Account)
const account = new azure.storage.Account("storage", {
    // The location for the storage account will be derived automatically from the resource group.
    resourceGroupName: resourceGroup.name,
    accountTier: "Standard",
    accountReplicationType: "LRS",
});

// Export the connection string for the storage account
export const connectionString = account.primaryConnectionString;

At the top, you'll see that two modules have been added for you. The Pulumi Core SDK module and the Azure Core SDK module. Depending on what you need, you only add the modules to your application that you are actually using. If you need additional modules, we can use npm install @pulumi/<module name> to get those SDK modules.

Next, we see code that is creating a new ResourceGroup in Azure to hold all of our new resources.

Then, we see code that is creating a new Storage account.

And finally, we have a snippet of code that is going to export the storage account's connection string for use later.

Anything that you export in your TypeScript will be published to the cloud for review or use by another Pulumi stack/application at a later date. If you don't want these properties publicly accessible, do not export them. We will demonstrate this later. This is important to understand when working with stacks that are dependent on other stacks.

The next step is deploying this stack! Go ahead! Type:

pulumi up

You will see the Pulumi CLI kick off your IaC application. It builds the app, does some analysis of what it wants to do, and then asks you if you'd like to continue!

If your TypeScript application won't build, the process stops here, and you have to fix it.

Once you accept, it finishes doing what you've coded, and it deploys your new Azure infrastructure to your subscription.

And it also publishes details into your Pulumi cloud account for this project/stack. You can see the exported connnectionString. Also available in the cloud is a historical log of what has happened in this stack in the Activity tab.

And here are the Azure resources that were created.

Notice that Pulumi has appended a segment of characters on your resource names to try and ensure they are unique within the subscription. I haven't tried to alter that behaviour. You can create a resource directly in Azure and import it into your stack and Pulumi will respect the name it was given.

That's pretty cool! The only problem is, I don't want a lone storage account in my Azure subscription.

So, what do we do now? Tear it all down and let Pulumi clean up everything it created.

pulumi destroy

This will ask you for confirmation, so you are protected that way. Just let the Pulumi CLI finish it's work and go look in your Azure subscriptions! It will be clean as a whistle!

Deploying an AKS

I hope that was a good introduction to Pulumi, but what we really wanted to do was build an application that would deploy our AKS into our subscription. Let's get to that.

Before getting started, delete all of the existing lines of code in your index.ts. We will not be using anything created during the initial scaffolding.

Importing more Pulumi SDK modules

Standing up an AKS service cluster is move involved that a simple storage account. We will need more SDK modules in our application in order to make that happen. Let's add some import statements into our Pulumi application.

1
2
3
4

import * as azure from "@pulumi/azure";
import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
import * as azuread from "@pulumi/azuread";

This is what the top of you index.ts should look like. If you are doing this in VS Code, you probably have some red squiggly lines under the bottom two imports. This is where we ask NPM to go get those modules for us!

npm install @pulumi/kubernetes Get kubernetes module of the SDKnpm install @pulumi/azuread Get Azure ActiveDirectory module of the SDK

Once that is done, the red squiggly lines should go away and you'll see that you can use those SDK modules in your application now.

Initial Configuration Values

The next part of our app initializes and exports configuration variables that we'll need for the AKS provisioning. The names for these variables are intended to be informative, but they are names that I've chosen. The values are determined by the intended usage. Azure expects some of these values to be specific, such as location or nodeSize. The nodeCount variable needs to be an int. The string const values that I export are for consistency in the same way that you would have an enum or a class containing consts values in a C# application. I believe this initial list of variables are the bare minimum you need to create a cluster. You may eventually have many more in your application.

This is what the configuration section will look like when it is complete. We will add these lines of code into the application as we work though the configuration setup so that we can pulumi up multiple times and see the incremental changes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

// Acquire stack configuration values and export application-defined configuration variables
const config = new pulumi.Config();
const password = config.requireSecret("password");
const sshPublicKey = config.require("sshPublicKey");
export const location = config.get("stackLocation") || (config.get("azure:location") || "WestUS");
export const nodeCount = config.getNumber("nodeCount") || 2;
export const nodeSize = config.get("nodeSize") || "Standard_B2s";
export const storageClassName = "managed-premium";
export const resourceGroupName = "rg_identity_dev_zwus_aks";
export const publicIpAddressName = "pip_identity_dev_zwus_aks";
export const k8sDnsName = "identity-auth-dev";

const clientConfig = azure.core.getClientConfig();
export const subscriptionId = clientConfig.subscriptionId;

Pulumi Config Object

const config = new pulumi.Config();

The first thing we do is ask the Pulumi SDK to get our stacks configuration in the form of an object of type pulumi.Config. This object lets us get configuration values (secret/non-secret) for our application, specific to this stack, from the Pulumi cloud. You're probably wondering how they got there though?

The Pulumi CLI has a number of methods that allow us to manage our stack configuration values. In this case, we need to get 5 different values from the cloud.

Passwords and Secrets

Here we get to meet another part of the Pulumi cloud infrastructure. Stacks can contain plaintext configuration information, and they can also contain secret configuration information. We can acquire this configuration information from the cloud when our application runs in order to provision our cluster. Let's work through this for a moment.

This password will be used for our administrative user in our AKS cluster. We probably don't want this to be saved as plaintext anywhere, so we're going to use the --secret flag when we use the Pulumi CLI to set this configuration value in our stack.

1

pulumi config set password --secret [your-cluster-password-here] # P@ssw0rd!

This command tells the Pulumi CLI to set a property on our stack configuration called password to the value provided and make sure it is treated securely.

Since we delete all of the text, let's pulumi up and get that value into the cloud to see what happens.

Other than the initial pulumi new azure-typescript CLI command, any changes we make to our local context wil not be available in the web portal until we use the pulumi up command.

In order to access this secret from the pulumi.Config object, we add this line of code to our application.

const password = config.require("password");

You can use config.requireSecret("password") to mark a variable as secret and at that point it's safe to export because it will always be encrypted/masked (in the state as well as CLI and console)

SSH Public Key

If you'd like to be able to SSH into your linux nodes (VMs) that are in the cluster, you'll need to provide an SSH key that is provisioned into your nodes. Using a tool called ssh-keygen we can create an SSH key and then we can put that key into our Pulumi stack config for use anytime we create the cluster.

1
2

ssh-keygen -t rsa -f key.rsa
pulumi config set sshPublicKey < key.rsa.pub

ssh-keygen will walk you through the process of creating an SSH key.

Then we will use the Pulumi config to set the sshPublicKey configuration variable on the stack. If you are running a PowerShell terminal, this won't work. PowerShell doesn't like the < operator. You can get around that by using this command.

cmd.exe /c "pulumi config set sshPublicKey < key.rsa.pub"

Now you can pulumi up and go take a look at your configuration in the web portal again.

In order to use this variable in our application, add this line of code to our application.

const sshPublicKey = config.require("sshPublicKey");

Location, NodeCount, NodeSize

Azure is going to want to know:

1. What region to create your resources in
2. How many nodes do we want in our cluster
3. What VMs SKUs (size) do we want to use for our cluster

The location configuration value is interesting!

1

export const stackLocation = config.get("stackLocation") || (config.get("azure:location") || "WestUS");

In this code, we look for a configuration value called stackLocation that we can set if we want. This supports DR/HA-specific stack scenarios that I'll discuss later. If it isn't present, we can use the the default location set in the azure:location configuration value that was set for us when we created the project. You can take a look again with the Pulumi CLI command pulumi config get "azure:location" or you can look in the web portal as well. In the event that there is no configuration values, we've provided a fallback value of WestUS.

I love being able to use programmatic logic in my infrastructure deployments!!

az account list-locations will list supported regions for the current subscription. It will spit out a JSON blob of regions and you can use the name property. It seems that commands that take a region parameter name are case-insensitive.

In order to create a k8s cluster, we need VMs (nodes) in the cluster. We can configure how big we want cluster to be and store that data in the stack configuration. Our fallback value is 2.

pulumi config set nodeCount 2

A k8 cluster only needs 1 node to operate. This is certainly sufficient for dev contexts, but you probably want 3+ nodes in production.

Add export const nodeCount = config.getNumber("nodeCount") || 2; to the application.

Azure also needs to know what SKU our nodes (VMs) should be.

az vm list-skus is the azure-cli command that will list out all of the SKUs you can pick from but it spews an enormous JSON blob that lists them all and their capabilities. You're probably better off visiting here to help you decide what SKU to use.

pulumi config set nodeSize Standard_B2s

Add export const nodeSize = config.get("nodeSize") || "Standard_B2s"; to the application.

Again, we provided a fallback value in the application.

pulumi up and look at the configuration values in the web portal.

Exports From Your Application

Finally, we want to provide some const values that will be available in the stack, displayed in the web portal, and also available to any other stack that belongs to the Pulumi organization.

1
2
3
4
5

export const storageClassName = "managed-premium";
export const resourceGroupName = "rg_identity_dev_zncu_aks";
export const publicIpAddressName = "pip_identity_dev_zncu_aks";
export const k8sDnsName = "identity-auth-dev";
export const acrSecretName = "docker-credentials";

pulumi up and you'll see all the rest of our configuration values in the web portal. Exports are shown in a different group in the web portal. The are considered outputs of the stack.

You should recognize when you export a value that you will get a value in the outputs section of the web portal and it also be in the config. You don't have to export const values if you want to avoid that confusion.

Getting the Azure Subscription Id

We need one more value for our application and that is the subscriptionId. In this case, we could use the pulumi config set command to manually set it, but we can get the subscription from the Azure context that we are already connected to. This is exposed via an SDK component.

1
2
3

// getClientConfig is an async call, so wrap in pulumi.output
const clientConfig = pulumi.output(azure.core.getClientConfig());
export const subscriptionId = clientConfig.subscriptionId;

Add AKS resources to Azure

Now that we have the basic configuration values that are required for our AKS cluster resources, we can start to add them into our Pulumi application.

After each section, you can pulumi up and see what happens. When you are done that increment, you can pulumi destroy to clean up the resources.

A Pre-Defined ResourceGroup

While Pulumi is quite capable of building a ResourceGroup from scratch, you may want to use one that already exists in your Azure subscription. Financial reporting, permissions, operational activities, etc may be leveraging ResourceGroups in this way. For this example, we are going to use a pre-existing ResourceGroup. This code is also the reason that we acquired the subscriptionId and set the resourceGroupName in our configuration section.

You will need to create this ResourceGroup in Azure via the azure-cli or in the Azure Portal. The `azure-cli command is

1
2

# this resourceGroupName matches our const value in the config section
az group create --location WestUs --name rg_identity_dev_zwus_aks

Here is the code to get the resourceGroup object for using in the application code.

1
2
3

// get the Azure Resource Group
var resouceGroupId = pulumi.interpolate `/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}`;
const resourceGroup = azure.core.ResourceGroup.get(resourceGroupName, resouceGroupId);

TypeScript string interpolation doesn't work very good with Pulumi Output objects. You need to use the pulumi.interpolate syntax to create strings from Output

Creating an Azure Service Principal

Your new AKS service instance is going to need to be able to create a lot of Azure resources. It will do all of this for you, but in order to create these resources, it will need to log in as a Service Principal that you've created in your subscription. The first thing we'll do is get our application to create that Service Principal.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

// Original example: https://github.com/pulumi/examples/blob/master/azure-ts-aks-helm/README.md
// Create an Azure AD Application
const adApp = new azuread.Application("aksSSO");
export const adAppId = adApp.applicationId;

// Create an Azure Service Principal for that application
const adSp = new azuread.ServicePrincipal("aksSSOSp", { applicationId: adApp.applicationId });
export const adSpId = adSp.id;

// Assign the password from our configuration values to the Service Principal
const adSpPassword:any = new azuread.ServicePrincipalPassword("aksSpPassword", {
    servicePrincipalId: adSpId,
    value: password,
    endDate: "2099-01-01T00:00:00Z"
});

We are not exporting any of these values. We won't need to see them in the Pulumi web portal or use them in any other project or stack. You can see this application and Service Principal in the AAD that your subscription is connected to.

Creating an AAD Service Principal sometimes takes a bit of time. Operations that depend on the SP being created (the AKS Service creation code) may fail until the SP is finished being created. If this happens, simply pulumi up again.

Storage Account for Database backups

Our business problem requires a database and a good thing to do once in a while is backup that database and put those backups somewhere. For this activity, we are creating an Azure Storage Account, in our AKS specific resource group, that we will use as a volume in the pgAdmin4 pod.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

// Create storage account for Azure Files
const storageAccountK8s = new azure.storage.Account("Identity",{
    resourceGroupName: resourceGroupName,
    accountTier: "Standard",
    accountReplicationType: "LRS",
});

export const storageAccountName = storageAccountK8s.name;
export const storageAccountKeyPrimary = pulumi.secret(storageAccountK8s.primaryAccessKey);
export const storageAccountKeySecondary = pulumi.secret(storageAccountK8s.secondaryAccessKey);
export const storageAccountConnectionStringPrimary = pulumi.secret(storageAccountK8s.primaryConnectionString);
export const storageAccountConnectionStringSecondary = pulumi.secret(storageAccountK8s.secondaryConnectionString);

const fileShare = new azure.storage.Share("k8sFileShare", {
    name: "k8s-file-share",
    storageAccountName: storageAccountK8s.name,
    quota: 10
});

export const fileShareName = fileShare.name;
export const fileShareUrl = fileShare.url;

You'll notice that during the creation of the storage account, we get back the connection strings and keys which we'll need to use later on. We can use the pulumi.secret() method to ensure that these are treated as secrets by the Pulumi cloud infrastructure.

Azure Kubernetes Service

We are finally going to create our Azure Kubernetes Service (AKS) instance! w00 h00!! Or I should say, we're going to code it up in Pulumi and we'll let Pulumi take care of creating it!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

// Creates an AKS cluster.
const k8sCluster = new azure.containerservice.KubernetesCluster("aksCluster", {
    resourceGroupName: resourceGroupName,
    kubernetesVersion: "1.17.3",
    location: stackLocation,
    defaultNodePool:{
        name: "aksagentpool",
        nodeCount: nodeCount,
        enableAutoScaling: false,
        vmSize: nodeSize
    },
    dnsPrefix: k8sDnsName,
    linuxProfile: {
        adminUsername: "aksuser",
        sshKey: { keyData: sshPublicKey }
    },
    servicePrincipal: {
        clientId: adAppId,
        clientSecret: password,
    },
/* This is commented out because we do not want to do this. Please see my
   blurb about LogAnalytics at the bottom of this post.
    addonProfile: {
        omsAgent: {
            enabled: true,
            logAnalyticsWorkspaceId: loganalytics.id,
        },
    } */
});

export const nodeResourceGroup = k8sCluster.nodeResourceGroup;

This is a pretty simple pulumi declaration given everything it took to get here. You'll notice the following parameters are mostly using our configuration. This is important when we add additional stacks (deployment target environments):

• resourceGroupName
• kubernetesVersion: "1.17.3"
  • you can only use kubernetes versions supported by Azure. This is the most recent at the time of writing
• location
• nodeCount
• vmSize
• dnsPrefix
• sshKey
• servicePrincpal
  • generated during creation

One of the things that does happen when Azure creates this AKS instance is that the cluster will use the Service Principal to create all of the resources and that all of the cluster resources will be placed in a in an auto-generated ResourceGroup. I haven't discovered to how to alter this behaviour. The name of the resource group that you create the AKS service in will be a component of this auto-generated ResourceGroup's name.

If you pulumi up after this section be aware that creating a cluster takes time and requires the Service Principal to exist as well. pulumi destroy also takes a few minutes to run when an AKS instance is involved.

Interacting with Kubernetes in our Cluster

Now that we have an k8s cluster running, we need to start interacting with the cluster and not Azure. In order to do that, we need a Pulumi object that is comparable to kubectl. In Pulumi, this is the k8s.Provider object.

1
2
3
4
5
6
7
8
9
10
11
12

// Expose a k8s provider instance using our custom cluster instance.
const k8sProvider = new k8s.Provider("aksK8s", {
    kubeconfig: k8sCluster.kubeConfigRaw,
});

// put our new clusterName in Pulumi service
export const clusterName = k8sCluster.name;
// put the az aks get-credentials command in Pulumi service
export const kubeCtrlCredentialsCommand = pulumi.interpolate `az aks get-credentials --resource-group ${resourceGroupName} --name ${clusterName} --context "MyProject.Identity"`;

// Export the kubeConfig
export const kubeConfig = pulumi.secret(k8sCluster.kubeConfigRaw);

In this code, you can see that we create a k8s.Provider instance using the kubeConfig that we can get from our AKS cluster instance. We also want to export that kubeConfig so that we can use it in our next Pulumi application that will create all of the k8s resources in the cluster. Remember, the kubeConfig is credentials to get into your cluster, so you should treat it as a very important secret.

I've also exported the new clusterName and a helper output value of the az aks get-credentials command that will help you put your new k8s credentials in your local kubeConfig file.

We will use the k8sProvider object in the remainder of this script to interact with our k8s cluster.

Kubernetes Secrets

The finish our basic AKS deployment, I am going to install a couple secrets that our k8s cluster will need to operate.

For our Azure Container Registry secrets, we need to get the ACR instance, ask it for it's secrets, and put them into k8s. This is the safest way to manage these secrets since we don't want them in our code base.

Please treat these and all other secrets with all due care.

1
2
3
4
5
6
7
8
9
10

const acrInstanceName = "depthconsulting";
const acrIdentifier:string =  "/subscriptions/<your subscriptionId>/resourceGroups/<your resourceGroup>/providers/Microsoft.ContainerRegistry/registries/<your RegistryName>";
const privateACRInstance = azure.containerservice.Registry.get(acrInstanceName, myAcrIdentifier);

const k8sDockerSecret = helpers.createImagePullSecret(
    acrSecretName, // secret name
    privateACRInstance.adminUsername,
    privateACRInstance.adminPassword,
    privateACRInstance.loginServer,
    k8sProvider);

There is a help function that we are using that hides the complexity of this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

export function createImagePullSecret(
    secretName: string,
    username: pulumi.Output<string>,
    password: pulumi.Output<string>,
    registry : pulumi.Output<string>,
    k8sProvider : k8s.Provider): k8s.core.v1.Secret {

    // Put the username password into dockerconfigjson format.
    let base64JsonEncodedCredentials : pulumi.Output<string> = 
        pulumi.all([username, password, registry])
        .apply(([username, password, registry]) => {
            const base64Credentials = Buffer.from(username + ':' + password).toString('base64')
            const json =  `{"auths":{"${registry}":{"auth":"${base64Credentials}"}}}`
            console.log(json)
            return Buffer.from(json).toString('base64')
        })

    return new k8s.core.v1.Secret(secretName, {
        metadata: {
            name: secretName,
        },
        type: 'kubernetes.io/dockerconfigjson',
        data: {
            ".dockerconfigjson": base64JsonEncodedCredentials,
        },
    }, { provider: k8sProvider })
};

We will also need to create a way for our k8s cluster to connect to the various storage providers that are available to us in Azure. In this case, we want to enable k8s to connect to the file storage we are going to use for our database backups. This uses some of the variables that we captured when creating our Storage Account as well as a helper function.

Microsoft Documentation

This is the kubectl command that would create this secret.

kubectl create secret generic azure-secret --from-literal=azurestorageaccountname=$AKS_PERS_STORAGE_ACCOUNT_NAME --from-literal=azurestorageaccountkey=$STORAGE_KEY

Now we convert that into a TypeScript function.

1
2
3
4
5

const azureStorageSecret = helpers.createAzureFileSecret(
    azureStorageSecretName,
    storageAccountName,
    storageAccountKeyPrimary,
    k8sProvider);

Here is a helper function that will create the secret properly for us.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

export function createAzureFileSecret(
    secretName: string,
    storageAccountName: pulumi.Output<string>,
    storageAccountKey: pulumi.Output<string>, 
    k8sProvider : k8s.Provider): k8s.core.v1.Secret {

    let dataValue = pulumi
        .all([storageAccountName, storageAccountKey])
        .apply(([san,sak]) =>{
            const b64SAN = Buffer.from(san).toString('base64');
            const b64SAK = Buffer.from(sak).toString('base64');
            return { azurestorageaccountname: b64SAN, azurestorageaccountkey: b64SAK };
        }
    );

    return new k8s.core.v1.Secret(secretName, {
        type: "kubernetes.io/generic",
        metadata: {
            name: secretName,
            namespace: "default"
        },
        data: dataValue
    },{provider: k8sProvider});
};

I'll eventually make this a more generic mechanism for creating secrets.

Our Full AKS Application

We have now completed our whole Pulumi application that will stand up a basic AKS cluster in Azure. Here is the entire script for completeness.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155