I found myself last week looking at a bit of code in K8s which I thought I could make better, so I set about trying to understand how to clone, change and test it.

Luckily K8s has some good docs, trust these over me as they’re a great guide. This blog is more of a brain dump of how I got on trying with Devcontainers and VSCode. This is my first try at this so I’ve likely got lots of things wrong.

Roughly I knew what I needed as I’d heard about:

1. Bazel for the Kubernetes build
2. Kind to run a cluster locally
3. Kubetest for running end2end tests

As the K8s build and testing cycle can use up quite a bit of machine power I didn’t want be doing all this on my laptop and ideally I wanted to capture all the setup in a nice repeatable way.

Enter DevContainers for VSCode, I’ve got them setup on my laptop to actually run on a meaty server for me and I can use them to capture all the setup requirements for building K8s.

After a lot of wrangling I got a Dockerfile and devcontainer.json file together which mount and install the required bits (warning: There may be lots wrong with this, I’m pretty new to the world of building K8s from source).

Now all I need to do is clone K8s, drop these two files into a .devcontainer folder and open the folder in VSCode. Once open you get prompted to open the folder in the devcontainer and away you go.

What do you do now?

• Make the change you want to make
• Build an image for kind with your changes:

kind build node-image --type bazel \
 -v 5 --image kindest/node:lawrence

• Start a cluster based off that image:

kind create cluster --name lg \
--image kindest/node:lawrence

• You can run the e2e tests too (still getting to grips with this)

kubetest --up --test --down \
--deployment=kind \
--kind-node-image=kindest/node:lawrence \
--provider=skeleton

The best part? All of this runs nicely in VSCode like its running locally but behind the scenes it’s on a remote server. The left window is my laptop and the right is the server doing a build.

Let me try it

Here is a Gist of my devcontainer files for you to play with. If you spot something silly I’m doing please let me know!

	// For format details, see https://aka.ms/vscode-remote/devcontainer.json or the definition README at
	// https://github.com/microsoft/vscode-dev-containers/tree/master/containers/go
	{
	"name": "Go",
	"dockerFile": "Dockerfile",
	"runArgs": [
	// Uncomment the next line to use a non-root user. On Linux, this will prevent
	// new files getting created as root, but you may need to update the USER_UID
	// and USER_GID in .devcontainer/Dockerfile to match your user if not 1000.
	// "-u", "vscode",
	"–cap-add=SYS_PTRACE",
	"–security-opt",
	"seccomp=unconfined",
	"–name", "kubernetesdevcontainer",
	// Mount go mod cache
	"-v", "k8s-gomodcache:/go/pkg",
	"-v", "k8s-bazelcache:/root/.cache/bazel/",
	// Cache vscode exentsions installs and homedir
	"-v", "k8s-vscodecache:/root/.vscode-server",
	// Keep command history
	"-v", "k8s-bashhistory:/root/commandhistory",
	// Mount docker socket for docker builds
	"-v", "/var/run/docker.sock:/var/run/docker.sock",
	// Use host network
	"–network=host",
	// Mount ssh
	"-v", "${env:HOME}${env:USERPROFILE}/.ssh:/root/.ssh",
	"-v", "/tmp:/tmp",
	// Mount azure, git and docker config
	"-v", "${env:HOME}${env:USERPROFILE}/.azure:/root/.azure",
	"-v", "${env:HOME}${env:USERPROFILE}/.gitconfig:/root/.gitconfig", // Required due to volume mount used for .vscode-server
	],
	"workspaceFolder": "/go/src/k8s.io/kubernetes",
	"workspaceMount": "src=${localWorkspaceFolder},dst=/go/src/k8s.io/kubernetes,type=bind",
	// Use 'settings' to set *default* container specific settings.json values on container create.
	// You can edit these settings after create using File > Preferences > Settings > Remote.
	"settings": {
	"terminal.integrated.shell.linux": "/bin/bash",
	"go.gopath": "/go",
	"go.useLanguageServer": true,
	"[go]": {
	"editor.snippetSuggestions": "none",
	"editor.formatOnSave": true,
	"editor.codeActionsOnSave": {
	"source.organizeImports": true,
	}
	},
	"gopls": {
	"usePlaceholders": true, // add parameter placeholders when completing a function
	// Experimental settings
	"completeUnimported": true, // autocomplete unimported packages
	"watchFileChanges": true, // watch file changes outside of the editor
	"deepCompletion": true, // enable deep completion
	},
	"files.eol": "\n", // formatting only supports LF line endings
	},
	// Uncomment the next line if you want to publish any ports.
	// "appPort": [],
	// Uncomment the next line to run commands after the container is created.
	// "postCreateCommand": "go version",
	// Add the IDs of extensions you want installed when the container is created in the array below.
	"extensions": [
	"ms-vscode.go"
	]
	}

view raw 1_devcontainer.json hosted with ❤ by GitHub

	FROM golang:1.13-stretch
	# Avoid warnings by switching to noninteractive
	ENV DEBIAN_FRONTEND=noninteractive
	# This Dockerfile adds a non-root 'vscode' user with sudo access. However, for Linux,
	# this user's GID/UID must match your local user UID/GID to avoid permission issues
	# with bind mounts. Update USER_UID / USER_GID if yours is not 1000. See
	# https://aka.ms/vscode-remote/containers/non-root-user for details.
	ARG USERNAME=vscode
	ARG USER_UID=1000
	ARG USER_GID=$USER_UID
	ENV GO111MODULE=on
	# Configure apt, install packages and tools
	RUN apt-get update \
	&& apt-get -y install –no-install-recommends apt-utils dialog rsync \
	# bazel stuff
	&& apt-get -y install openjdk-8-jdk pkg-config zip g++ zlib1g-dev unzip patch \
	#
	# Verify git, process tools, lsb-release (common in install instructions for CLIs) installed
	&& apt-get -y install git iproute2 procps lsb-release \
	#
	# Install Azure CLI
	&& curl -sL https://aka.ms/InstallAzureCLIDeb | bash \
	#
	# Clean up
	&& apt-get autoremove -y \
	&& apt-get clean -y \
	&& rm -rf /var/lib/apt/lists/*
	ENV GIT_PROMPT_START='\033[1;36mkube-devcon>\033[0m\033[0;33m\w\a\033[0m'
	# Save command line history
	RUN echo "export HISTFILE=/root/commandhistory/.bash_history" >> "/root/.bashrc" \
	&& echo "export PROMPT_COMMAND='history -a'" >> "/root/.bashrc" \
	&& mkdir -p /root/commandhistory \
	&& touch /root/commandhistory/.bash_history
	# Git command prompt
	RUN git clone https://github.com/magicmonty/bash-git-prompt.git ~/.bash-git-prompt –depth=1 \
	&& echo "if [ -f \"$HOME/.bash-git-prompt/gitprompt.sh\" ]; then GIT_PROMPT_ONLY_IN_REPO=1 && source $HOME/.bash-git-prompt/gitprompt.sh; fi" >> "/root/.bashrc"
	ENV DEVCONTAINER="TRUE"
	# Install docker used by go releaser
	RUN bash -c "cd /tmp && curl -fsSLO https://download.docker.com/linux/static/stable/x86_64/docker-19.03.5.tgz && tar –strip-components=1 -xvzf docker-19.03.5.tgz -C /usr/local/bin"
	# Install Go tools
	RUN \
	# –> Delve for debugging
	go get github.com/go-delve/delve/cmd/dlv@v1.3.2 \
	# –> Go language server
	&& go get golang.org/x/tools/gopls@v0.3.2 \
	# –> Go symbols and outline for go to symbol support and test support
	&& go get github.com/acroca/go-symbols@v0.1.1 && go get github.com/ramya-rao-a/go-outline@7182a932836a71948db4a81991a494751eccfe77 \
	&& rm -rf /go/src/ && rm -rf /go/pkg
	# Install K8s stuff we need
	RUN \
	# Build and install kubetest from test infra
	git clone https://github.com/kubernetes/test-infra.git && cd test-infra && GO111MODULE=on go install ./kubetest \
	# Install kind
	&& curl -Lo ./kind "https://github.com/kubernetes-sigs/kind/releases/download/v0.7.0/kind-linux-amd64" \
	&& chmod +x ./kind \
	&& mv ./kind /usr/local/bin/kind \
	# Install bazel
	&& wget -O bazelinstaller.sh https://github.com/bazelbuild/bazel/releases/download/2.1.0/bazel-2.1.0-installer-linux-x86_64.sh \
	&& chmod +x bazelinstaller.sh \
	&& ./bazelinstaller.sh \
	# Install required bazel version (may change)
	&& curl -LO https://releases.bazel.build/0.23.2/release/bazel-0.23.2-linux-x86_64 && chmod +x bazel-0.23.2-linux-x86_64 && cp bazel-0.23.2-linux-x86_64 /usr/local/lib/bazel/bin \
	# Install kubectl
	&& curl -sSL -o /usr/local/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/v1.16.2/bin/linux/amd64/kubectl \
	&& chmod +x /usr/local/bin/kubectl
	# Configure bash
	RUN echo "alias kindbuild='kind build node-image –type bazel -v 5 –image lawrencegripper/node:\$(git rev-list –max-count=1 HEAD) && kind create cluster –name lg –image lawrencegripper/node:\$(git rev-list –max-count=1 HEAD)'" >> ~/.bashrc
	RUN echo "source /etc/bash_completion" >> "/root/.bashrc"
	RUN echo "alias k=kubectl" >> "/root/.bashrc"
	RUN echo "source <(kubectl completion bash)" >> "/root/.bashrc"
	RUN echo "source <(kubectl completion bash | sed 's/kubectl/k/g')" >> "/root/.bashrc"

view raw 2_Dockerfile hosted with ❤ by GitHub