Add a Kubernetes Account Deployment Target in Spinnaker

How Spinnaker operates when deploying to Kubernetes targets

At a high level, Spinnaker operates in the following way when deploying to Kubernetes:

• Spinnaker is configured with one or more “Cloud Provider” Kubernetes accounts (which you can think of as deployment targets)
• For each Kubernetes account, Spinnaker is provided a kubeconfig to connect to that Kubernetes cluster
• The kubeconfig should have the following contents:
  • A Kubernetes kubeconfig cluster, which is typically a URL and potentially certificate validation information (CA cert or a flag to skip validation)
  • A Kubernetes kubeconfig user, which is basically a way to get a set of credentials to connect to the Kubernetes cluster. This can be a token, a set of commands to run to get a token, or a client certificate
  • A Kubernetes kubeconfig context, which tells the kubectl tool which credential set to use with which cluster, as well as other information such as default namespace
  • Metadata such as which context to use by default
• Each Kubernetes account is configured in the SpinnakerService manifest under spec.spinnakerConfig.config.providers.kubernetes.accounts key if using the Operator, or under the master .hal/config YAML file (also known as a “Halconfig”), as an entry in the deploymentConfiguration’s providers.kubernetes.accounts array if using Halyard. Each entity has these (and other) fields:
  • name: A Spinnaker-internal name used to refer to the Kubernetes account. This is the item you will select in various Kubernetes stages to indicate that you would like to deploy to this particular Kubernetes account (which again, consists of a Kubernetes cluster/credential/context set)
  • kubeconfigFile: A file path referencing the contents of the kubeconfig file for connecting to the target cluster. When using the Operator, this can be any name that should match an entry in spec.spinnakerConfig.files where the file contents is copied. If using Halyard, this is a physical file path. This field supports referencing files stored in external secret engines
  • onlySpinnakerManaged: When true, Spinnaker only caches and displays applications that have been created by Spinnaker. Before placing in a false state, you should review the Kubernetes cluster configuration. When false, Spinnaker analyzes the cluster and automatically attempts to configure and populate applications for resources already present in Kubernetes, unless limited with omitNamespaces. You should note the increased possibilities of misconfigured Autogenerated Application Placeholders in the deployments.
  • namespaces: An array of namespaces that Spinnaker will be allowed to deploy to. If this is left blank, Spinnaker will be allowed to deploy to all namespaces
  • omitNamespaces: If namespaces is left blank, you can blacklist specific namespaces to indicate to Spinnaker that it should not deploy to those namespaces
• Spinnaker uses kubectl under the hood, so any semantics that kubectl specifies, Spinnaker will also use
• kubectl (and the corresponding kubeconfig) will be run from the Spinnaker Clouddriver Kubernetes pod
• If the kubeconfig is properly referenced and available, Operator or Halyard will take care of the following for you:
  • Creating a Kubernetes secret (with a dynamically-generated name) containing your kubeconfig in the namespace where Spinnaker lives
  • Dynamically generating a clouddriver.yml file (if using Halyard, this is placed locally in .hal/default/staging/clouddriver.yml) that properly references the kubeconfig from where it is mounted within the Clouddriver container
  • Creating/Updating the Kubernetes Deployment (spin-clouddriver) which runs Clouddriver so that it is aware of the secret and properly mounts it in the Clouddriver pod (if using Halyard, it will be placed in /home/spinnaker/.hal/default/staging/dependencies/<some-filename>)
  • Creating a secret containing clouddriver.yml (and optionally, clouddriver-local.yml) and including it in the spin-clouddriver deployment

So here are some takeaways and guiding principles that result from the above:

• Spinnaker relies on the kubeconfig to authenticate against and access your Kubernetes cluster.
• You can use a command-based or auth-provider-based kubeconfig user (such as aws-iam-authenticator), but this will rely on the command binary and all relevant files existing in the Clouddriver, and all authentication (such as IAM roles) being properly attached to the Clouddriver container.
• Alternately (and perhaps, preferably), you can do the following:
  • Create a Kubernetes service account in your Kubernetes cluster. This service account will exist in some specific namespace.
  • Grant the service account permissions on your cluster. This can be one or more of the following:
    • cluster-admin access, to be able to access all namespaces in your cluster
    • Full wildcard (apiGroup * / resource * / verb *) access to one or more namespaces in your cluster
    • More specific permissions in your cluster (this is not covered in the scope of this document)
• It is preferable to create a distinct kubeconfig file for each Spinnaker Cloud Provider Kubernetes account; each of these kubeconfigs should have one cluster, one user, and one context referencing the cluster and user. Its default context should reference the single context

Prerequisites for adding a Kubernetes Service Account

This document assumes the following:

• Your Spinnaker is up and running
• Your Spinnaker was installed and configured via the Operator or Halyard
• If Spinnaker was installed with the Operator, you have access to the SpinnakerService manifest and have the kubeconfig for the Spinnaker cluster.
• If Spinnaker was installed with Halyard, you have access to your current halconfig, and a way to operate hal against it.
• You have a valid kubeconfig that currently has Cluster Admin access to your target Kubernetes cluster so you can create the relevant Kubernetes entities (service account, role, and rolebinding) in the target cluster; you have kubectl and are able to use it to interact with your target Kubernetes cluster.

The first several steps of this document will take place on a system that has kubectl and the kubeconfig.

Workflow for adding a Kubernetes Service Account

• Create the namespace in which the Service Account will live (if it does not yet exist)
• Create the service account in the Service Account namespace
• If granting cluster admin (cluster-admin), a clusterrolebinding attaching the cluster-admin ClusterRole to the service account
• If granting namespace-specific access, the following:
  • For each namespace, the namespace (if it does not exist)
  • For each target namespace, an admin role
  • For each target namespace, a rolebinding attaching the admin role to the service account
• Create a minified kubeconfig containing only the service account
• Add the account to Spinnaker

Setting up the Kubernetes Service Account

In this section, you create the following:

• A namespace to create the service account in
• A service account in that namespace
• A role and rolebinding in that namespace, granting permissions to the service account
• A kubeconfig containing credentials for the service account

This document uses the newly-created Armory spinnaker-tools Go CLI (available on Github) to create many of these resources. There are separate instructions to perform these steps manually.

Download the latest spinnaker-tools release

Go to https://github.com/armory/spinnaker-tools/releases to download the latest release for your operating system (OSX and Linux available). You can also use curl:

# If you're not already in the directory
cd ~/eks-spinnaker
# Replace this with the correct version and link for your workstation (use https://github.com/armory/spinnaker-tools/releases/download/latest/spinnaker-tools-linux if you're on Linux instead of Mac)
curl -L https://github.com/armory/spinnaker-tools/releases/download/latest/spinnaker-tools-darwin -o spinnaker-tools
chmod +x spinnaker-tools

Set up bash parameters

Set up bash environment variables that you will use later.

# If you're using a different source kubeconfig, specify it here
export SOURCE_KUBECONFIG=${HOME}/.kube/config

# Specify the name of the kubernetes context that has permissions in your target cluster.
# To get context names, you can run "kubectl config get-contexts".
export CONTEXT="aws-armory-dev"

# Enter the namespace where you want the Spinnaker service account to live
export SPINNAKER_NAMESPACE="spinnaker-system"

# Enter the name of the service account you want to create in the target namespace.
# If you are creating multiple Kubernetes Cloud Provider Accounts to point to
# different namespaces in to the same Kubernetes cluster, make sure you use a
# unique service account name.
export SPINNAKER_SERVICE_ACCOUNT_NAME="spinnaker-dev-sa"

# Specify where you want the new kubeconfig to be created
export DEST_KUBECONFIG=${PWD}/kubeconfig-dev-sa

# If you want to deploy to a specific set of namespaces, enter the namespaces
# that you want to deploy to (space-delimited).
# These namespaces can already exist, or you can create them.
export TARGET_NAMESPACES_COMMA_SEPARATED=dev-1,dev-2

Option 1: Create the service account with cluster-admin permissions

./spinnaker-tools create-service-account \
  --kubeconfig ${SOURCE_KUBECONFIG} \
  --context ${CONTEXT} \
  --output ${DEST_KUBECONFIG} \
  --namespace ${SPINNAKER_NAMESPACE} \
  --service-account-name ${SPINNAKER_SERVICE_ACCOUNT_NAME}

Option 2: Create the service account with namespace-specific permissions

./spinnaker-tools create-service-account \
  --kubeconfig ${SOURCE_KUBECONFIG} \
  --context ${CONTEXT} \
  --output ${DEST_KUBECONFIG} \
  --namespace ${SPINNAKER_NAMESPACE} \
  --service-account-name ${SPINNAKER_SERVICE_ACCOUNT_NAME} \
  --target-namespaces ${TARGET_NAMESPACES_COMMA_SEPARATED}

Add the kubeconfig and cloud provider to Spinnaker

• Operator
• Halyard

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
  name: spinnaker
spec:
  spinnakerConfig:
    config:
      providers:
        kubernetes:
          enabled: true
          accounts:
          - name: spinnaker-dev # Account name you want Spinnaker to use to identify the deployment target
            requiredGroupMembership: []
            providerVersion: V2
            permissions: {}
            dockerRegistries: []
            configureImagePullSecrets: true
            cacheThreads: 1
            namespaces: [] # Change if you only want to deploy to specific namespaces
            omitNamespaces: []
            kinds: []
            omitKinds: []
            customResources: []
            cachingPolicies: []
            oAuthScopes: []
            onlySpinnakerManaged: true
            kubeconfigFile: kubeconfig-spinnaker-dev
          primaryAccount: spinnaker-dev  # Change to a desired account from the accounts array
      features:
        artifacts: true # Not strictly necessary for Kubernetes but will be useful in general
    files:
      kubeconfig-spinnaker-dev: |
        <FILE CONTENTS HERE>

kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest file>

# Replace with the name of your kubeconfig file
export KUBECONFIG_FILE=KUBECONFIG_FILE_NAME # This must be updated

# Enter the account name you want Spinnaker to use to identify the deployment target (should be the same as above)
export ACCOUNT_NAME="spinnaker-dev"

# If you only want to deploy to specific namespaces (should be the same as above)
export TARGET_NAMESPACES=(dev-1 dev-2)

# Feel free to reference a different location
KUBECONFIG_DIRECTORY=~/.secret/
cp ${KUBECONFIG_FILE} ${KUBECONFIG_DIRECTORY}
export KUBECONFIG_FULL=$(realpath ${KUBECONFIG_DIRECTORY}${KUBECONFIG_FILE})

# Enable the kubernetes provider - this is probably already enabled, if Spinnaker is installed in Kubernetes
hal config provider kubernetes enable
# Enable artifacts; not strictly necessary for Kubernetes but will be useful in general
hal config features edit --artifacts true

# Add account
hal config provider kubernetes account add ${ACCOUNT_NAME} \
  --provider-version v2 \
  --kubeconfig-file ${KUBECONFIG_FULL}

# Loop and add namespaces
for TARGET_NS in ${TARGET_NAMESPACES[@]}; do
  hal config provider kubernetes account edit ${ACCOUNT_NAME} \
    --add-namespace ${TARGET_NS}
done

# Apply changes
hal deploy apply

Verify the Kubernetes account appears in the Spinnaker UI

After you apply your changes, you should see the new Kubernetes account in your Spinnaker UI and be able to deploy to it. It may take a while for the Clouddriver to start up and read information from your new Kubernetes account.

Don’t forget to clear your browser cache / hard refresh your browser (cmd-shift-r or control-shift-r)