v2.23.0 Armory Release (OSS Spinnaker™ v1.23.2)

2020/11/20 Release Notes

Note: If you’re experiencing production issues after upgrading Spinnaker, rollback to a previous working version and please report issues to http://go.armory.io/support.

Required Halyard or Operator version

To install, upgrade, or configure Armory 2.23.0, use one of the following tools:

• Armory-extended Halyard 1.10.0 or later
• Armory Operator 1.2.1 or later

Security

Armory scans the codebase as we develop and release software. For information about CVE scans for this release, contact your Armory account representative.

Breaking changes

Breaking changes are kept in this list for 3 minor versions from when the change is introduced. For example, a breaking change introduced in 2.21.0 appears in the list up to and including the 2.24.x releases. It would not appear on 2.25.x release notes.

Suffix no longer added to jobs created by Kubernetes Run Job stage

Spinnaker no longer automatically appends a unique suffix to the name of jobs created by the Kubernetes Run Job stage. Prior to this release, if you specified metadata.name: my-job, Spinnaker updates the name to my-job-[random-string] before deploying the job to Kubernetes. As of this release, the job’s name will be passed through to Kubernetes exactly as supplied.

To continue having a random suffix added to the job name, set the metadata.generateName field instead of metadata.name, which causes the Kubernetes API to append a random suffix to the name.

This change is particularly important for users who are using the preconfigured job stage for Kubernetes or are sharing job stages among different pipelines. In these cases, jobs often running concurrently, and it is important that each job have a unique name. In order to retain the previous behavior, manually update your Kubernetes job manifests to use the generateName field.

Previously, this behavior was opt-in.

Impact

As of Armory 2.22, this behavior is the default. Users can still opt out of the new behavior by setting kubernetes.jobs.append-suffix: true in clouddriver-local.yml. This causes Spinnaker to continue to append a suffix to the name of jobs as in prior releases.

The ability to opt out of the new behavior will be removed in Armory 2.23 (OSS 1.23). The above setting will have no effect, and Spinnaker will no longer append a suffix to job names. We recommended that 2.22 users note which jobs are using the old behavior and prepare to remove the setting before upgrading to Armory 2.23 in the future.

Introduced in: Armory 2.22

ManifestForceCacheRefreshTask removed from Orca

When you upgrade to 2.23.0 or later, you might encounter the following error:

2021-01-29 23:57:19.691 ERROR 1 --- [    scheduler-2] c.netflix.spinnaker.q.redis.RedisQueue   : Failed to read message 8f072714f1df6dbf3af93a4f4fe4cae2, requeuing...
com.fasterxml.jackson.databind.JsonMappingException: No task found for 'com.netflix.spinnaker.orca.clouddriver.tasks.manifest.ManifestForceCacheRefreshTask' (through reference chain: com.netflix.spinnaker.orca.q.RunTask["taskType"])

The ManifestForceCacheRefreshTask task is no longer a required task when deploying a manifest. In earlier releases, forcing the cache to refresh was part of the deployment process for manifests. Because of this change, if a task was running or retried before the upgrade, the error shows up in logs as an exception.

Workaround

Before starting, make sure that you have access to the Redis instance that Orca uses.

To resolve this issue, delete the message from the queue:

1. Verify that there are pipeline execution failure messages that contain ManifestForceCacheRefreshTask:

   Redis

   hgetall orca.task.queue.messages
   

   The command returns information similar to the following:

   1) "93ac65e03399a4cfd3678e1355936ab2"
   2) "{\"kind\":\"runTask\",\"executionType\":\"PIPELINE\",\"executionId\":\"01EVFCCDG3Q2209E0Z1QTNC0FS\",   \"application\":\"armoryhellodeploy\",\"stageId\":\"01EVFCCDG3TJ7AFPYEJT1N8RDJ\",\"taskId\":\"5\",\"taskType\":\"com.netflix.spinnaker.   orca.clouddriver.tasks.manifest.ManifestForceCacheRefreshTask\",\"attributes\":[{\"kind\":\"attempts\",\"attempts\":1}],   \"ackTimeoutMs\":600000}"
   
   

2. Delete the message(s):

   Redis

   hdel orca.task.queue.messages 
   

   The command returns information similar to the following:

   93ac65e03399a4cfd3678e1355936ab2
   (integer) 1
   

Known issues

Clouddriver resources

There is a known issue with Clouddriver that affects the performance of Armory Enterprise, causing it to consume more resources. This can lead to a situation where pods do not have enough resources to start.

Zombie Executions

Starting in Spinnaker 2.23.0, ManifestForceCacheRefreshTask was removed, as Kubernetes manifest related stages now do live lookups. While upgrading to Spinnaker 2.23.0 or later, if there is a running pipeline that contains a Kubernetes manifest related stage, it becomes a zombie execution. This causes Orca, Spinnaker’s orchestration service, to fail to complete any Kubernetes manifest related stage in that pipeline.

Workarounds:

To resolve the issue, cancel any zombie executions. For information about how to cancel them, see the Orca Zombie Execution runbook.

Affected versions: 2.23.0 and later

Lambda UI issue

There is a UI bug related to the caching agent that prevents Lambda functions from being displayed in the UI when there are no other clusters associated with the Application. In other words, in order for the function to show up in “Functions” tab, there needs to be a cluster (such as an AWS ASG/EC2 instance) deployed for that application.

Affected versions: 2.23.0 (1.23.0) and later

Fixed issues

All the known issues from the previous release (v2.22.2) have been fixed:

• Orca Plugins using Plugin SDK has been addressed in the Plugin V2 framework
• Spinnaker liveManifestCalls set to true can cause major pipeline errors
• GCE predictive autoscaling exception

Highlighted updates

Artifacts

Helm repo indexes are now supported for artifacts

You can provide helm/index as a type when specifying an artifact path.

Versions now supported in artifacts using Oracle Objects

In the artifact selection UI, you can use # after the artifact to indicate the version you want to use. For example:

Baking

The bundled Packer version has been upgraded to 1.6.4.

Configuration

Dynamic Accounts

Spinnaker reads account definitions (cloud providers, CI, metric stores, etc.) from Spring properties. This works well in a world with a handful of accounts that rarely change but causes operational pain when provisioning accounts dynamically or with account information stored externally. See the Spinnaker Account Management proposal for details.

The following providers can support loading credentials dynamically:

• CloudFoundry
• Kubernetes
• AWS

Canary analysis with Dynatrace

See the Use Canary Analysis with Dynatrace in Spinnaker guide for how to configure and use this new feature.

Deployment targets - AWS

Support externalID for granting access to your AWS resources to a third party

Spinnaker can assume a role into third party resources (delegated access) that require AWS external IDs. To configure an AWS provider, add the externalId property when specifying the accountId and assumeRole properties. For example:

aws:
  name: delegated-prod
  accountId: 1234
  assumeRole: role/spinnaker
  externalId: "unique ID assigned to Armory"

Enabling AWS Lambda in configurations has changed

The lambda.enabled key has now moved under features.

Old:

aws:
  lambda:
    enabled: true
  accounts:
    - name: test
      lambdaEnabled: true

New:

aws:
  features:
    lambda:
      enabled: true
  accounts:
    - name: test
      lambdaEnabled: true

Fixes

There have been several fixes for Launch Templates and for the Lambda provider. See the open source changelog.

Deployment targets - Google

Google AppEngine add deploy global configuration stage

This new stage was created in order to support various configuration settings for an app engine application. You can find these settings in Google Cloud’s Configuration Files content.

For example, appengine supports cron configuration. You can update or deploy cron configuration similar to how you deploy services. This stage replicates the functionality offered and allows you to deploy cron, dispatch, index, and queue configuration files to your appengine environments.

Load Balancers

Support for AWS cross zone load balancing for Network Load Balancers

Spinnaker now supports the AWS cross zone load balancing setting. You’ll see a new checkbox in the UI when configuring NLBs. This is turned on by default for new load NLBs.

Plugins

Plugin V2 Framework

The v2 plugin framework is now in place! The V2 plugin framework simplifies configuration of plugins and includes many quality of life changes for plugin developers interested in making spring based plugins. See V2 Compatibility for details.

Storage

Support for Redis SSL connection paths

To turn on SSL, add an s to redis:// in the connection string:

redis:
  connection: rediss://localhost:6379

User Interface

Support for displaying colors in console outputs

Pipelines can now be triggered by Helm Charts when they are published

Clusters page

If multiple containers exist for a server group, pods get collapsed by default.

Spinnaker Community Contributions

There have also been numerous enhancements, fixes, and features across all of Spinnaker’s other services. See the
Spinnaker v1.23.2 changelog for details.

Detailed updates

Bill Of Materials (BOM)

Here’s the BOM for this version.

version: 2.23.0
timestamp: "2020-11-20 15:20:46"
services:
    clouddriver:
        commit: a941921d
        version: 2.23.24
    deck:
        commit: f3a47075
        version: 2.23.8
    dinghy:
        commit: 2437ecbb
        version: 2.23.4
    echo:
        commit: 17b5072e
        version: 2.23.3
    fiat:
        commit: 9bdf56a1
        version: 2.23.2
    front50:
        commit: 71810ee6
        version: 2.23.3
    gate:
        commit: ccd981fb
        version: 2.23.2
    igor:
        commit: 3932c3ec
        version: 2.23.1
    kayenta:
        commit: d4ae95aa
        version: 2.23.2
    monitoring-daemon:
        version: 2.23.0
    monitoring-third-party:
        version: 2.23.0
    orca:
        commit: 3d0b945c
        version: 2.23.9
    rosco:
        commit: 76ab237d
        version: 2.23.4
    terraformer:
        commit: 3287d6fc
        version: 2.23.1
dependencies:
    redis:
        version: 2:2.8.4-2
artifactSources:
    dockerRegistry: docker.io/armory

Armory

Armory Orca - 2.22.2…2.23.9

• feat(plugins): include plugin downloader and plugin manifest (#144)
• chore(aquasec): updating to show scan results in comments (#152)
• feat(manifests): adding manifests for staging (#153)
• fix(manifest): Updating deployment manifest (#154)
• feat(chore): add rules for auto armory-common autobumps (#156)
• fix(chore): typo for regex autobump (#157)
• chore(spinnakerRelease): updated to support 2.23.x
• use master-20200921140017
• feat(ubi): add build for ubi image. (#159)
• feat(ubi): update ubi LICENSES. (#161)
• fix(ubi): deploy ubi to jfrog. (#162)
• fix(GHA): set-env deprecated (bp #170) (#171)
• Revert “fix(spinnakerBump): AutoBump Spinnaker Versions/ArmoryCommons Versions (bp #169)” (#173)

Armory Clouddriver - 2.22.11…2.23.24

• feat(chore): add rules for auto armory-common autobumps (#197)
• fix(chore): typo for regex autobump (#198)
• feat(cve): Aquasec action version bump (#201)
• feat(ubi): add build for ubi image. (#202)
• fix(mergify): automerges when ‘ready to merge’ label is added
• fix(docker): downgrade aws-iam-authenticator (#205)
• fix(docker): python downgrade (#212)
• chore(build): resolve CVEs and reduce docker layers (#213)
• fix(ubi): deploy ubi to jfrog. (#214)
• chore(spinnakerRelease): updated to support 2.23.x (#204)
• use master-20201029170017
• empty commit to bump versions
• empty commit to bump versions
• empty commit to bump versions
• update using master-20201109170017
• fix(GHA): set-env deprecated (bp #224) (#225)

Armory Fiat - 2.22.3…2.23.2

• feat(plugins): include plugin downloader and plugin manifest (#98)
• chore(aquasec): updating to show scan results in comments (#104)
• feat(chore): add rules for auto armory-common autobumps (#106)
• fix(chore): typo for regex autobump (#107)
• feat(ubi): add build for ubi image. (#109)
• feat(ubi): update ubi LICENSES. (#110)
• fix(ubi): deploy ubi to jfrog. (#111)
• fix(build): force 2.23.x release (#121)
• chore(build): trying to get 2.23 out the door (#123)
• chore(build): release-2.23.x has v2.23.0 tag on it (#124)

Armory Front50 - 2.22.2…2.23.3

• feat(plugins): include plugin downloader and plugin manifest (#142)
• chore(build): upgrade scan action (#148)
• feat(chore): add rules for auto armory-common autobumps (#150)
• feat(ubi): add build for ubi image. (#154)
• fix(ubi): deploy ubi to jfrog. (#155)
• fix(GHA): set-env deprecated (bp #165) (#166)

Armory Igor - 2.22.3…2.23.1

• feat(plugins): include plugin downloader and plugin manifest (#119)
• chore(build): upgrade scan action (#125)
• feat(chore): add rules for auto armory-common autobumps (#127)
• fix(chore): typo for regex autobump (#128)
• fix(ubi): deploy ubi to jfrog. (#130)
• fix(ubi): fix release version. (#132)
• fix(GHA): set-env deprecated (bp #141) (#142)
• fix(build): force build (#144)

Armory Rosco - 2.22.5…2.23.4

• feat(plugins): include plugin downloader and plugin manifest (#95)
• chore(build): upgrade scan action (#101)
• chore(integration): add helm & kustomize tests (#102)
• feat(chore): add rules for auto armory-common autobumps (#104)
• fix(chore): typo for regex autobump (#105)
• feat(ubi): add build for ubi image. (#107)
• feat(kustomize): update version of kustomize used (#108)
• chore(licenses): update ubi LICENSES. (#111)
• fix(kustomize): Update kustomize with latest bug fixes (#113)
• chore(e2e): added basic aws ami bake e2e test (#117)
• feature(jobs): k8s job executor (#121)
• feat(tests): adding integration test that checks packer hcl template support (#124)
• feat(test): adding int test to bake with SSM on a private subnet (#125)
• chore(docs): adding walkthrough to run packer with ssm on rosco (#126)
• fix(GHA): set-env deprecated (#129)
• feat(fargate-job-executor): add fargate job executor (#127)
• feat(fargate-job-executor): merge master to include the Fargate Job Executor in the 2.23.x releases (#134)
• fix(build): force build with tag (#139)
• fix(build): force build with tag (#139) (#140)

Armory Gate - 2.22.3…2.23.2

• fix(dependencies): remove * + and x for reproducible builds (#170)
• feat(chore): add rules for auto armory-common autobumps (#172)
• fix(chore): typo for regex autobump (#173)
• feat(ubi): add build for ubi image. (#176)
• fix(mergify): automerges when ‘ready to merge’ label is added
• feat(plugins): deck proxy integration test (#137)
• fix(ubi): deploy ubi to jfrog. (#177)
• chore(release): merge master into 2.23 branch (#188)

Armory Deck - 2.22.7…2.23.8

• feat(mptv2): add feature flag for mptv2 && upgrade to oss 1.22.x (#649)
• feat(build): add details on how to build from a OSS forked release branch (#652)
• feat(chore): add rules for auto armory-common autobumps (#657)
• chore(mergify): fix autobump for armory-commons (#658)
• chore(spinnakerRelease): updated to support 2.23.x
• feat(settings.js): add a comment on why to keep MPTV2 feature flag
• feat(settings.js): add a comment on why to keep MPTV2 feature flag (#664)
• fix(package.json): added missing package
• fix(deps): remove unused dependency with AGPL license (#665)
• feat(ubi): add build for ubi image. (#662)
• fix(ubi): deploy ubi to jfrog. (#666)
• feat(build): use 1.23.0-master-20201012170017-rc1
• fix(typo): an egregious typo in the Terraform help text (#669)
• chore(spinnakerRelease): updated to support 2.23.x using master-20200918230017 (#663)
• feat(kayenta): extending kayenta module to add dynatrace UI (#675) (#678)
• chore(release): merge master into 2.23 ahead of release (#683)
• fix(dinghy): fix NPE for commits and files. (#684) (#685)

Dinghy™ - 2.22.0…2.23.4

• feat(chore): add rules for auto armory-common autobumps (#273)
• fix(chore): typo for regex autobump (#274)
• feat(ubi): add build for ubi image. (#276)
• fix(ubi): deploy ubi to jfrog. (#277)
• feat(permissions): support update application permissions (#280)
• feat(logevents): logevents changes for deck ui integration (#284)
• fix(nilcommits): fix nil commits for deck ui (#286) (#287)
• fix(nofiles): only show file changes in logevents (#289) (#290)
• feat(mainbranch): main branch support switch when master fails (#292) (#293)

Armory Echo - 2.22.2…2.23.3

• feat(plugins): include plugin downloader and pluginManifest (#219)
• fix(plugins): Don’t fail when debug service is not provided because diagnostics.enabled is false (#224)
• chore(aquasec): updating to show scan results in comments (#229)
• feat(chore): add rules for auto armory-common autobumps (#231)
• fix(chore): typo for regex autobump (#232)
• feat(ubi): add build for ubi image. (#234)
• feat(ubi): update ubi LICENSES. (#235)
• fix(ubi): deploy ubi to jfrog. (#236)
• fix(GHA): set-env deprecated (bp #244) (#245)
• Revert “fix(spinnakerBump): AutoBump Spinnaker Versions/ArmoryCommons Versions (bp #243)” (#247)
• Mergify/bp/release 2.23.x/pr 243 (#248)

Terraformer™ - 2.22.2…2.23.1

• chore(build): upgrade scan action (#248)
• feat(tests): apply and destroy commands (#249)
• fix(test): fix input for destroy command test (#252)
• feat(integration): add test for git ssh modules with named profiles (#254)
• feat(chore): add rules for auto armory-common autobumps (#256)
• fix(chore): typo for regex autobump (#257)
• fix(plan): add targets to plan command (#262)
• chore(versions): terraform 0.13.1,0.13.2,0.13.3 (#261)
• fix(tests): compare array elements instead of string (#263)
• fix(targets): add targets to destroy command (#265)
• feat(ubi): add build for ubi image. (#260)
• fix(ubi): deploy ubi to jfrog. (#266)
• chore(cve): fix CVE-2020-13757 (#272)
• fix(GHA): set-env deprecated (#276)
• fix(build): force build (#279)

Armory Kayenta - 2.22.3…2.23.2

• feat(plugins): include plugin downloader and plugin manifest (#116)
• feat(chore): add rules for auto armory-common autobumps (#123)
• fix(chore): typo for regex autobump (#124)
• fix(cve): CVE fixes CVE-2020-9484 and CVE-2020-7692 (#128)
• Dynatrace(feat): implementation for dynatrace kayenta (#127)
• feat(ubi): add build for ubi image. (#126)
• fix(ubi): deploy ubi to jfrog. (#132)
• feat(aquasec): upgrade aquasec image, and adding feat to publish comments about aquasec analysis (#134)
• fix(build): Build/release 2.23.x/oss 1.23.2 (#146)