All Versions
Latest Version
Avg Release Cycle
28 days
Latest Release
1324 days ago

Changelog History
Page 1

  • v6.7.2 Changes

    November 30, 2020

    ๐Ÿš€ This release focuses on optimizations and bug fixes.

    ๐Ÿฑ โœˆ๏ธ Features

    ๐Ÿฑ Optionally skip resource checking for put-only resources. (#6271) @evanchaoli ๐Ÿ”—
    Resource checking for put resources can be optionally turned off by using the feature flag CONCOURSE_ENABLE_SKIP_CHECKING_NOT_IN_USE_RESOURCES on the web command. Only resources which are used as inputs to a job will be checked as a result. This can lower the resource checking load on some instances. This feature was originally added in 6.0.0 and reverted in 6.6.0 because of two side effects: 1) put-only resources will no longer show version history in the UI, and 2) custom resource types of put-only resources will no longer be automatically checked. Please note that these side effects still exist at the moment.

    โž• Add index to improve pipeline and build deletion (#6273) @vito ๐Ÿ”—
    ๐Ÿ›  Pipeline deletion had become slow and even caused some deadlocks in the database. This migration fixes those issues.

    ๐Ÿฑ Ensure pipelines contain at least one job (#6310) @taylorsilva ๐Ÿ”—
    ๐Ÿ”ง Pipelines are now validated to ensure that they contain at least one job. Pipeline configurations with no jobs will be rejected.

    ๐Ÿฑ ๐Ÿž Bug Fixes

    โœ‚ Remove any existing guardian assets on worker startup (#6257) @taylorsilva ๐Ÿ”—
    ๐Ÿš€ The guardian runtime executable contains some other executables as dependencies, such as runc. Interestingly, it writes them to disk (/var/gdn/assets) at startup, and proceeds to run them from there. If the dependencies already exist from a previous version of guardian then they're not overwritten preventing the updated dependencies from being used. The worker will now clear out any preexisting guardian assets every time it starts. This fixes in-place upgrade scenarios on BOSH deployments where guardian kept using older versions of runc.

    ๐Ÿ— Abort a rerun build if input version is gone (#6265) @xtremerui ๐Ÿ”—
    ๐Ÿ— Sometimes a rerun of a build can get stuck in the pending state forever. This can happen if the resource version used for the original build no longer exists. For example, an older commit disappears when a user force pushes to a git repository. Builds will now abort instead of getting stuck when the resource version no longer exists.

    ๐Ÿ›  Fix regression preventing the use of nested fields with fly set-pipeline --var (#6280) @aoldershaw ๐Ÿ”—

    • A regression was introduced in 6.5.0 that prevented the use of nested fields when setting a pipeline variable via fly set-pipeline --var
      • For instance, pre 6.5.0, fly set-pipeline --var --var foo.baz=456 would create the variable foo with value {bar: 123, baz: 456} (that can be referenced in the pipeline config as ((, ((foo.baz)))
      • In 6.5.x/6.6.x, the same command would create variables "" = 123 and "foo.baz" = 456 (that would have to be referenced in the pipeline config as (("")) and (("foo.baz")), respectively)

      - If you want to set a variable with a . in it, you can now quote the flag: e.g. fly set-pipeline --var '""=123' (note that this requires quoting the entire flag in '...' to avoid shell expansion)

    ๐Ÿฑ Only increment checks-enqueued when a check is created. (#6282) @evanchaoli ๐Ÿ”—
    There was a metrics bug where checks-enqueued would get incremented even if a check already exists. This gave it a higher value than checks that have just been created.

    ๐Ÿฑ Prevent set_pipeline runtime error (#6307) @evanchaoli ๐Ÿ”—
    ๐Ÿ”ง set_pipeline of a YML pipeline configuration file with no jobs: or resources: no longer causes a runtime error: invalid memory address or nil pointer dereference. This is a backport #6116.

    ๐Ÿฑ ๐Ÿคท Miscellaneous

    • โšก๏ธ Optimize fly tests (#6299) @muntac ๐Ÿ”—
  • v6.7.1 Changes

    November 02, 2020

    ๐Ÿš€ This release bumps the docker-image resource type to v1.5.1 which contains a fix for the HEAD => GET request flow.

    NOTE: the docker-image resource type does not support configuring credentials in registry_mirror. This issue is somewhat fundamental, as the the docker CLI which the resource type uses doesn't support it either. If your registry mirror requires credentials, we recommend using the registry-image resource type instead - it's much more efficient, and it will replace the docker-image resource type entirely once it's ready.

  • v6.7.0 Changes

    October 29, 2020

    ๐Ÿš€ The primary focus of this release is to adapt to Docker Hub's upcoming rate limits for free users, which will take effect November 2.

    ๐Ÿš€ We recommend that anyone who uses images hosted on Docker Hub - either without authentication or with a free account - to upgrade to this release, which includes the following:

    • ๐Ÿš€ registry-image resource type v0.14.0 and docker-image resource type v1.5.0
      • Both resource types now use a HEAD request rather than GET during the check operation so that they don't count towards the rate limit. This was shipped in v6.6.0, but registries which do not support HEAD broke - now they'll fall back on GET.
    • ๐Ÿ”ง Ability to configure defaults for resource types (#6079).
      • The check fix is not enough in isolation - a large enough Concourse cluster might still run into rate limits when it comes time to fetch the image.
      • With resource type defaults, a cluster-wide pull-through cache may be configured by setting a default registry_mirror config for the registry-image resource type and the docker-image resource type. (Note that their configuration is slightly different.)

    More details follow!

    ๐Ÿฑ โœˆ๏ธ Features

    • ๐Ÿ‘ Allow configuring source "defaults" for resource types (#6079) @evanchaoli ๐Ÿ”—

    ๐Ÿ”ง With this feature, a cluster admin may configure default source configuration for core resource types on the web node. For example, when the following file is specified to ATC --base-resource-type-defaults=brt_default.yml:

    registry-image: registry\_mirror: host:

    then all registry-image will pull images from the mirror

    0๏ธโƒฃ At pipeline level, defaults can be defined with resource types, for example:

    resource\_types: - name: rt-gitlabtype: registry-imagesource: repository: \<uri of a gitlab resource type\>defaults: url: \<gitlab url\>project: \<project-id\>resources: - name: gitlab-mrtype: rt-gitlabsource: kind: MergeRequest- name: gitlab-releasetype: rt-gitlabsource: kind: Release

    0๏ธโƒฃ In this sample, when defines the gitlab resource-type, GitLab url and project-id are defined as defaults of the resource type, so that all resources of the type will inherit those defaults.

    ๐Ÿฑ ๐Ÿž Bug Fixes

    • ๐Ÿฑ containerd does not grant access to /dev/console (#6181) @jamieklassen ๐Ÿ”—
      • If you are running concourse with the containerd backend inside a runc-managed process and your version of runc is v1.0.0-rc91 or above, creating privileged containers no longer fails with EPERM.
  • v6.6.0 Changes

    October 02, 2020

    ๐Ÿฑ ๐Ÿšจ Breaking

    • ๐Ÿš€ The concourse worker process now depends, via gdn v1.19.16, on runc v1.0.0-rc91. There is a known issue with using BPM to manage newer versions of runc, which has been resolved in bpm-release v1.1.9. So if your concourse is deployed using BOSH, you will also need to use at least BPM 1.1.9. After upgrading, if you start to see build steps erroring with a message like:

      runc run: exit status 1: container_linux.go:349: starting container process caused "process_linux.go:439: container init caused \"process_linux.go:405: setting cgroup config for procHooks process caused \\"failed to write \\\\"c 5:1 rwm\\\\" to \\\\"/sys/fs/cgroup/devices/system.slice/concourse.service/garden/a206550f-f6dd-4609-4f13-0a11afd3fd93/devices.allow\\\\": write /sys/fs/cgroup/devices/system.slice/concourse.service/garden/a206550f-f6dd-4609-4f13-0a11afd3fd93/devices.allow: operation not permitted\\"\""

    โฌ†๏ธ then you probably need to upgrade BPM. Thanks @tlwr for investigating this!

    ๐Ÿฑ โœˆ๏ธ Features

    ๐Ÿ‘ Allow rotating the encryption key via concourse migrate (#5980) @aoldershaw ๐Ÿ”—

    • concourse migrate can be called with --old-encryption-key to rotate the database encryption key as a one-time operation
    • concourse web still accepts --old-encryption-key

    - You should stop any ATCs prior to running this command

    ๐Ÿ‘Œ Support SAML 2.0 as an auth backend (#5998) @adnankobir ๐Ÿ”—

    ๐Ÿ”ง Use the configured resolv.conf for containerd (#6001) @YoussB ๐Ÿ”—

    ๐Ÿฑ set_pipeline step header indicates whether changes were applied (#6011) @andy-paine ๐Ÿ”—

    - If a set_pipeline step made any changes, it will indicate by highlighting in yellow and displaying pipeline config changed message when hovered over the step.

    ๐Ÿ‘ Allow the fly http transport to use client certificates (#6017) @avanier ๐Ÿ”—

    - Adds new --client-cert and --client-key flags to the fly login command. The provided client certificates will then be used by fly's http transport.

    โž• Add new default values to the Guardian flags network-pool and max-containers (#6031) @muntac ๐Ÿ”—

    ๐Ÿ‘ Allow statx in containerd (#6045) @muntac ๐Ÿ”—

    - Update containerd's seccomp profile to allow the statx system call. This lets basic commands like ls -l to be executed.

    ๐Ÿฑ Enable secret caching for var_sources. (#6046) @evanchaoli ๐Ÿ”—

    ๐Ÿฑ Custom background image (#6053) @andy-paine ๐Ÿ”—

    ๐Ÿฑ ๐Ÿž Bug Fixes

    ๐Ÿฑ Return real JSON from fly get-team --json (#5996) @agurney ๐Ÿ”—

    - The --json flag on fly get-team is meant to yield a structured JSON representation of the team data, including auth details, but instead just gave {}. Now it works as intended.

    ๐Ÿฑ Admins can only login to teams that exist (#6021) @izabelacg ๐Ÿ”—

    - Previously admins could log into any team, even if the team did not exist. Admins can still log into any team but now fly verifies that the team exists before saving the target to .flyrc.

    โฑ Request schedule when cancelling a pending build (#6023) @xtremerui ๐Ÿ”—

    - Fix a bug that a build can't be cancelled if it is in pending state because of unsatisfiable inputs

    ๐Ÿฑ Enable fly validate-pipeline to accept --enable-across-step (#6027) @flavorjones ๐Ÿ”—

    - Enable fly validate-pipeline to accept --enable-across-step and recognize across as a valid step.

    ๐Ÿ›  Fix build reaper missing builds due to pagination (#6051) @chenbh ๐Ÿ”—

    • Fixed the build reaper missing builds

    - Fixed pagination bugs on the resource version and job builds pages

    ๐Ÿ›  Fix a bug where running fly execute fails in an environment when all workers are tagged. (#6059) @aledeganopix4d ๐Ÿ”—

    ๐Ÿ›  Fix horizontal scrolling on build page (#6083) @zoetian ๐Ÿ”—

    ๐Ÿฑ Only set Guardian flag default when flag not provided by user (#6087) @muntac ๐Ÿ”—

    • Only pass garden the configured defaults within Concourse for the guardian flags max-containers and network-pool if it is not set through the garden config file, environment variables or flags.

    ๐Ÿฑ ๐Ÿคท No Impact

    ๐Ÿฑ Deflake k8s topgun (#6030) @jamieklassen ๐Ÿ”—

    ๐Ÿ— build(deps): bump yargs-parser from 13.1.1 to 13.1.2 (#6047) @dependabot ๐Ÿ”—

    ๐Ÿฑ Remind people they need JS to run the browser app (#6065) @steadysupply ๐Ÿ”—

    ๐Ÿฑ Use atc.TaskEnv type in task step (#6066) @vito ๐Ÿ”—

    ๐Ÿ”€ Make --client-cert flag name consistent between fly login and fly sync (#6068) @aoldershaw ๐Ÿ”—

    ๐Ÿฑ Skip yarn benchmark until dependency is fixed (#6078) @clarafu ๐Ÿ”—

  • v6.5.1 Changes

    August 27, 2020

    ๐Ÿฑ ๐Ÿž Bug Fixes

    ๐Ÿฑ Make claims LRU cache safe for concurrent use (#6019) @chrisfarms ๐Ÿ”—

    • Avoid a potential race in access token caching, causing the ATC to panic
  • v6.5.0 Changes

    August 21, 2020

    ๐Ÿฑ ๐Ÿšจ Breaking

    ๐Ÿฑ Generate opaque OAuth2 access tokens (#5897) @aoldershaw ๐Ÿ”—

    • There were several issues that users encountered (particularly after v6.1.0) as a result of long access tokens. Concourse now generates much shorter access tokens rather than using the raw user data.
    • Users' last activity is now tracked on login rather than on every request. Updating the last activity on every request caused database problems at scale. Note: last activity is only relevant to fly active-users

    - This is only a breaking change for any custom automation built around Concourse that authenticates with the Concourse API

    ๐Ÿ›  Fix Gitlab auth to reference a user's username instead of their full name (#5925) @chenbh ๐Ÿ”—

    - If a team was configured for Gitlab auth with the --gitlab-user flag, you must ensure the user referenced is a valid Gitlab username, rather than a user's full name. If not, you must reconfigure the team with fly set-team to reference the username. Teams configured with --gitlab-group are unaffected.

    โž• Add format: trim to load_var step and make it the default (#5894) @evanchaoli ๐Ÿ”—

    • format: trim removes all trailing and leading whitespace from the input file

    - The prior behavior of keeping all whitespace can be used by specifying format: raw (see load_var.format)

    ๐Ÿฑ Prefix containerd specific flags with CONCOURSE_CONTAINERD (#5911) @muntac ๐Ÿ”—

    • This is only a breaking change if you were using the experimental containerd backend. The default guardian backend remains unaffected and is still configured with CONCOURSE_GARDEN environment variables

    ๐Ÿฑ โœˆ๏ธ Features

    ๐Ÿฑ Display favorite pipelines section at the top of the dashboard (#5922) @zoetian ๐Ÿ”—

    ๐Ÿฑ Pipelines can be favorited from the sidebar (#5860) @zoetian ๐Ÿ”—

    ๐Ÿฑ Pipelines can be favorited from the dashboard (#5904) @zoetian ๐Ÿ”—

    ๐Ÿฑ Pipelines can be favorited from the pipeline page (#5979) @zoetian ๐Ÿ”—

    ๐Ÿฑ Automatically archive abandoned pipelines (#5854) @taylorsilva ๐Ÿ”—

    • An abandoned pipeline is one that was once set by a set_pipeline step but no longer is. See the set_pipeline docs for more information

    - The temporary feature flag --enable-archive-pipeline was removed as the archiving pipelines feature is complete

    โž• Add experimental across step for running build plans across a matrix of values (#5887) @aoldershaw ๐Ÿ”—

    • This feature must be enabled using --enable-across-step as it is subject to change - don't be alarmed if your pipelines with across stop working in a future release!

    - There's no official documentation yet, but there are some examples of usage patterns in the RFC

    ๐Ÿ‘ Allow dot and colon in variable path (#5898) @xtremerui ๐Ÿ”—

    • You can now interpolate variables with special characters . and : in the name by wrapping them in double quotes

    - e.g. (("some.secret".field1)) accesses field1 of the secret some.secret

    ๐Ÿฑ Emit warnings for invalid identifiers (#5864) @mouellet ๐Ÿ”—

    • A warning will be emitted for every identifier that doesn't match the validation rules described in the identifier schema

    - After 1 year, we will error when identifiers don't match the validation rules - so, we recommend adhering to the new validation as soon as possible!

    ๐Ÿ‘Œ Support set-pipeline: self for configuring current pipeline (#5737) @evanchaoli ๐Ÿ”—

    - This feature is experimental, and may be removed a future version

    ๐Ÿ— Rerun builds with baggageclaim network errors (#5958) @kirillbilchenko ๐Ÿ”—

    • The --enable-rerun-when-worker-disappears flag now supports rerunning builds after any network error from the ATC to the worker's baggageclaim. Such network errors are common when the worker disappears.

    - Builds will now be rerun when this flag is enabled and the failing step is a nested step (e.g. within an in_parallel)

    ๐Ÿฑ Mount Btrfs loopback with discard option (#5916) @muntac ๐Ÿ”—

    - This punches holes in the underlying loop file making it sparse, and will potentially result in better disk utilization

    ๐Ÿ— Speed up querying for unencrypted builds (#5902) @clarafu ๐Ÿ”—

    - If your environment had a large number of builds and an encrypted database, you might have noticed your web node being slow to start up. An index was added to help speed up the querying of unencrypted builds which is run during the web startup.

    ๐Ÿ‘Œ Support reloading TSA worker keys via SIGHUP signal (#5652) @gaelL ๐Ÿ”—

    ๐Ÿ— get/put/task steps now emit the selected worker to the build log (#5846) @evanchaoli ๐Ÿ”—

    - The build log page and fly watch now display the worker name for get/put/task steps

    โœจ Enhance TasksWaiting metric to include teamId, workerTags, and platform labels (#5981) @aledeganopix4d ๐Ÿ”—

    โž• Add the token text to the fly_success page (#4951) @jvshahid ๐Ÿ”—

    ๐Ÿ”ง TSA's garden client timeout can be configured using --tsa-garden-request-timeout (#5845) @xtreme-sameer-vohra ๐Ÿ”—

    ๐Ÿ‘ Allow fly set-pipeline to take configuration from stdin (#5861) @evanchaoli ๐Ÿ”—

    - The --config flag of the fly set-pipeline command now supports - for reading pipeline config from stdin

    โž• Add --team flag to fly pause-pipeline and fly hide-pipeline commands (#5917) @owenfarrell ๐Ÿ”—

    โž• Add --team flag to fly destroy-pipeline command (#5924) @khng ๐Ÿ”—

    ๐Ÿฑ Always print warning after fly set-pipeline if pipeline is paused (#5883) @agurney ๐Ÿ”—

    โž• Add flag --ignore-event-parsing-errors to fly watch to ignore event parsing errors when an unknown event type or version is encountered. (#5957) @xtreme-sameer-vohra ๐Ÿ”—

    โœ‚ Remove unnecessary updates to the resource's check_error value (#5990) @clarafu ๐Ÿ”—

    - The query will only update the resource check error to NULL if it is not already NULL.

    ๐Ÿ‘Œ Support max container limit with containerd backend (#5836) @xtreme-sameer-vohra ๐Ÿ”—

    ๐Ÿฑ ๐Ÿž Bug Fixes

    ๐Ÿฑ Prevent potential database deadlock after pipeline deletion (#5987) @aoldershaw ๐Ÿ”—

    ๐Ÿ‘ Allow usage of images from custom resource types in fly execute (#5866) @jvshahid ๐Ÿ”—

    Save job.disable_manual_trigger to the database (#5908) @jamieklassen ๐Ÿ”—

    - Fixed the disable_manual_trigger: field on jobs -- since v6.0.0 it had no effect and jobs with this setting could actually still be manually triggered.

    ๐Ÿฑ Return error when a step only contains modifiers (#5910) @taylorsilva ๐Ÿ”—

    โฌ†๏ธ Bump baggageclaim to v1.8.0 to fix deeply-nested volumes with overlay driver (#5961) @vito ๐Ÿ”—

    ๐Ÿฑ ๐Ÿคท No Impact

    ๐Ÿš€ Edit pr template to include new release note generation process (#5903) @clarafu ๐Ÿ”—

    ๐Ÿ›  Fix flakey container sweeper test (#5923) @xtreme-vikram-yadav ๐Ÿ”—

    ๐Ÿฑ Elm tests use Data module for identifiers (#5928) @aoldershaw ๐Ÿ”—

    โœ‚ Remove ignored windows fly test case (#5946) @aoldershaw ๐Ÿ”—

    โœ‚ Remove private_plan from index (#5950) @izabelacg ๐Ÿ”—

    ๐Ÿฑ Prevent pipeline from getting GC'd in testflight (#5951) @aoldershaw ๐Ÿ”—

    ๐Ÿฑ Fly commands don't assume tokens are JWTs (#5955) @aoldershaw ๐Ÿ”—

    ๐Ÿฑ Use access token for Concourse API in topgun (#5965) @aoldershaw ๐Ÿ”—

    ๐Ÿฑ Deflake metrics tests (#5975) @jamieklassen ๐Ÿ”—

    ๐Ÿ”ง Skip failing k8s topgun tests for containerd without a dns proxy or dns servers configured (#5976) @xtreme-sameer-vohra ๐Ÿ”—

    ๐Ÿ›  Fix go 1.15 int-to-string conversion warning in tests (#5983) @aoldershaw ๐Ÿ”—

    ๐Ÿ— build(deps): bump lodash from 4.17.15 to 4.17.19 in /web/wats (#5890) @dependabot ๐Ÿ”—

    ๐Ÿ— build(deps): bump elliptic from 6.5.2 to 6.5.3 (#5940) @dependabot ๐Ÿ”—

  • v6.4.1 Changes

    August 04, 2020
    ๐Ÿ”’ ๐Ÿ”— security
    • ๐Ÿ›  Fix Gitlab connector configuration using Full Name instead of Username
      • Any Concourse teams configured with Gitlab users may need to be updated. Previously a Gitlab users Full Name was used to add them to a Concourse team. Now the users Username in Gitlab is used by Concourse to verify team membership. If the Full Name and Username are the same then no change is necessary.
  • v6.4.0 Changes

    July 15, 2020
    ๐Ÿฑ ๐Ÿ”— feature
    • @mouellet added a way of renaming pipeline resources while preserving version history by updating the resource name (as well as any reference in steps) and specifying its old name as old_name. After the pipeline has been configured, the old_name field can be removed. #5833
    ๐Ÿฑ ๐Ÿ”— feature
    • ๐Ÿ’ป Archived pipelines can be displayed in the web UI via a toggle switch in the top bar. #5777, #5760
    ๐Ÿฑ ๐Ÿ”— fix
    ๐Ÿฑ ๐Ÿ”— fix
    ๐Ÿฑ ๐Ÿ”— fix
    • Pipelines can be re-ordered in the dashboard when filtering. This was a regression introduced in 6.0. #5821
    ๐Ÿฑ ๐Ÿ”— feature
    • ๐Ÿ’… Style improvements to the sidebar. #5778
    ๐Ÿฑ ๐Ÿ”— fix
    • ๐Ÿ’ป The sidebar can now be expanded in the UI - no more long pipeline names being cut off! #5782
    ๐Ÿฑ ๐Ÿ”— feature
    • โž• Add --include-archived flag for fly pipelines command. #5673
    ๐Ÿฑ ๐Ÿ”— fix
    • fly login now accepts arbitrarily long tokens when pasting the token manually into the console. Previously, the limit was OS dependent (with OSX having a relatively small maximum length of 1024 characters). This has been a long-standing issue, but it became most noticeable after 6.1.0 which significantly increased the size of tokens. Note that the pasted token is now hidden in the console output. #5770
    ๐Ÿฑ ๐Ÿ”— feature
    • โž• Add --team flag for fly set-pipelines command #5805
    ๐Ÿฑ ๐Ÿ”— fix
    • ๐Ÿ›  Fix a validation issue where a step can be set with 0 attempts causing the ATC to panic. #5830
    ๐Ÿฑ ๐Ÿ”— fix
    • ๐Ÿ›  Fix tooltip on pipeline names. PR: #5855
    ๐Ÿฑ ๐Ÿ”— feature
    • ๐Ÿ”จ Refactor existing step structure to simplify introducing new steps. The primary user facing changes are stricter validation and slightly different step validation messages. Previously, fields that were not part of a step wouldn't have failed validation,however, now they will. This will impact stateful actions such as set-pipeline. PR: #5504, #5878
    ๐Ÿฑ ๐Ÿ”— feature
    • โšก๏ธ @tlwr updated the set_pipeline step to be used across teams. This is 'experimental' for now, please share your feedback in the thread. PR: #5729
    ๐Ÿฑ ๐Ÿ”— feature
    • @evanchaoli added OPA integration to Concourse to enable policy enforcement. The RFC #41 for the integration is still in progress so it is considered 'experimental'. PR: #5034
    ๐Ÿฑ ๐Ÿ”— feature
    • โž• Add support for field in containerd runtime. Issue: #5144 PR: #5704
  • v6.3.1 Changes

    August 04, 2020
    ๐Ÿ”’ ๐Ÿ”— security
    • ๐Ÿ›  Fix Gitlab connector configuration using Full Name instead of Username
      • Any Concourse teams configured with Gitlab users may need to be updated. Previously a Gitlab users Full Name was used to add them to a Concourse team. Now the users Username in Gitlab is used by Concourse to verify team membership. If the Full Name and Username are the same then no change is necessary.
  • v6.3.0 Changes

    June 11, 2020
    ๐Ÿฑ ๐Ÿ”— feature

    ๐Ÿš€ After switching the default resource checking component to be lidar during the v6.0.0 release, users have noticed the resource checking rate has been very spiky. This spiky behaviour is caused by the fact that lidar runs all the checks that have been queued up at once, causing the spikes of checks to happen all at the same time when the checker runs on its interval.

    We added a rate limit to resource checking in order to help spread out the rate of checks. Before, without the rate limit resource checking was like .|.|.|.| and now it's like ~~~~~~~.

    0๏ธโƒฃ This rate limit is defaulted to be determined by calculating the number of checkables (resources and resource types) that need to be checked per second in order to check everything within the default checking interval. This means that if there are 600 checkables and the default checking interval is 60 seconds, we would need to run 10 checks per second in order to check everything in 60 seconds. This rate limit of checks can be modified through the max-checks-per-second flag to be a static number and also turned off by setting it to -1.

    0๏ธโƒฃ On top of adding a rate limit, we also changed the default for the lidar-scanner-interval to be 10 seconds rather than the previous default of 1 minute. This will also help spread out the number of checks to happen on more intervals of the checker. #5676

    ๐Ÿš€ When we upgraded our large deployment to v6.3.0 with the default rate limiter on resource checking, we saw that the number of checks started became much more spread out when compared to before. This first metric shows the number of resource checks started per minute before v6.3.0.

    The following metric shows the number of checks started per minute after adding the rate limiting.

    ๐Ÿฑ ๐Ÿ”— feature
    • โฑ When distributed tracing is configured, Concourse will now emit spans for several of its backend operations, including resource scanning, check execution, and job scheduling. These spans will be appropriately linked when viewed in a tracing tool like Jaeger, allowing operators to better observe the events that occur between resource checking and build execution. #5659
    ๐Ÿฑ ๐Ÿ”— feature

    ๐Ÿ”ง When distributed tracing is configured, all check, get, put, and task containers will be run with the TRACEPARENT environment variable set, which contains information about the parent span following the w3c trace context format:


    ๐Ÿ‘ Using this information, your tasks and custom resource_types can emit spans to a tracing backend, and these spans will be appropriately linked to the step in which they ran. This can be particularly useful when integrating with downstream services that also support tracing. #5653

    ๐Ÿฑ ๐Ÿ”— feature
    • ๐Ÿ”ง The set_pipeline step now supports vars within its plan configuration (the file:, vars:, and var_files: fields), thanks to @evanchaoli! #5277
    ๐Ÿฑ ๐Ÿ”— feature
    • โฑ When the scheduler tries to start a build with a version that does not exist, it will print an error message to the pending preparation build page. This should help give visibility into why a build is stuck pending. #5477
    ๐Ÿฑ ๐Ÿ”— feature
    • โฑ When the scheduler starts a build, it will send a notification to the build tracker to run it. Without this notification from the scheduler to the build tracker, it can take up to approximately 10 seconds before your build gets run after being in a started state. #5631
    ๐Ÿฑ ๐Ÿ”— feature
    • ๐Ÿ‘ Proxy support for NewRelic emitter
    ๐Ÿฑ ๐Ÿ”— feature
    • โž• Add tracing to allow users and developers to observe volume streaming from source to destination volumes. #5579
    ๐Ÿฑ ๐Ÿ”— feature
    ๐Ÿฑ ๐Ÿ”— fix
    • ๐Ÿ›  Fixed a bug introduced in v6.1.0 that caused admin users to no longer be able to login to any team. #5741
    ๐Ÿฑ ๐Ÿ”— fix
    • โฑ The algorithm within the job scheduler component will now take into account whether or not a version exists in the database while determining a set of input versions for the next build of a job. Previously, when you had an input with passed constraints, it could pick a version that did not exist in the database because it picked versions purely based on if they have passed the upstream passed constraint jobs, regardless if that version did not exist anymore. This fix makes the algorithm skip over versions that do not exist. #5724
    ๐Ÿฑ ๐Ÿ”— fix
    • ๐Ÿ›  Fixed a bug where task caches were getting garbage collected everytime you set the pipeline. #5497
    ๐Ÿฑ ๐Ÿ”— fix
    • ๐Ÿ›  Fixed a bug "invalid memory address or nil pointer dereference" in NewRelic emitter. #5697