PAM release notes

These are the new features, enhancements, resolved issues, and deprecations for Puppet Application Manager.

Restriction: Because kURL does not support upgrading more than two Kubernetes versions at once, if you're upgrading from an older version of PAM, you might need to follow a specific upgrade path to avoid failures.
  • If you're on PAM version 1.56.0 or earlier, you must upgrade to PAM 1.80.0 before upgrading to PAM 1.81.1 or later.

28 September 2022 (Puppet Application Manager 1.81.1)

New in this release:
  • Kubernetes version upgrade. For standalone and HA installations, this version includes an upgrade of Kubernetes to version 1.23.9.
    Important upgrade information: The upgrade process takes place on all nodes, and first upgrades Kubernetes to version 1.22 before upgrading to version 1.23.9. For a three-node cluster, you can expect the upgrade process to take around an hour. Confirmations are required during the upgrade process.

    Additionally, because kURL can only be upgraded two minor versions at a time, if you're upgrading from PAM version 1.56.0 or earlier, you must upgrade to PAM 1.80.0 before upgrading to PAM 1.81.1.

    For legacy installations, Kubernetes remains on version 1.19.15. If you're not sure which installation type you're running, see How to determine your version of Puppet Application Manager.

16 August 2022 (Puppet Application Manager 1.80.0)

New in this release:
  • Component upgrades to address CVEs. To address various CVEs, this version upgrades containerd to 1.4.13, KOTS to 1.80.0, ekco to 0.19.6, and Goldpinger to 3.5.1-5.2.0.
Resolved in this release:
  • Fixed an issue where legacy encryption keys didn't load properly during snapshot restores.

2 August 2022 (Puppet Application Manager 1.76.2)

New in this release:
  • Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of OpenEBS to version 3.2.0, an upgrade of Weave to version 2.8.1-20220720, an upgrade of Project Contour to version 1.21.1, and an upgrade of MinIO to version 2022-07-17T15-43-14Z.

20 July 2022 (Puppet Application Manager 1.76.1)

New in this release:
  • Support for Red Hat Enterprise Linux version 8.6. Beginning with version 1.76.1, PAM can be successfully installed on systems running Red Hat Enterprise Linux version 8.6.
  • More log data is now retained. To ensure that you and our Support team have the data you need in debugging scenarios, the size of the pod logs has been increased from 10 files of 10MiB each to 10 files of 50MiB each. This change increases the storage used in /var/log/pods by 400MiB.
  • Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of Velero to version 1.9.0 and an upgrade of the Prometheus bundle to version 0.57.0-36.2.0.
  • Other component upgrades. This version also includes an upgrade of Registry to version 2.8.1 and an upgrade of MinIO to version 2022-07-06T20-29-49Z.
Resolved in this release:
  • Velero pods no longer get stuck in a pending state when creating a snapshot to be saved to internal storage on a Puppet-supported cluster.

23 June 2022 (Puppet Application Manager 1.72.1)

New in this release:
  • Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of ekco to version 0.19.2 and an upgrade of kURL to v2022.06.17-0.

26 May 2022 (Puppet Application Manager 1.70.1)

New in this release:
  • Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of Project Contour to version 1.21.0, an upgrade of Velero to version 1.8.1, and an upgrade of the Prometheus bundle to version 0.56.2-35.2.0.
Resolved in this release:
  • Image garbage collection in Kubernetes installer-created clusters (embedded clusters) no longer removes images outside of the application's dedicated registry namespace.
  • The Deploy button is now present in newly updated versions after the configuration is updated from the previously deployed version.
  • Legends are now shown properly for the performance graphs on the dashboard.

12 April 2022 (Puppet Application Manager 1.68.0)

New in this release:
  • Install a specific version of an application. When installing a Puppet application using the automated installation method, you now have the option to specify the application's version by passing the --app-version-label=<version> flag to the kubectl kots install command. For more information, go to Automate PAM and Puppet application online installations.
  • Status reporting improvements. The status reporting tools can now detect when an application is being upgraded.
  • Component upgrades to address CVEs. To address various CVEs in Envoy, this version includes an upgrade of Project Contour to version 1.20.1.
  • Other component upgrades. This version includes an upgrade of KOTS to version 1.68.0, which enables Kubernetes audit event logging by default and adds a 1 GB storage requirement for /var/log/apiserver.
Resolved in this release:
  • During image garbage collection, images still in use by the cluster are no longer in danger of being deleted from the private registry in a Kubernetes installer-created cluster.

1 March 2022 (Puppet Application Manager 1.64.0)

Resolved in this release:
  • Diffs are now shown correctly in the PAM UI.
  • The OpenSSL package is no longer a prerequisite for successful installation on newer Red Hat Enterprise Linux 7 systems.
  • You can now successfully install Puppet Application Manager on Red Hat Enterprise Linux 8 systems without the need to force-install the kurl-local audit-libs library.

17 February 2022 (Puppet Application Manager 1.62.0)

Important: Version 1.0.2 of the puppetlabs/pam_firewall module is now available. To avoid conflicts, upgrade the module before upgrading Puppet Application Manager to version 1.62.0.
New in this release:
  • Kubernetes version upgrade. For standalone and HA installations, this version includes an upgrade of Kubernetes to version 1.21.8.
    Important upgrade information: The upgrade process takes place on all nodes, and first upgrades Kubernetes to version 1.20 before upgrading to version 1.21.8. For a three-node cluster, you can expect the upgrade process to take around an hour. Confirmations are required during the upgrade process.
    For legacy installations (installed before May 2021), this version includes an upgrade of Kubernetes to version 1.19.15.
    Tip: See How to determine your version of Puppet Application Manager if you're not sure which installation type you're running.
  • Prometheus enabled on standalone architecture. Beginning with version 1.62.0 Prometheus is enabled by default on all new and existing standalone Puppet Application Manager installations. Prometheus requires an additional 350m CPU and 500MiB of memory, so ensure your system is properly sized before upgrading. Prometheus is an optional component; if you need to disable it to conserve resources, see Optional components.
  • Automatic certificate rotation. By default, the self-signed certificates used by Project Contour and Envoy expire after one year. This version includes an update that auto-rotates those certificates before they expire.
  • Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of containerd to version 1.4.12.
  • Other component upgrades. This version includes an upgrade of KOTS to version 1.62.0.
Deprecated in this release:
  • Legacy architecture. The legacy architecture, which was the version of Puppet Application Manager available for installation prior to May 2021, is now deprecated. (See How to determine your version of Puppet Application Manager if you need to confirm whether you're running the legacy architecture.) The legacy architecture utilizes Rook 1.0, which is incompatible with Kubernetes version 1.20 and newer versions. Kubernetes version 1.19 is no longer receiving security updates. Puppet will continue to update legacy architecture components other than Kubernetes until 30 June 2022. If security advisories against Kubernetes 1.19 arise, the remediation path is to migrate to one of the newer architectures by following the instructions in Migrating PAM data to a new system.
    Important: Before beginning the migration process from a legacy deployment you must upgrade to PAM version 1.62.0 with the force-reapply-addons flag included in the upgrade command. Find upgrade instructions at PAM legacy upgrades and PAM offline legacy upgrades.

30 November 2021 (Puppet Application Manager 1.56.0)

This release includes an upgrade of KOTS to version 1.56.0, which adds the following improvements:
  • Improved support bundles: Adds an option to upload a support bundle directly from Puppet Application Manager.
  • Improved troubleshooting: Adds detailed information on failing pods to the Troubleshoot tab.

6 October 2021 (Puppet Application Manager 1.52.1)

New in this release:
  • Improved statuses. More granular status levels are now available from the Application tab.
  • Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of Kubernetes to 1.19.15.
  • Other component upgrades. This version includes an upgrade of KOTS to version 1.52.1.
Resolved in this release:
  • Generating a support bundle no longer results in unusually high memory use.
  • Preflight check logs post to info level for progress messages and to error level for error messages.

25 August 2021 (Puppet Application Manager 1.49.0)

New in this release:
  • Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of Kubernetes to 1.19.13, an upgrade of Project Contour to 1.18.0, and an upgrade of Velero to 1.6.2.
  • Goldpinger. High availability architectures now include Goldpinger, which aids the debugging of network issues.
  • containerd upgrade. This version includes an upgrade of containerd to version 1.4.6, and removes the need to use the force-reapply-addons option when upgrading.
  • Other component upgrades. This version includes an upgrade of KOTS to version 1.49.0, an upgrade of ekco to 0.11.0, an upgrade of Prometheus to 0.49.0, and an upgrade of Rook to 1.5.12.

30 June 2021 (Puppet Application Manager 1.44.1)

New in this release:
  • Certificate auto-rotation for standalone architecture. Certificates are now automatically rotated for the Kubernetes API and Puppet Application Manager UI in the standalone architecture. With this change, certificate auto-rotation is now supported in all Puppet Application Manager architectures.
  • Rook upgrades. This version includes an upgrade of Rook in the high availability architecture to 1.5.11 and the version of Rook in the legacy architecture to 1.0.4-14.2.21. These upgrades address a vulnerability in Ceph components (CVE-2021-20288).
  • Prometheus upgrade. This version includes an upgrade of Prometheus in the high availability and legacy architectures to 0.48.1. Additionally, Prometheus disk usage is now limited in order to preserve the storage space required for the usage charts on the Application tab.
  • Other component upgrades. This version includes an upgrade of KOTS to version 1.44.1, an upgrade of Project Contour to version 1.15.1, and an upgrade of Weave to version 2.8.1.
Resolved in this release:
  • Snapshots can now successfully use the Other S3-Compatible Storage option as the storage destination.

    To apply this update, add the force-reapply-addons option during upgrade. For example:

    curl <url> | bash -s force-reapply-addons

26 May 2021

New in this release:

  • runC. The version of runC has been upgraded to v1.0.0-rc95 to address CVE-2021-30465.

Known issues in this release:

  • Running the KOTS installer with the airgap and kurl-registry-ip flags results in an error.

    As a workaround (if you do not have any applications already installed in the cluster), delete the registry service, recreate the registry service IP and then re-run the installation script with the kurl-registry-ip flag.

10 May 2021 (Puppet Application Manager 1.40.0)

New in this release:
  • Distinct architectures for standalone and high availability deployments of the Puppet Application Manager platform. Standalone supports lower system requirements and resolves inherent flaws in using Ceph on a single node. High availability uses an updated version of Rook for faster, more reliable distributed storage.
    Note: It is not possible currently to upgrade to these architectures from existing installations. However, migrating applications between them is on the roadmap for a future release.
  • The previous architecture is maintained as the legacy configuration. This version includes an upgrade of Kubernetes to 1.19.10; this upgrade process upgrades through Kubernetes 1.18, and happens on all nodes. It can take ~1 hour to do for a 3-node cluster, and requires confirmations during that period. It also includes an upgrade of Project Contour to version 1.14.1, adds Metrics Server 0.4.1, an upgrade of ekco to 0.10.1, and an upgrade of Prometheus to 2.26.0.

    For more information on legacy upgrades, see PAM legacy upgrades.

15 April 2021 (Puppet Application Manager 1.38.0)

New in this release:
  • Snapshots. Puppet Application Manager now supports full (instance-level) snapshots, which can be used for application rollbacks and disaster recovery. For more information, see Backing up Puppet Application Manager using snapshots.
  • Component upgrades. This version includes an upgrade of KOTS to version 1.38.0.

17 February 2021 (Puppet Application Manager 1.29.3)

New in this release:
  • Support for Ubuntu 20.04. You can now run Puppet Application Manager on Ubuntu 20.04.
  • Component upgrades. This version includes an upgrade of Prometheus to version 2.22.1 and Prometheus Operator to version 0.44.1, an upgrade of KOTS to version 1.29.3, an upgrade of Project Contour to version 1.12.0, and an upgrade of ekco to version 0.10.0.

3 February 2021 (Puppet Application Manager 1.29.2)

New in this release:
  • Component upgrades. This version includes an upgrade of KOTS to version 1.29.2, an upgrade of Project Contour to version 1.11.0, and an upgrade of containerd to version 1.4.3.
Resolved in this release:
  • During their initial preflight checks, new installations now pull images successfully and no longer report a Failed to pull image error.

7 December 2020

New in this release:
  • Support for Red Hat Enterprise Linux (RHEL) 8 and CentOS 8. You can now run Puppet Application Manager on RHEL version 8 and CentOS version 8. To support this change, containerd is now used independently of Docker during the installation process.
  • Component upgrades. This version includes an upgrade of Kubernetes to version 1.17.13.