PAM release notes

22 October 2024 (Puppet Application Manager 1.112.4)

Note: Customers using the pam_firewall module must upgrade the module to version 1.0.5 prior to upgrading PAM to 1.112.4.

New in this release:

Kubernetes version upgrade. For standalone and HA installations, this version includes an upgrade of Kubernetes to version 1.30.5.
Important upgrade information: The upgrade process takes place on all nodes, upgrading Kubernetes to version 1.30.5 on each. For a three-node cluster, you can expect the upgrade process to take around an hour. Confirmations are required during the upgrade process.
Additionally, please keep in mind that kURL can only be upgraded two minor versions at a time.
Component upgrades to address security issues. This version upgrades the following:
Note: Before updating, ensure MinIO has 10GB of free space.
- KOTS: 1.112.4
- kURL: v2024.09.26-0
- containerd: 1.6.33
- Flannel: 0.25.6
- Project Contour: 1.30.0
- Velero: 1.14.0
- Metrics Server: 0.6.4
- ekco: 0.28.7
- Prometheus: 0.76.1-62.6.0
- Registry: 2.8.3
- OpenEBS: 4.1.0
- MinIO: 2024-08-26T15-33-07Z
- Rook: 1.12.8
- Goldpinger: 3.10.0-6.2.0

23 July 2024 (Puppet Application Manager 1.110.0)

New in this release:

Component upgrades to address security issues. This version upgrades the following:
Note: Before updating, ensure MinIO has 10GB of free space.
- KOTS: 1.110.0
- kURL: v2024.07.02-0
- containerd: 1.6.32
- Flannel: 0.25.4
- Project Contour: 1.29.0
- Velero: 1.13.2
- Metrics Server: 0.6.4
- ekco: 0.28.7
- Prometheus: 0.74.0-59.0.0
- Registry: 2.8.3
- OpenEBS: 4.0.0
- MinIO: 2024-05-10T01-41-38Z
- Rook: 1.12.8
- Goldpinger: 3.10.0-6.2.0

21 May 2024 (Puppet Application Manager 1.109.0)

New in this release:

Kubernetes version upgrade. For standalone and HA installations, this version includes an upgrade of Kubernetes to version 1.28.9.
Important upgrade information: The upgrade process takes place on all nodes, upgrading Kubernetes to version 1.28.9 on each. For a three-node cluster, you can expect the upgrade process to take around an hour. Confirmations are required during the upgrade process.
Additionally, please keep in mind that kURL can only be upgraded two minor versions at a time.
Component upgrades to address security issues. This version upgrades the following:
Note: Before updating, ensure MinIO has 10GB of free space.
- KOTS: 1.109.0
- kURL: v2024.05.03-0
- containerd: 1.6.31
- Flannel: 0.25.1
- Project Contour: 1.28.3
- Velero: 1.13.2
- Metrics Server: 0.6.4
- ekco: 0.28.6
- Prometheus: 0.73.1-58.1.1
- Registry: 2.8.3
- OpenEBS: 4.0.0
- MinIO: 2024-04-06T05-26-02Z
- Rook: 1.12.8
- Goldpinger: 3.10.0-6.2.0

26 March 2024 (Puppet Application Manager 1.108.0)

New in this release:

Component upgrades to address security issues. This version upgrades the following:
Note: Before updating, ensure MinIO has 10GB of free space.
- KOTS: 1.108.0
- kURL: v2024.02.23-0
- containerd: 1.6.28
- Flannel: 0.24.2
- Project Contour: 1.27.0
- Velero: 1.12.3
- Metrics Server: 0.6.4
- ekco: 0.28.4
- Prometheus: 0.71.2-56.6.0
- OpenEBS: 3.10.0
- MinIO: 2024-02-17T01-15-57Z

13 February 2024 (Puppet Application Manager 1.107.0)

New in this release:

Component upgrades to address security issues. This version upgrades the following:
Note: Before updating, ensure MinIO has 10GB of free space.
- KOTS: 1.107.0
- kURL: v2024.01.09-0
- containerd: 1.6.26
- Flannel: 0.24.0
- Project Contour: 1.27.0
- Registry: 2.8.3
- Velero: 1.12.2
- ekco: 0.28.4
- Prometheus: 0.70.0-55.0.0
- OpenEBS: 3.10.0
- MinIO: 2024-01-01T16-36-33Z
- Rook: 1.12.8

7 November 2023 (Puppet Application Manager 1.103.3)

New in this release:

Kubernetes version upgrade. For standalone and HA installations, this version includes an upgrade of Kubernetes to version 1.28.2.
Important upgrade information: The upgrade process takes place on all nodes, and first upgrades Kubernetes to version 1.27.6 before upgrading to version 1.28.2 on each. For a three-node cluster, you can expect the upgrade process to take around an hour. Confirmations are required during the upgrade process.
Additionally, please keep in mind that kURL can only be upgraded two minor versions at a time.
Component upgrades to address security issues. This version upgrades the following:
Note: Before updating, ensure MinIO has 10GB of free space.
- KOTS: 1.103.3
- kURL: v2023.10.26-0
- containerd: 1.6.24
- Flannel: 0.22.3
- Project Contour: 1.26.1
- Registry: 2.8.3
- Velero: 1.12.1
- OpenEBS: 3.9.0
- MinIO: 2023-10-16T04-13-43Z
- Rook: 1.12.6

26 September 2023 (Puppet Application Manager 1.102.2)

New in this release:

Migrated from Weave to Flannel. Flannel has replaced Weave as the Kubernetes CNI on Puppet-supported clusters, as Weave is no longer supported. The installation has additional interactive prompts to support this change.
Important upgrade information:
- IPv6 and dual-stack networks are not supported on Flannel.
- Pod-to-pod networking now depends on UDP port 8472 being open instead of ports 6783 and 6784.
Added a host preflight. Added a host preflight in the installer to stop installation if the installer detects the presence of a default REJECT rule in the FORWARD chain of iptables.
Important upgrade information: This is a known issue with the Flannel installation. To check for a REJECT rule in the FORWARD chain of iptables, run:
```
iptables -vL FORWARD
```
If there are any REJECT rules, those rules must be removed prior to the upgrade. They can be restored afterwards.
Component upgrades to address security issues. This version upgrades the following:
Note: Before updating standalone installations, ensure there is at least 10GB of free space in /var/openebs to allow for migration of MinIO in this release.
- KOTS: 1.102.2
- kURL: v2023.09.15-0
- containerd: 1.6.22
- Weave: REMOVED
- Flannel: 0.22.2
- Project Contour: 1.25.2
- Velero: 1.11.1
- Kubernetes Metrics Server: 0.6.4
- ekco: 0.28.3
- Prometheus: 0.68.0-51.0.0
- OpenEBS: 3.8.0
- MinIO: 2023-09-04T19-57-37Z
- Rook: 1.12.3
Note: If you are using the firewall module to manage your PAM install, you must update it to version 1.0.4 to support this PAM release.

18 July 2023 (Puppet Application Manager 1.100.3)

New in this release:

Kubernetes version upgrade. For standalone and HA installations, this version includes an upgrade of Kubernetes to version 1.26.6.
Important upgrade information: The upgrade process takes place on all nodes, and first upgrades Kubernetes to version 1.25 before upgrading to version 1.26.6 on each. For a three-node cluster, you can expect the upgrade process to take around an hour. Confirmations are required during the upgrade process.
Additionally, please keep in mind that kURL can only be upgraded two minor versions at a time.
Component upgrades to address security issues. This version upgrades, adds, and removes the following:
Note: Before updating, ensure MinIO has 10GB of free space.
- KOTS: 1.100.3
- kURL: v2023.06.27-0
- Prometheus: 0.65.2-46.8.0
- OpenEBS: 3.7.0
- MinIO: 2023-06-19T19-52-50Z
- Rook: 1.11.8
Note: If you are using the firewall module to manage your PAM install, you must update it to version 1.0.3 to support this PAM release.

8 June 2023 (Puppet Application Manager 1.99.0)

New in this release:

Component upgrades to address security issues. This version upgrades the following:
Note: Before updating, ensure MinIO has 10GB of free space.
- KOTS: 1.99.0
- kURL: v2023.05.22-0
- containerd: 1.6.21
- Weave: 2.8.1-20230417
- Project Contour: 1.25.0
- Registry: 2.8.2
- Velero: 1.11.0
- ekco: 0.27.1
- Prometheus: 0.65.1-45.28.0
- OpenEBS: 3.6.0
- MinIO: 2023-05-18T00-05-36Z
- Rook: 1.11.5
- Goldpinger: 3.7.0-6.0.1
Note: For offline HA installs the Rook update in this release can cause significant downtime (around 4 hours) while downloading additional files. It is possible to do some of this prior to upgrading Puppet Application Manager from 1.97.0 to 1.99.0 to decrease the downtime.

25 April 2023 (Puppet Application Manager 1.97.0)

New in this release:

Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of OpenEBS to version 3.5.0, an upgrade of kURL to v2023.04.11-0, an upgrade of containerd to 1.6.20, an upgrade of Weave to version 2.8.1-20230324, an upgrade of Project Contour to version 1.24.3, an upgrade of ekco to 0.26.5, an upgrade of Velero to version 1.10.2, an upgrade of the Prometheus bundle to version 0.63.0-45.9.1, and upgrade of Kubernetes Metrics Server to version 0.6.3, an upgrade of KOTS to 1.97.0, an upgrade of MinIO to version 2023-03-24T21-41-23Z, and an upgrade of Goldpinger to 3.7.0-5.6.0.
Note: Before updating, ensure MinIO has 10GB of free space.

Deprecated in this release:

force-reapply-addons flag. Starting with Puppet Application Manager 1.97.0, the force-reapply-addons flag is deprecated and generates a warning on use. This flag is only required when upgrading to a Puppet Application Manager version prior to 1.97.0.

28 February 2023 (Puppet Application Manager 1.94.0)

New in this release:

Kubernetes version upgrade. For standalone and HA installations, this version includes an upgrade of Kubernetes to version 1.24.10.
Important upgrade information: The upgrade process takes place on all nodes, and first upgrades Kubernetes to version 1.24.10 on each. For a three-node cluster, you can expect the upgrade process to take around an hour. Confirmations are required during the upgrade process.
Additionally, because kURL can only be upgraded two minor versions at a time, if you're on PAM version 1.80.0 or earlier, you must upgrade to PAM 1.81.1 before upgrading to PAM 1.94.0.

This release also includes component upgrades to address security issues and general bug fixes.

10 January 2023 (Puppet Application Manager 1.91.3)

New in this release:

Component upgrades to address security issues and support RHEL 8.7. This version upgrades the following:
Note: Before updating, ensure MinIO has 10GB of free space.
- KOTS: 1.91.3
- MinIO: 2022-10-20T00-55-09Z
- OpenEBS: 3.3.0
- Prometheus: 0.60.1-41.7.3
- ekco: 0.26.1
- Velero: 1.9.4
- Project Contour: 1.23.1
- kURL: v2022.12.12-0
- Weave: 2.8.1-20221122
- Goldpinger: 3.7.0-5.5.0

28 September 2022 (Puppet Application Manager 1.81.1)

New in this release:

Kubernetes version upgrade. For standalone and HA installations, this version includes an upgrade of Kubernetes to version 1.23.9.
Important upgrade information: The upgrade process takes place on all nodes, and first upgrades Kubernetes to version 1.22 before upgrading to version 1.23.9. For a three-node cluster, you can expect the upgrade process to take around an hour. Confirmations are required during the upgrade process.
Additionally, because kURL can only be upgraded two minor versions at a time, if you're upgrading from PAM version 1.56.0 or earlier, you must upgrade to PAM 1.80.0 before upgrading to PAM 1.81.1.

For legacy installations, Kubernetes remains on version 1.19.15. If you're not sure which installation type you're running, see How to determine your version of Puppet Application Manager.

16 August 2022 (Puppet Application Manager 1.80.0)

New in this release:

Component upgrades to address CVEs. To address various CVEs, this version upgrades containerd to 1.4.13, KOTS to 1.80.0, ekco to 0.19.6, and Goldpinger to 3.5.1-5.2.0.

Resolved in this release:

Fixed an issue where legacy encryption keys didn't load properly during snapshot restores.

2 August 2022 (Puppet Application Manager 1.76.2)

New in this release:

Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of OpenEBS to version 3.2.0, an upgrade of Weave to version 2.8.1-20220720, an upgrade of Project Contour to version 1.21.1, and an upgrade of MinIO to version 2022-07-17T15-43-14Z.
Note: Before updating, ensure MinIO has 10GB of free space.

20 July 2022 (Puppet Application Manager 1.76.1)

New in this release:

Support for Red Hat Enterprise Linux version 8.6. Beginning with version 1.76.1, PAM can be successfully installed on systems running Red Hat Enterprise Linux version 8.6.
More log data is now retained. To ensure that you and our Support team have the data you need in debugging scenarios, the size of the pod logs has been increased from 10 files of 10MiB each to 10 files of 50MiB each. This change increases the storage used in /var/log/pods by 400MiB.
Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of Velero to version 1.9.0 and an upgrade of the Prometheus bundle to version 0.57.0-36.2.0.
Other component upgrades. This version also includes an upgrade of Registry to version 2.8.1 and an upgrade of MinIO to version 2022-07-06T20-29-49Z.
Note: Before updating, ensure MinIO has 10GB of free space.

Resolved in this release:

Velero pods no longer get stuck in a pending state when creating a snapshot to be saved to internal storage on a Puppet-supported cluster.

23 June 2022 (Puppet Application Manager 1.72.1)

New in this release:

Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of ekco to version 0.19.2 and an upgrade of kURL to v2022.06.17-0.

26 May 2022 (Puppet Application Manager 1.70.1)

New in this release:

Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of Project Contour to version 1.21.0, an upgrade of Velero to version 1.8.1, and an upgrade of the Prometheus bundle to version 0.56.2-35.2.0.

Resolved in this release:

Image garbage collection in Kubernetes installer-created clusters (embedded clusters) no longer removes images outside of the application's dedicated registry namespace.
The Deploy button is now present in newly updated versions after the configuration is updated from the previously deployed version.
Legends are now shown properly for the performance graphs on the dashboard.

12 April 2022 (Puppet Application Manager 1.68.0)

New in this release:

Install a specific version of an application. When installing a Puppet application using the automated installation method, you now have the option to specify the application's version by passing the --app-version-label=<version> flag to the kubectl kots install command. For more information, go to Automate PAM and Puppet application online installations.
Status reporting improvements. The status reporting tools can now detect when an application is being upgraded.
Component upgrades to address CVEs. To address various CVEs in Envoy, this version includes an upgrade of Project Contour to version 1.20.1.
Other component upgrades. This version includes an upgrade of KOTS to version 1.68.0, which enables Kubernetes audit event logging by default and adds a 1 GB storage requirement for /var/log/apiserver.

Resolved in this release:

During image garbage collection, images still in use by the cluster are no longer in danger of being deleted from the private registry in a Kubernetes installer-created cluster.

1 March 2022 (Puppet Application Manager 1.64.0)

Resolved in this release:

Diffs are now shown correctly in the PAM UI.
The OpenSSL package is no longer a prerequisite for successful installation on newer Red Hat Enterprise Linux 7 systems.
You can now successfully install Puppet Application Manager on Red Hat Enterprise Linux 8 systems without the need to force-install the kurl-local audit-libs library.

17 February 2022 (Puppet Application Manager 1.62.0)

Important: Version 1.0.2 of the puppetlabs/pam_firewall module is now available. To avoid conflicts, upgrade the module before upgrading Puppet Application Manager to version 1.62.0.

New in this release:

Kubernetes version upgrade. For standalone and HA installations, this version includes an upgrade of Kubernetes to version 1.21.8.
Important upgrade information: The upgrade process takes place on all nodes, and first upgrades Kubernetes to version 1.20 before upgrading to version 1.21.8. For a three-node cluster, you can expect the upgrade process to take around an hour. Confirmations are required during the upgrade process.
For legacy installations (installed before May 2021), this version includes an upgrade of Kubernetes to version 1.19.15.
Tip: See How to determine your version of Puppet Application Manager if you're not sure which installation type you're running.
Prometheus enabled on standalone architecture. Beginning with version 1.62.0 Prometheus is enabled by default on all new and existing standalone Puppet Application Manager installations. Prometheus requires an additional 350m CPU and 500MiB of memory, so ensure your system is properly sized before upgrading. Prometheus is an optional component; if you need to disable it to conserve resources, see Optional components.
Automatic certificate rotation. By default, the self-signed certificates used by Project Contour and Envoy expire after one year. This version includes an update that auto-rotates those certificates before they expire.
Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of containerd to version 1.4.12.
Other component upgrades. This version includes an upgrade of KOTS to version 1.62.0.

Deprecated in this release:

Legacy architecture. The legacy architecture, which was the version of Puppet Application Manager available for installation prior to May 2021, is now deprecated. (See How to determine your version of Puppet Application Manager if you need to confirm whether you're running the legacy architecture.) The legacy architecture utilizes Rook 1.0, which is incompatible with Kubernetes version 1.20 and newer versions. Kubernetes version 1.19 is no longer receiving security updates. Puppet will continue to update legacy architecture components other than Kubernetes until 30 June 2022. If security advisories against Kubernetes 1.19 arise, the remediation path is to migrate to one of the newer architectures by following the instructions in Migrating PAM data to a new system.
Important: Before beginning the migration process from a legacy deployment you must upgrade to PAM version 1.62.0 with the force-reapply-addons flag included in the upgrade command. Find upgrade instructions at PAM legacy upgrades and PAM offline legacy upgrades.

30 November 2021 (Puppet Application Manager 1.56.0)

This release includes an upgrade of KOTS to version 1.56.0, which adds the following improvements:

Improved support bundles: Adds an option to upload a support bundle directly from Puppet Application Manager.
Improved troubleshooting: Adds detailed information on failing pods to the Troubleshoot tab.

6 October 2021 (Puppet Application Manager 1.52.1)

New in this release:

Improved statuses. More granular status levels are now available from the Application tab.
Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of Kubernetes to 1.19.15.
Other component upgrades. This version includes an upgrade of KOTS to version 1.52.1.

Resolved in this release:

Generating a support bundle no longer results in unusually high memory use.
Preflight check logs post to info level for progress messages and to error level for error messages.

25 August 2021 (Puppet Application Manager 1.49.0)

New in this release:

Component upgrades to address CVEs. To address various CVEs, this version includes an upgrade of Kubernetes to 1.19.13, an upgrade of Project Contour to 1.18.0, and an upgrade of Velero to 1.6.2.
Goldpinger. High availability architectures now include Goldpinger, which aids the debugging of network issues.
containerd upgrade. This version includes an upgrade of containerd to version 1.4.6, and removes the need to use the force-reapply-addons option when upgrading.
Other component upgrades. This version includes an upgrade of KOTS to version 1.49.0, an upgrade of ekco to 0.11.0, an upgrade of Prometheus to 0.49.0, and an upgrade of Rook to 1.5.12.

30 June 2021 (Puppet Application Manager 1.44.1)

New in this release:

Certificate auto-rotation for standalone architecture. Certificates are now automatically rotated for the Kubernetes API and Puppet Application Manager UI in the standalone architecture. With this change, certificate auto-rotation is now supported in all Puppet Application Manager architectures.
Rook upgrades. This version includes an upgrade of Rook in the high availability architecture to 1.5.11 and the version of Rook in the legacy architecture to 1.0.4-14.2.21. These upgrades address a vulnerability in Ceph components (CVE-2021-20288).
Prometheus upgrade. This version includes an upgrade of Prometheus in the high availability and legacy architectures to 0.48.1. Additionally, Prometheus disk usage is now limited in order to preserve the storage space required for the usage charts on the Application tab.
Other component upgrades. This version includes an upgrade of KOTS to version 1.44.1, an upgrade of Project Contour to version 1.15.1, and an upgrade of Weave to version 2.8.1.

Resolved in this release:

Snapshots can now successfully use the Other S3-Compatible Storage option as the storage destination.
To apply this update, add the force-reapply-addons option during upgrade. For example:
```
curl <url> | bash -s force-reapply-addons
```

26 May 2021

New in this release:

runC. The version of runC has been upgraded to v1.0.0-rc95 to address CVE-2021-30465.

Known issues in this release:

Running the KOTS installer with the airgap and kurl-registry-ip flags results in an error.
As a workaround (if you do not have any applications already installed in the cluster), delete the registry service, recreate the registry service IP and then re-run the installation script with the kurl-registry-ip flag.

10 May 2021 (Puppet Application Manager 1.40.0)

New in this release:

Distinct architectures for standalone and high availability deployments of the Puppet Application Manager platform. Standalone supports lower system requirements and resolves inherent flaws in using Ceph on a single node. High availability uses an updated version of Rook for faster, more reliable distributed storage.
Note: It is not possible currently to upgrade to these architectures from existing installations. However, migrating applications between them is on the roadmap for a future release.
The previous architecture is maintained as the legacy configuration. This version includes an upgrade of Kubernetes to 1.19.10; this upgrade process upgrades through Kubernetes 1.18, and happens on all nodes. It can take ~1 hour to do for a 3-node cluster, and requires confirmations during that period. It also includes an upgrade of Project Contour to version 1.14.1, adds Metrics Server 0.4.1, an upgrade of ekco to 0.10.1, and an upgrade of Prometheus to 2.26.0.
For more information on legacy upgrades, see PAM legacy upgrades.

15 April 2021 (Puppet Application Manager 1.38.0)

New in this release:

Snapshots. Puppet Application Manager now supports full (instance-level) snapshots, which can be used for application rollbacks and disaster recovery. For more information, see Backing up Puppet Application Manager using snapshots.
Component upgrades. This version includes an upgrade of KOTS to version 1.38.0.

17 February 2021 (Puppet Application Manager 1.29.3)

New in this release:

Support for Ubuntu 20.04. You can now run Puppet Application Manager on Ubuntu 20.04.
Component upgrades. This version includes an upgrade of Prometheus to version 2.22.1 and Prometheus Operator to version 0.44.1, an upgrade of KOTS to version 1.29.3, an upgrade of Project Contour to version 1.12.0, and an upgrade of ekco to version 0.10.0.

3 February 2021 (Puppet Application Manager 1.29.2)

New in this release:

Component upgrades. This version includes an upgrade of KOTS to version 1.29.2, an upgrade of Project Contour to version 1.11.0, and an upgrade of containerd to version 1.4.3.

Resolved in this release:

During their initial preflight checks, new installations now pull images successfully and no longer report a Failed to pull image error.

7 December 2020

New in this release:

Support for Red Hat Enterprise Linux (RHEL) 8 and CentOS 8. You can now run Puppet Application Manager on RHEL version 8 and CentOS version 8. To support this change, containerd is now used independently of Docker during the installation process.
Component upgrades. This version includes an upgrade of Kubernetes to version 1.17.13.