<!-- BEGIN MUNGE: GENERATED_TOC -->
- [v1.28.15](#v12815)
- [Downloads for v1.28.15](#downloads-for-v12815)
- [Source Code](#source-code)
- [Client Binaries](#client-binaries)
- [Server Binaries](#server-binaries)
- [Node Binaries](#node-binaries)
- [Container Images](#container-images)
- [Changelog since v1.28.14](#changelog-since-v12814)
- [Changes by Kind](#changes-by-kind)
- [Feature](#feature)
- [Bug or Regression](#bug-or-regression)
- [Other (Cleanup or Flake)](#other-cleanup-or-flake)
- [Dependencies](#dependencies)
- [Added](#added)
- [Changed](#changed)
- [Removed](#removed)
- [v1.28.14](#v12814)
- [Downloads for v1.28.14](#downloads-for-v12814)
- [Source Code](#source-code-1)
- [Client Binaries](#client-binaries-1)
- [Server Binaries](#server-binaries-1)
- [Node Binaries](#node-binaries-1)
- [Container Images](#container-images-1)
- [Changelog since v1.28.13](#changelog-since-v12813)
- [Changes by Kind](#changes-by-kind-1)
- [Feature](#feature-1)
- [Bug or Regression](#bug-or-regression-1)
- [Dependencies](#dependencies-1)
- [Added](#added-1)
- [Changed](#changed-1)
- [Removed](#removed-1)
- [v1.28.13](#v12813)
- [Downloads for v1.28.13](#downloads-for-v12813)
- [Source Code](#source-code-2)
- [Client Binaries](#client-binaries-2)
- [Server Binaries](#server-binaries-2)
- [Node Binaries](#node-binaries-2)
- [Container Images](#container-images-2)
- [Changelog since v1.28.12](#changelog-since-v12812)
- [Changes by Kind](#changes-by-kind-2)
- [API Change](#api-change)
- [Bug or Regression](#bug-or-regression-2)
- [Dependencies](#dependencies-2)
- [Added](#added-2)
- [Changed](#changed-2)
- [Removed](#removed-2)
- [v1.28.12](#v12812)
- [Downloads for v1.28.12](#downloads-for-v12812)
- [Source Code](#source-code-3)
- [Client Binaries](#client-binaries-3)
- [Server Binaries](#server-binaries-3)
- [Node Binaries](#node-binaries-3)
- [Container Images](#container-images-3)
- [Changelog since v1.28.11](#changelog-since-v12811)
- [Important Security Information](#important-security-information)
- [CVE-2024-5321: Incorrect permissions on Windows containers logs](#cve-2024-5321-incorrect-permissions-on-windows-containers-logs)
- [Changes by Kind](#changes-by-kind-3)
- [Feature](#feature-2)
- [Bug or Regression](#bug-or-regression-3)
- [Dependencies](#dependencies-3)
- [Added](#added-3)
- [Changed](#changed-3)
- [Removed](#removed-3)
- [v1.28.11](#v12811)
- [Downloads for v1.28.11](#downloads-for-v12811)
- [Source Code](#source-code-4)
- [Client Binaries](#client-binaries-4)
- [Server Binaries](#server-binaries-4)
- [Node Binaries](#node-binaries-4)
- [Container Images](#container-images-4)
- [Changelog since v1.28.10](#changelog-since-v12810)
- [Changes by Kind](#changes-by-kind-4)
- [API Change](#api-change-1)
- [Feature](#feature-3)
- [Bug or Regression](#bug-or-regression-4)
- [Dependencies](#dependencies-4)
- [Added](#added-4)
- [Changed](#changed-4)
- [Removed](#removed-4)
- [v1.28.10](#v12810)
- [Downloads for v1.28.10](#downloads-for-v12810)
- [Source Code](#source-code-5)
- [Client Binaries](#client-binaries-5)
- [Server Binaries](#server-binaries-5)
- [Node Binaries](#node-binaries-5)
- [Container Images](#container-images-5)
- [Changelog since v1.28.9](#changelog-since-v1289)
- [Changes by Kind](#changes-by-kind-5)
- [Bug or Regression](#bug-or-regression-5)
- [Other (Cleanup or Flake)](#other-cleanup-or-flake-1)
- [Dependencies](#dependencies-5)
- [Added](#added-5)
- [Changed](#changed-5)
- [Removed](#removed-5)
- [v1.28.9](#v1289)
- [Downloads for v1.28.9](#downloads-for-v1289)
- [Source Code](#source-code-6)
- [Client Binaries](#client-binaries-6)
- [Server Binaries](#server-binaries-6)
- [Node Binaries](#node-binaries-6)
- [Container Images](#container-images-6)
- [Changelog since v1.28.8](#changelog-since-v1288)
- [Important Security Information](#important-security-information-1)
- [CVE-2024-3177: Bypassing mountable secrets policy imposed by the ServiceAccount admission plugin](#cve-2024-3177-bypassing-mountable-secrets-policy-imposed-by-the-serviceaccount-admission-plugin)
- [Changes by Kind](#changes-by-kind-6)
- [Feature](#feature-4)
- [Bug or Regression](#bug-or-regression-6)
- [Dependencies](#dependencies-6)
- [Added](#added-6)
- [Changed](#changed-6)
- [Removed](#removed-6)
- [v1.28.8](#v1288)
- [Downloads for v1.28.8](#downloads-for-v1288)
- [Source Code](#source-code-7)
- [Client Binaries](#client-binaries-7)
- [Server Binaries](#server-binaries-7)
- [Node Binaries](#node-binaries-7)
- [Container Images](#container-images-7)
- [Changelog since v1.28.7](#changelog-since-v1287)
- [Changes by Kind](#changes-by-kind-7)
- [Feature](#feature-5)
- [Bug or Regression](#bug-or-regression-7)
- [Other (Cleanup or Flake)](#other-cleanup-or-flake-2)
- [Dependencies](#dependencies-7)
- [Added](#added-7)
- [Changed](#changed-7)
- [Removed](#removed-7)
- [v1.28.7](#v1287)
- [Downloads for v1.28.7](#downloads-for-v1287)
- [Source Code](#source-code-8)
- [Client Binaries](#client-binaries-8)
- [Server Binaries](#server-binaries-8)
- [Node Binaries](#node-binaries-8)
- [Container Images](#container-images-8)
- [Changelog since v1.28.6](#changelog-since-v1286)
- [Changes by Kind](#changes-by-kind-8)
- [Feature](#feature-6)
- [Bug or Regression](#bug-or-regression-8)
- [Dependencies](#dependencies-8)
- [Added](#added-8)
- [Changed](#changed-8)
- [Removed](#removed-8)
- [v1.28.6](#v1286)
- [Downloads for v1.28.6](#downloads-for-v1286)
- [Source Code](#source-code-9)
- [Client Binaries](#client-binaries-9)
- [Server Binaries](#server-binaries-9)
- [Node Binaries](#node-binaries-9)
- [Container Images](#container-images-9)
- [Changelog since v1.28.5](#changelog-since-v1285)
- [Changes by Kind](#changes-by-kind-9)
- [Feature](#feature-7)
- [Bug or Regression](#bug-or-regression-9)
- [Other (Cleanup or Flake)](#other-cleanup-or-flake-3)
- [Dependencies](#dependencies-9)
- [Added](#added-9)
- [Changed](#changed-9)
- [Removed](#removed-9)
- [v1.28.5](#v1285)
- [Downloads for v1.28.5](#downloads-for-v1285)
- [Source Code](#source-code-10)
- [Client Binaries](#client-binaries-10)
- [Server Binaries](#server-binaries-10)
- [Node Binaries](#node-binaries-10)
- [Container Images](#container-images-10)
- [Changelog since v1.28.4](#changelog-since-v1284)
- [Changes by Kind](#changes-by-kind-10)
- [Feature](#feature-8)
- [Bug or Regression](#bug-or-regression-10)
- [Other (Cleanup or Flake)](#other-cleanup-or-flake-4)
- [Dependencies](#dependencies-10)
- [Added](#added-10)
- [Changed](#changed-10)
- [Removed](#removed-10)
- [v1.28.4](#v1284)
- [Downloads for v1.28.4](#downloads-for-v1284)
- [Source Code](#source-code-11)
- [Client Binaries](#client-binaries-11)
- [Server Binaries](#server-binaries-11)
- [Node Binaries](#node-binaries-11)
- [Container Images](#container-images-11)
- [Changelog since v1.28.3](#changelog-since-v1283)
- [Important Security Information](#important-security-information-2)
- [CVE-2023-5528: Insufficient input sanitization in in-tree storage plugin leads to privilege escalation on Windows nodes](#cve-2023-5528-insufficient-input-sanitization-in-in-tree-storage-plugin-leads-to-privilege-escalation-on-windows-nodes)
- [Changes by Kind](#changes-by-kind-11)
- [API Change](#api-change-2)
- [Feature](#feature-9)
- [Bug or Regression](#bug-or-regression-11)
- [Dependencies](#dependencies-11)
- [Added](#added-11)
- [Changed](#changed-11)
- [Removed](#removed-11)
- [v1.28.3](#v1283)
- [Downloads for v1.28.3](#downloads-for-v1283)
- [Source Code](#source-code-12)
- [Client Binaries](#client-binaries-12)
- [Server Binaries](#server-binaries-12)
- [Node Binaries](#node-binaries-12)
- [Container Images](#container-images-12)
- [Changelog since v1.28.2](#changelog-since-v1282)
- [Changes by Kind](#changes-by-kind-12)
- [Feature](#feature-10)
- [Failing Test](#failing-test)
- [Bug or Regression](#bug-or-regression-12)
- [Other (Cleanup or Flake)](#other-cleanup-or-flake-5)
- [Dependencies](#dependencies-12)
- [Added](#added-12)
- [Changed](#changed-12)
- [Removed](#removed-12)
- [v1.28.2](#v1282)
- [Downloads for v1.28.2](#downloads-for-v1282)
- [Source Code](#source-code-13)
- [Client Binaries](#client-binaries-13)
- [Server Binaries](#server-binaries-13)
- [Node Binaries](#node-binaries-13)
- [Container Images](#container-images-13)
- [Changelog since v1.28.1](#changelog-since-v1281)
- [Changes by Kind](#changes-by-kind-13)
- [API Change](#api-change-3)
- [Feature](#feature-11)
- [Bug or Regression](#bug-or-regression-13)
- [Dependencies](#dependencies-13)
- [Added](#added-13)
- [Changed](#changed-13)
- [Removed](#removed-13)
- [v1.28.1](#v1281)
- [Downloads for v1.28.1](#downloads-for-v1281)
- [Source Code](#source-code-14)
- [Client Binaries](#client-binaries-14)
- [Server Binaries](#server-binaries-14)
- [Node Binaries](#node-binaries-14)
- [Container Images](#container-images-14)
- [Changelog since v1.28.0](#changelog-since-v1280)
- [Important Security Information](#important-security-information-3)
- [CVE-2023-3955: Insufficient input sanitization on Windows nodes leads to privilege escalation](#cve-2023-3955-insufficient-input-sanitization-on-windows-nodes-leads-to-privilege-escalation)
- [CVE-2023-3676: Insufficient input sanitization on Windows nodes leads to privilege escalation](#cve-2023-3676-insufficient-input-sanitization-on-windows-nodes-leads-to-privilege-escalation)
- [Changes by Kind](#changes-by-kind-14)
- [Other (Cleanup or Flake)](#other-cleanup-or-flake-6)
- [Dependencies](#dependencies-14)
- [Added](#added-14)
- [Changed](#changed-14)
- [Removed](#removed-14)
- [v1.28.0](#v1280)
- [Downloads for v1.28.0](#downloads-for-v1280)
- [Source Code](#source-code-15)
- [Client Binaries](#client-binaries-15)
- [Server Binaries](#server-binaries-15)
- [Node Binaries](#node-binaries-15)
- [Container Images](#container-images-15)
- [Changelog since v1.27.0](#changelog-since-v1270)
- [Urgent Upgrade Notes](#urgent-upgrade-notes)
- [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade)
- [Changes by Kind](#changes-by-kind-15)
- [Deprecation](#deprecation)
- [API Change](#api-change-4)
- [Feature](#feature-12)
- [Documentation](#documentation)
- [Failing Test](#failing-test-1)
- [Bug or Regression](#bug-or-regression-14)
- [Other (Cleanup or Flake)](#other-cleanup-or-flake-7)
- [Dependencies](#dependencies-15)
- [Added](#added-15)
- [Changed](#changed-15)
- [Removed](#removed-15)
- [v1.28.0-rc.1](#v1280-rc1)
- [Downloads for v1.28.0-rc.1](#downloads-for-v1280-rc1)
- [Source Code](#source-code-16)
- [Client Binaries](#client-binaries-16)
- [Server Binaries](#server-binaries-16)
- [Node Binaries](#node-binaries-16)
- [Container Images](#container-images-16)
- [Changelog since v1.28.0-rc.0](#changelog-since-v1280-rc0)
- [Changes by Kind](#changes-by-kind-16)
- [API Change](#api-change-5)
- [Feature](#feature-13)
- [Bug or Regression](#bug-or-regression-15)
- [Dependencies](#dependencies-16)
- [Added](#added-16)
- [Changed](#changed-16)
- [Removed](#removed-16)
- [v1.28.0-rc.0](#v1280-rc0)
- [Downloads for v1.28.0-rc.0](#downloads-for-v1280-rc0)
- [Source Code](#source-code-17)
- [Client Binaries](#client-binaries-17)
- [Server Binaries](#server-binaries-17)
- [Node Binaries](#node-binaries-17)
- [Container Images](#container-images-17)
- [Changelog since v1.28.0-beta.0](#changelog-since-v1280-beta0)
- [Changes by Kind](#changes-by-kind-17)
- [API Change](#api-change-6)
- [Feature](#feature-14)
- [Dependencies](#dependencies-17)
- [Added](#added-17)
- [Changed](#changed-17)
- [Removed](#removed-17)
- [v1.28.0-beta.0](#v1280-beta0)
- [Downloads for v1.28.0-beta.0](#downloads-for-v1280-beta0)
- [Source Code](#source-code-18)
- [Client Binaries](#client-binaries-18)
- [Server Binaries](#server-binaries-18)
- [Node Binaries](#node-binaries-18)
- [Container Images](#container-images-18)
- [Changelog since v1.28.0-alpha.4](#changelog-since-v1280-alpha4)
- [Changes by Kind](#changes-by-kind-18)
- [Deprecation](#deprecation-1)
- [API Change](#api-change-7)
- [Feature](#feature-15)
- [Failing Test](#failing-test-2)
- [Bug or Regression](#bug-or-regression-16)
- [Other (Cleanup or Flake)](#other-cleanup-or-flake-8)
- [Dependencies](#dependencies-18)
- [Added](#added-18)
- [Changed](#changed-18)
- [Removed](#removed-18)
- [v1.28.0-alpha.4](#v1280-alpha4)
- [Downloads for v1.28.0-alpha.4](#downloads-for-v1280-alpha4)
- [Source Code](#source-code-19)
- [Client Binaries](#client-binaries-19)
- [Server Binaries](#server-binaries-19)
- [Node Binaries](#node-binaries-19)
- [Container Images](#container-images-19)
- [Changelog since v1.28.0-alpha.3](#changelog-since-v1280-alpha3)
- [Urgent Upgrade Notes](#urgent-upgrade-notes-1)
- [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade-1)
- [Changes by Kind](#changes-by-kind-19)
- [Deprecation](#deprecation-2)
- [API Change](#api-change-8)
- [Feature](#feature-16)
- [Bug or Regression](#bug-or-regression-17)
- [Other (Cleanup or Flake)](#other-cleanup-or-flake-9)
- [Dependencies](#dependencies-19)
- [Added](#added-19)
- [Changed](#changed-19)
- [Removed](#removed-19)
- [v1.28.0-alpha.3](#v1280-alpha3)
- [Downloads for v1.28.0-alpha.3](#downloads-for-v1280-alpha3)
- [Source Code](#source-code-20)
- [Client Binaries](#client-binaries-20)
- [Server Binaries](#server-binaries-20)
- [Node Binaries](#node-binaries-20)
- [Container Images](#container-images-20)
- [Changelog since v1.28.0-alpha.2](#changelog-since-v1280-alpha2)
- [Changes by Kind](#changes-by-kind-20)
- [Deprecation](#deprecation-3)
- [API Change](#api-change-9)
- [Feature](#feature-17)
- [Bug or Regression](#bug-or-regression-18)
- [Other (Cleanup or Flake)](#other-cleanup-or-flake-10)
- [Dependencies](#dependencies-20)
- [Added](#added-20)
- [Changed](#changed-20)
- [Removed](#removed-20)
- [v1.28.0-alpha.2](#v1280-alpha2)
- [Downloads for v1.28.0-alpha.2](#downloads-for-v1280-alpha2)
- [Source Code](#source-code-21)
- [Client Binaries](#client-binaries-21)
- [Server Binaries](#server-binaries-21)
- [Node Binaries](#node-binaries-21)
- [Container Images](#container-images-21)
- [Changelog since v1.28.0-alpha.1](#changelog-since-v1280-alpha1)
- [Urgent Upgrade Notes](#urgent-upgrade-notes-2)
- [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade-2)
- [Changes by Kind](#changes-by-kind-21)
- [Feature](#feature-18)
- [Bug or Regression](#bug-or-regression-19)
- [Other (Cleanup or Flake)](#other-cleanup-or-flake-11)
- [Dependencies](#dependencies-21)
- [Added](#added-21)
- [Changed](#changed-21)
- [Removed](#removed-21)
- [v1.28.0-alpha.1](#v1280-alpha1)
- [Downloads for v1.28.0-alpha.1](#downloads-for-v1280-alpha1)
- [Source Code](#source-code-22)
- [Client Binaries](#client-binaries-22)
- [Server Binaries](#server-binaries-22)
- [Node Binaries](#node-binaries-22)
- [Container Images](#container-images-22)
- [Changelog since v1.27.0](#changelog-since-v1270-1)
- [Changes by Kind](#changes-by-kind-22)
- [Deprecation](#deprecation-4)
- [API Change](#api-change-10)
- [Feature](#feature-19)
- [Documentation](#documentation-1)
- [Failing Test](#failing-test-3)
- [Bug or Regression](#bug-or-regression-20)
- [Other (Cleanup or Flake)](#other-cleanup-or-flake-12)
- [Dependencies](#dependencies-22)
- [Added](#added-22)
- [Changed](#changed-22)
- [Removed](#removed-22)
<!-- END MUNGE: GENERATED_TOC -->
# v1.28.15
## Downloads for v1.28.15
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes.tar.gz) | 8e6efde16d8d5afac89242b7312bdf9c44aaeedce72549cc8cf69b26fe351d460e6ad2155fb74ba65208430c1ee26e22036639a412b5a027f9ba30f3176050c6
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-src.tar.gz) | 22396ec27d684605e58651cc205c3b001d7294c37afb204b1ca28efc3812a28ebd567862d57dc475c3b50f23fd93f3bbf59d244cab141002ee63579f1ad09fda
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-client-darwin-amd64.tar.gz) | 7c4ef89a195bab81073ecf79e16a459eece07ca3373707e1f5f5fad7fc0a1815cf32adc2bde81e31ae62c3065e77b8cfc380ad5f6d1e69aa0086258d4c872c87
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-client-darwin-arm64.tar.gz) | 8f17cf6057debc5d8b6371148f8f5dae5a8e38849a8f5db90161a7e408e605a6d84f42c22fb8e0f9e106a0c8ba3b219b76d76c086d1a53656d6e2fc7890ff32d
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-client-linux-386.tar.gz) | 871f52337f29b16b031f96fa5b5e8733253fa00b7f2e83da711faba50561765bb0e2e12ccda1fd35685dd4b6ec85d2865209571c104d40ed948570a7fd8cf565
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-client-linux-amd64.tar.gz) | a5dd50d3151ec3624f7323ffb8efb7c8ab667f5aa9119661441126b03a9d3add8a1d6317b840970c3c4045f4e9ac4611e8bb9e929184712588b760f7b4bd1738
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-client-linux-arm.tar.gz) | c0733ab134b9a1fb011cfcad34ca832b4a10316b270bfa1a1a245ea93b6611ebc83611a62ef3c6e21107492586f361596e4b9a44010d2aea7a925cb511d6d6f4
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-client-linux-arm64.tar.gz) | b7308b84a398ab2cbfa218fe6b4c96bbb08c00beb89b285738e0ff0be79c84f7eeb23fb340af6f1eb2be36c5a361430c168647bd8539dc9043ea8c2ab293213a
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-client-linux-ppc64le.tar.gz) | 30f2fd3e3bbaf0b6052f4b5ecc52ed3a0884f9fcd96e97aac8c58691ccc7ca9806a17e9b86ceba187589ced0ac519442df84172666b1fa66da56e9f885c1526d
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-client-linux-s390x.tar.gz) | 4f696c6679fdecf7748babb69ed60fbe8e7f7df49ba76f99db9385f0c1257ba2176db23652e5f0ff6a0e8b420263b80816ef286c834b7ffd059206e3424b37f7
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-client-windows-386.tar.gz) | 9ee1ec80139ac66070737ac92883c893da35b9287bdc7339a2636fdcf2a6e6ea62a30b4a461f56750105355d66e852d076be23c3eed908231edbcf7972474f17
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-client-windows-amd64.tar.gz) | 75f856c5872f955204262a905a33e8082beb43d477f30efe50b27c7a292158a755ec5b5d1db759e890a2ed88443ad54a380b83e01502a71fe6c5be2dab1410f6
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-client-windows-arm64.tar.gz) | 2e9640b0ba6280fddf2cf08964d014247cded80c0825506f5db3598b4623528446f006f031476cf382997f93bbebfff88dfa643776170607d8ed7d51d54373fd
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-server-linux-amd64.tar.gz) | 62c6191ffad2dfd84e1e89026f47cadafab05ceec6f3535e6ba124edd11aaba98ab89636b8e5d02522bc08f63249b50b636f5ee208827425ba18cfe63e280019
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-server-linux-arm64.tar.gz) | 0f0b1ed46ead1dc66cc5eafa8787df904b02cc2010c4f60e90490b478a29f8a20b87059553789ff4e4aac020ae51881f0bd3c75caa3a92daa1bd673ce9397ea6
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-server-linux-ppc64le.tar.gz) | 6a779633e62c354893d6f4e0a17d88d1de24ae1bf7b0894895ba757eef396b55d57b1441147a771583a90b460534248ea751754030f49e53823396d22dbfdca6
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-server-linux-s390x.tar.gz) | e910cafa6ebef3ded936710ddd7b770303bc73f87dcfa70a055711b06c9913ec16b9e12e6c864b3b44820e3381bf1222c6f138921daa2547bbe8fd5aab710fd1
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-node-linux-amd64.tar.gz) | ffa81fea3e829ed5190b6b58dc681aa71ce025411aea01c91110ffd173577318791a00b36d813dc47a6ec62a386e812df07876bc14599765345825ed7fac5620
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-node-linux-arm64.tar.gz) | 89a780314e665384c3398bca810eabb7e6babccec362b234c5cc0da41a54a97579dbb1165b164f99fea77dcdd69119800dc5477e94a6c201398e87048733509b
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-node-linux-ppc64le.tar.gz) | 8d7a9092e7544e41036d270fc8c4c41e36af02352d5ef299152b44244b7ac45a71fb2fe05b5ef1176b4cc8b7ee885964f981876ddeafc218ce1f0cd0679a24a8
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-node-linux-s390x.tar.gz) | ee36f33ba3a086825f11f68414062ba09bfcda47505f047af697572e83cec2211d5d72abe7d310cc2560815fddd1e433b6f43712caf63d4d2b1289e5a405af4f
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.15/kubernetes-node-windows-amd64.tar.gz) | 22544d1275bb90f888f58aa76b0f2d1aee2a1f547b5bc4c56808b979948cc48d57873c2d146536539fcf33a45a1f4d9df933e6721ce14f931aebe2f2155572e8
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.15](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.15](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.15](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.15](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.15](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.15](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x)
## Changelog since v1.28.14
## Changes by Kind
### Feature
- Kubernetes is now built with go 1.22.7 ([#127605](https://github.com/kubernetes/kubernetes/pull/127605), [@haitch](https://github.com/haitch)) [SIG Release and Testing]
- Kubernetes is now built with go 1.22.8 ([#128129](https://github.com/kubernetes/kubernetes/pull/128129), [@haitch](https://github.com/haitch)) [SIG Release and Testing]
### Bug or Regression
- Ensure daemonset controller to count old unhealthy pods towards max unavailable budget ([#127776](https://github.com/kubernetes/kubernetes/pull/127776), [@ncdc](https://github.com/ncdc)) [SIG Apps]
- Fix a bug on the endpoints controller that does not reconcile the Endpoint object after this is truncated (it gets more than 1000 endpoints addresses) ([#127417](https://github.com/kubernetes/kubernetes/pull/127417), [@aojea](https://github.com/aojea)) [SIG Apps, Network and Testing]
- Kubeadm: fix wrong member list reported when removing an etcd member ([#127963](https://github.com/kubernetes/kubernetes/pull/127963), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle]
- Kubeadm: when adding new control plane nodes with "kubeamd join", ensure that the etcd member addition is performed only if a given member URL does not already exist in the list of members. Similarly, on "kubeadm reset" only remove an etcd member if its ID exists. ([#127622](https://github.com/kubernetes/kubernetes/pull/127622), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle]
### Other (Cleanup or Flake)
- Kubeadm: removed `socat` and `ebtables` from kubeadm preflight checks ([#127416](https://github.com/kubernetes/kubernetes/pull/127416), [@saschagrunert](https://github.com/saschagrunert)) [SIG Cluster Lifecycle]
## Dependencies
### Added
_Nothing has changed._
### Changed
_Nothing has changed._
### Removed
_Nothing has changed._
# v1.28.14
## Downloads for v1.28.14
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes.tar.gz) | bcc95b06cfeeb358a32863a0b576b9ac9bd7e4ad47836efd024f2075a829cb7f3bcc0511f59e4a1b6bb07abf2d0cb2f49cc6e399a32839d78abaa2d335bc557a
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-src.tar.gz) | f51815cfd553dc80e678a723ba7eeb996dfd5a9c65180e59f948128a434d401f1263011186a8d6890025a22cc8a61af91c76e233277b61222069923f5a5a36a5
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-client-darwin-amd64.tar.gz) | 51b77913295cd4376a6e7e73f7f26e114ca75577ee2540fe760734b4e2a64c9b01452cb4dac26196dea848158f224635186fa5d63dcbdddbef3a6fc239934320
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-client-darwin-arm64.tar.gz) | e025ffe193a074ae5b90d1c09d373a804835b889932cb88b27c85fa5f5d8f2b142ee3d38b5648280a0b3407380450a91ddf8a3a5920fd04e9fee7b1af3979765
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-client-linux-386.tar.gz) | be0fd5ae7137c39584bc503b3748c48c2069c002253a07e5cb5704893e131daa821d3f3adaa452a8f3047f1c45d117ad7f24b6ac3c1ef6bddee51744510760a4
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-client-linux-amd64.tar.gz) | 85b0c2856944afa22c134879b5845c2dc8e86d5e18c25ae2a248bae0fc1dc710654ff6a417571e02cae12ccef948eb4da2e6e17905e90b8709b4bf138bc63889
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-client-linux-arm.tar.gz) | 729dc75f8d8b55c754f9927fac470365617486f579acf1d666b2b23edafbb3a48d9c0d5b97f08a6ab44d3123c80d664dd44ce7bb17aa5506f367a732f9476712
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-client-linux-arm64.tar.gz) | ac802a2ebac4640639ba2b1208df3eb7a76c0117c1f78bfe81e991abebf5c91b38d455c2de9b3f4acbfab453cd897a325c7eaf2557dbffb8cb5bac42212711de
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-client-linux-ppc64le.tar.gz) | 05935726d78c0f2ed531306cab38120009c89082fd30d4f72a309720f7e142662cdb0237fe47ad33905cae6855c4152f09fd9b74cdfdd6fa231367e82cf91f4e
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-client-linux-s390x.tar.gz) | 7da906efc6491fc9a3d98b635195774980b57f74c012fc886beeaa3aab77bc248265831bb8eb2d0393e157e3c8241c9f38677dad5f7019b54e48b9066ced1acc
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-client-windows-386.tar.gz) | 4c2db2367d4d5f153081d5138b18c8053955e104f4bf28971129a42170ca3494a057b55070258c5d44b57e6b9f976e76f3d463f0967084da1015a4c6d1c8234f
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-client-windows-amd64.tar.gz) | f8a705c42404d1f616ace4bcc99ce7b82bf41e7846110c25f2e45af64bd206ecf9dd706c03f24bafd3b5e53a8a4fef506ea7c63091f75e0fee9eb6e4212215a5
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-client-windows-arm64.tar.gz) | 373e684fccb055a1bfba71c03d6642305108da08886c49f699190b09267533afec3cac1af27f7d667b2a4ca7e2efb064a178be925c60da2fb732d890753a4b12
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-server-linux-amd64.tar.gz) | 9d44d672d72c35cc8bba69d016afc1c2df370397d0311073d479f8432728f3e619723326b0b2cdbc9fd3f81ad53d0d58e0c020933a8c5c20a5d765b67b07f72d
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-server-linux-arm64.tar.gz) | 9d73a58605f0eaece9fd68bdccad0f53627301711cae6d22d3e6f45c38fefa308fb96014ce69e1f95cb17028623e037573de7f428fd7003304b385867eb7c2e0
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-server-linux-ppc64le.tar.gz) | 236596cdf3d546b9b9a0c061fba87867c8e78fe33acafe9dcc3309a280bb4f640113b9dbb8062dc73c2db237cdb4414761a793fc56bbbbd34a159f1c7c4b0746
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-server-linux-s390x.tar.gz) | dc324a1d966f2a29592640622ad4a43b98aea990761e863a336fe171910347eccb06a914d0fa2512c456d68c48a6c88f22970e60de6249e9629f0293525d7dec
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-node-linux-amd64.tar.gz) | 2e581581239da10ed36806fd7dd6bb3b0d19d664c4327bb1b2ac1bbdea2d080393a0e16a25eb272c3442131ce29b6351279e6617ca834bf17efc1eea99eb3b49
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-node-linux-arm64.tar.gz) | 8b1221d9cebabc54ece385eceefd0e8f97fb17bcae3d42b6e217bd93ce200110a0f0d9869fdd4adbd5d36f75b7d4a52b2a5fffe25a5706bd38e6bdd3d04d9da6
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-node-linux-ppc64le.tar.gz) | 4fbd541f9ad568889d6941ec14c0df8299db1c5415f4e173c0932ed2f971cffafb2ceb58fab6e2628bfb1acb2707d72badca5fe80a33c02f800c67f21a28e513
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-node-linux-s390x.tar.gz) | 3150d68910c611a034392eab6330f17af749f2068a6287ef59a8818366f0634903174cf52559b4755a928c386a10ae48850f63ead01707ed5187723c3c8c12b6
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.14/kubernetes-node-windows-amd64.tar.gz) | 8fcab760be5391b4c4605eab078d22ed2f18e02745e5f272d7c8c90524e7742db2bc427d4dfc966fd93bd15f0d643222e1f72e50ca34951e75e7809f926ed439
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.14](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.14](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.14](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.14](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.14](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.14](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x)
## Changelog since v1.28.13
## Changes by Kind
### Feature
- Kubernetes is now built with go 1.22.6 ([#126973](https://github.com/kubernetes/kubernetes/pull/126973), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing]
### Bug or Regression
- Fix a scheduler preemption issue where the victim pod was not deleted due to incorrect status patching. This issue occurred when the preemptor and victim pods had different QoS classes in their status, causing the preemption to fail entirely. ([#126695](https://github.com/kubernetes/kubernetes/pull/126695), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling]
- Fix race condition in kube-proxy initialization that could blackhole UDP traffic to service VIP. ([#126692](https://github.com/kubernetes/kubernetes/pull/126692), [@wedaly](https://github.com/wedaly)) [SIG Network]
- Terminated Pods on a node will not be re-admitted on kubelet restart. This fixes the problem of Completed Pods awaiting for the finalizer marked as Failed after the kubelet restart. ([#127210](https://github.com/kubernetes/kubernetes/pull/127210), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Node and Testing]
## Dependencies
### Added
_Nothing has changed._
### Changed
_Nothing has changed._
### Removed
_Nothing has changed._
# v1.28.13
## Downloads for v1.28.13
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes.tar.gz) | 726de1abc9ecb5063d82e87cc9fcb6aeb6d07738febfdda980409bc735babd0fc52c9759cccb23d7f6f5e2abff66cf6780f39a8a30bfa7e77d05c7a0e7773634
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-src.tar.gz) | 1575a17efe3e356ac07947f8b635723c85ad798b362e2e6dcfca5ac90a36b11a96ba424475ee33db609f08ab0ae201ff238f9df69ea34202bb276ea0f4389daf
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-client-darwin-amd64.tar.gz) | 93a9fd8cf7a17f8f283e716ac84bd69f6822a0c7633b84923969152ebb620dd046449ed6c085b3b463a68163d6303e6fb4b5c120a8b84d7a9df960894333f005
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-client-darwin-arm64.tar.gz) | 87db4b5ebbc8d94c9789d14f5db936f8cadffb402ab4bb94e3f2a8ed1c484942ef894ff7479a9cb7337105ad6a631d1e8c98cef71ae89c3c3b0efa8aa6d20ab7
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-client-linux-386.tar.gz) | 11170705ab0a5fcdee50ff5e5f8c021c496984556732ddf95148e9705d89362f863552b48630dd140f4ca068255fe949189bc4acd8b5ae207a2c825c039884ee
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-client-linux-amd64.tar.gz) | bb281c33a0eedd469415ccc8b6fdf1bd296dfbb2cdba3fd4d47713c718c94494defb5dab38076db96722b3e7b3b7b6179c0f5b1ec70d66e7a8761f96a144ff17
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-client-linux-arm.tar.gz) | 002640f2b3feca8863ca873eb1e4d34f6700e0add6a76c27f5b55bb60a8923a7f84175e62f7588ce8a0168dfb375a2459df2f287bf1eb8745232f66c578a076f
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-client-linux-arm64.tar.gz) | 3b5798c6f31945e6583ca90b3ca7e67d74be7c223033d42405d8868e2dd325a218ec9ac73615fb0f114fbaa60bfaf877b3fe788be5d582d25b60f5ad19b28b15
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-client-linux-ppc64le.tar.gz) | 0d28aa8028987c6c7baf68ddad4ab496752d705d265e557f572b408f59cfeca185fc75693798b6f1bcd4e9cf6d0078bfc1e01710bebd2a9b479b02110d181330
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-client-linux-s390x.tar.gz) | 5a3f30d5c4672b2dc17882d382d2362ed124a08bf792d849516e3e8361f5d938adec9b1993edadb92f39d3d44f40a50746e884d932035928dd3304946ac545f4
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-client-windows-386.tar.gz) | ce32c6bf9e2909a9c5df04dab021da8bf402fe66294c8104f7009c47378dda9d2aeef70c80c8cbe966d6b4e019052866c66d2b48cd713c0824ae7d6d9ad53634
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-client-windows-amd64.tar.gz) | 7d65ac8e90d85aff503ae2670589d08627c643df35d9114e244531e18f01905d239edf7fcb7461519b0245c8a8a86a9875d045146ce17586ea71561a15b26ceb
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-client-windows-arm64.tar.gz) | af6249fc4868e879eb0dcbe4c44ed16f26958c5c2327008fbc60c56fad2875db69006afa6d50e4766875b40fc4e36df5e45d83d56e55a1e61b19fbc0d1dbe4b5
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-server-linux-amd64.tar.gz) | f39475348cd6887a94bfe6ffec0ceffd73e9af17d90a48b7a2d7da7371be7a4b58028b3edcf0a29b765c1dca5247900ac496920e7e1700997bb03b1813eec9d0
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-server-linux-arm64.tar.gz) | 3686940d035cf4121eb9ac5fb48a9d4a93ffd93d9bec9b9df2f96881206a5ec4f6ab8498f8858c7203de65c62ee5103d6c10d78d0877ccf254ed1dd6cca9e343
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-server-linux-ppc64le.tar.gz) | 0c1eba26869338906eb61a0c8cf6f52b4304511c10b29df7cca068decf553f7cec8ede38ac21d3a760d64905b384184b5d9ca1bdf82484042c85ebd1aed48beb
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-server-linux-s390x.tar.gz) | ba0169a6762a4d1a3c1572de0e6a373a7d3f967a7e44865a70b1028b7a523d00912a7e23aa682414b46c5b0fab05c22b8c9afdfb71ff7dff7394e490ca54942b
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-node-linux-amd64.tar.gz) | 668dcb5cef22b64728fcf1c4f1f8305b4a63a776b14d6876de64e3a5b301f3954c3581e1649fb50dfb331677e1b9a4417c67b4c6faebeb1889fffa4b33987a60
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-node-linux-arm64.tar.gz) | 547ca9c6a08c3220074e8edf0bdad72c0659dda2e34ea8d2ac4cad9dd524fa0c80b34651db3f8c6bcbf9b868eab527d6137b09a90ca9118bf9c14da843149251
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-node-linux-ppc64le.tar.gz) | 18e55651fe396320d9582872d04f7a9f5d92d873e9a6049edf3b2802fa2ae1d883849f200b7e0c30a88d2449c7b4600caa02c4e242e9c500e260c05af19218c8
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-node-linux-s390x.tar.gz) | 05418f0b4685eab307208b53a6545cc9acfc3e1d31897d83c1c06efc65d104427343c2d08106ef1b0f35336d334759c5b58743318570edbda05f1f64281c937f
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.13/kubernetes-node-windows-amd64.tar.gz) | 127fc7a0c505dab827e00d4cf63491187be9ad6c292516e1913f3bbd38dd6a5f8a3fcee2cb5eb784ae8fb3ac10362b36a7f3aff62d85d3b693c91a15c6502100
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.13](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.13](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.13](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.13](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.13](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.13](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x)
## Changelog since v1.28.12
## Changes by Kind
### API Change
- Fixed a bug in the API server where empty collections of ValidatingAdmissionPolicies did not have an `items` field. ([#126159](https://github.com/kubernetes/kubernetes/pull/126159), [@xyz-li](https://github.com/xyz-li)) [SIG API Machinery]
- Use omitempty for optional Job Pod Failure Policy fields ([#126046](https://github.com/kubernetes/kubernetes/pull/126046), [@mimowo](https://github.com/mimowo)) [SIG Apps]
### Bug or Regression
- Fix the bug where PodIP field is temporarily removed for a terminal pod ([#125404](https://github.com/kubernetes/kubernetes/pull/125404), [@mimowo](https://github.com/mimowo)) [SIG Node and Testing]
- Fixed a bug in ValidatingAdmissionPolicy that caused policies which were using CRD parameters to fail to synchronize ([#123003](https://github.com/kubernetes/kubernetes/pull/123003), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing]
- Kube-apiserver: fixes a 1.27+ regression watching a single namespace via the deprecated /api/v1/watch/namespaces/$name endpoint where watch events were not delivered after the watch was established ([#126150](https://github.com/kubernetes/kubernetes/pull/126150), [@xyz-li](https://github.com/xyz-li)) [SIG API Machinery and Testing]
- Kube-apiserver: fixes a potential crash serving CustomResourceDefinitions that combine an invalid schema and CEL validation rules. ([#126167](https://github.com/kubernetes/kubernetes/pull/126167), [@cici37](https://github.com/cici37)) [SIG API Machinery and Testing]
- Mount-utils: treat syscall.ENODEV as corrupted mount ([#126174](https://github.com/kubernetes/kubernetes/pull/126174), [@dobsonj](https://github.com/dobsonj)) [SIG Storage]
- StatefulSet autodelete will respect controlling owners on PVC claims as described in https://github.com/kubernetes/enhancements/pull/4375 ([#126581](https://github.com/kubernetes/kubernetes/pull/126581), [@mattcary](https://github.com/mattcary)) [SIG Apps, Storage and Testing]
- Stop using wmic on Windows to get uuid in the kubelet ([#126012](https://github.com/kubernetes/kubernetes/pull/126012), [@marosset](https://github.com/marosset)) [SIG Node and Windows]
## Dependencies
### Added
_Nothing has changed._
### Changed
_Nothing has changed._
### Removed
_Nothing has changed._
# v1.28.12
## Downloads for v1.28.12
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes.tar.gz) | f8bd6e65af7abffde8ba359254ddc167c2945464e989f56ca52a9ec75357fecd64f1c149c2f44be15d737221d9823acf57453d0a498f9232fee84a5dedb334f6
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-src.tar.gz) | b9e0d50b7726886013dab700aa55f3f63498f0e32608643ba244efd0969fcd579c67f1efe8254d9515c7a8e01b95f1bde145bf0cd3ee27ea8daacffa3100e6bf
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-client-darwin-amd64.tar.gz) | ad03e83915a45c9099f3609cef0421f8828b124a633cddb2bac8e9e0f1533a8dd5e4763cf4384c69b73359f0fa0bf4c981a36d2f8919a74f3bd9e117ecb0ebe1
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-client-darwin-arm64.tar.gz) | 654248ad1a3b684ec6876cd69bea449a7fdeef8711334c71110d1f6693ec8259f5a7c0aa7f6ece1a853be698ee529b1f0a6fcc8ef5e70562f221a71d41981747
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-client-linux-386.tar.gz) | d157ed5c78b0f2c234fbdb52ce0be3e6fc2bed12c6c6d3cad5bd4d83b46b7fbb9efd0e4c02624b9915f48f79174e75a18d6cc59d02cfe8ee9d4a9ed7f84aac02
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-client-linux-amd64.tar.gz) | 096468159d29ea3bde4ae798eab257e967bcfd4837d5cd31ded074482d1338b4c3faaf6f41a61406c0ba58246104ca3b1847772479373055500ee442f8b5b375
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-client-linux-arm.tar.gz) | 92e4bdeb0e44a9f92d43443383f0415298ad80c1aaf07d7aaf8b7cd68442ba2d841dd394f549c94d1dab45263369ee45b44de419456d6fa87adfac9a9c06df18
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-client-linux-arm64.tar.gz) | de0131addf3719195a99d5efc8bde70229c18ac66c6b69dbf53131dc1719264e8570ef5c7fe4fd1da01d32fde8ad775ac294f0fd36051884c7ee8f4c9e749905
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-client-linux-ppc64le.tar.gz) | 79ccb60e7f23cfb1b6c4c6f224336df1718a9a36294906a14bb442062a4bacf23fcc166b1a0799d0b6ff8e0abc2648617942b4d2bd7a66953f7758569156a7bd
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-client-linux-s390x.tar.gz) | a82d50c28db1c00088c12c1dd69e08f30b691bf6b833f5038be73b1b23eb70c3c1d1ceaf4093efaa7e0b4dbe80c9f6d258e803e1e7df6e85b289a28496a9e05e
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-client-windows-386.tar.gz) | 70ac0d538e9122cb523da5c6455c4c1ac964fe8bcaae5a0ded8af7eb155550a881c334f4cda5dd169832ebe75fec4559085d942515f2bd2a88ea4ec20773babc
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-client-windows-amd64.tar.gz) | afbfe1cb54926997eb66d015d37bf4804f7d1af85a8453ddf1b44000b99d77140f502f313c330b5b9261b55aed6b046b02cbde898c51a443e4c859f40e9ec339
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-client-windows-arm64.tar.gz) | 71036bc67f853cf8477f9231a316f7e10414d4ced28bdad301919e4dff87db6c1870049e141299cb3ad4e085d5bfe401bc598246aeff4ca6305c42b2a6ebca54
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-server-linux-amd64.tar.gz) | bd63f5561c198c3e36b076c2fac01d5dd1c70d2d82ced73da38b6a8bca6f4cc6b3c1dd0692206f02d2425009c084984ec692f11d88d2d41d538af48be259f10d
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-server-linux-arm64.tar.gz) | 187d0e2b6bb7bfcf7a675df934bf7e4b66547a357faafe26a9ae343983fff6a644b0303e12c0095ffb6c501c4dc2e1b1fbf9d03a019a5994f5e3e8fb1333a0c8
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-server-linux-ppc64le.tar.gz) | 165ff3f9e187b044538c7c5dc5811ac455fc5bb9ea31e805f3914d84e927b84c80241e2d5a1e42b66f9038dcac347d79b89a2ed6c4157bbf92aedfae9fec2679
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-server-linux-s390x.tar.gz) | 8a39aafaed3de50c25091c67dad8e5b494b7b6692ba17c516105e807a6213421b24573e3b630b50a5e7cb87a0fd12eaf69ac8dece3d62f86fdab84e4be3f8958
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-node-linux-amd64.tar.gz) | 62b418226adbe1dd21bee280a9e49eefa55d4b064c010aedb8e0cca5b13ba8ad4bd3a4f774b036fe5cdd6ae70e56fae4ed5f018feb509b87b2effb28a1074774
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-node-linux-arm64.tar.gz) | 066b5b7b0c388256f2fedda56ebc16e5cc4d3665f4d585acd9728c7cff70aca2778fc7975a501d04c136d137fa1cf3a310cf14a0f420ff21f8d59d4ea6c13e6b
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-node-linux-ppc64le.tar.gz) | a375fac0c44a008176c3dc88bd95d7f7480db22be7ede0bada8d003e9732b1a74018d98916676e82fb01a23b644bebb1d94c8cf3adfe11be07475c245390d96c
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-node-linux-s390x.tar.gz) | 3e4c7b70f2cdf8a399afd9b6d4008904c5ab048ad09dd04d51217e79dbee9c55d6cc1dcfb1b53ebe292a166c05f5fb119d73f1b7ae4044cb80df7113395032fb
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.12/kubernetes-node-windows-amd64.tar.gz) | 646c6cf360506e70d7a43739a0f3295c60dc5dccd40f017cbbe02961111fa94ee4d8cf0fae24e11720dfe8aeaef7edb5f43718485a28a227b38d485558e6aa9e
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.12](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.12](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.12](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.12](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.12](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.12](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x)
## Changelog since v1.28.11
## Important Security Information
This release contains changes that address the following vulnerabilities:
### CVE-2024-5321: Incorrect permissions on Windows containers logs
A security issue was discovered in Kubernetes clusters with Windows nodes
where BUILTIN\Users may be able to read container logs and NT
AUTHORITY\Authenticated Users may be able to modify container logs.
**Affected Versions**:
- kubelet <= 1.27.15
- kubelet <= 1.28.11
- kubelet <= 1.29.6
- kubelet <= 1.30.2
**Fixed Versions**:
- kubelet 1.27.16
- kubelet 1.28.12
- kubelet 1.29.7
- kubelet 1.30.3
This vulnerability was reported by Paulo Gomes @pjbgf from SUSE.
**CVSS Rating:** Medium (6.1) [CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:L/A:N](https://www.first.org/cvss/calculator/3.1#CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:L/A:N)
## Changes by Kind
### Feature
- Kubernetes is now built with go 1.22.4 ([#125670](https://github.com/kubernetes/kubernetes/pull/125670), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing]
- Kubernetes is now built with go 1.22.5 ([#125897](https://github.com/kubernetes/kubernetes/pull/125897), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing]
### Bug or Regression
- Add `/sys/devices/virtual/powercap` to default masked paths. It avoids the potential security risk that the ability to read these files may offer a power-based sidechannel attack against any workloads running on the same kernel. ([#125970](https://github.com/kubernetes/kubernetes/pull/125970), [@carlory](https://github.com/carlory)) [SIG Node]
- Fix a bug that Pods could stuck in the unschedulable pod pool
if they're rejected by PreEnqueue plugins that could change its result by a change in resources apart from Pods.
DRA plugin is the only plugin that meets the criteria of the bug in in-tree,
and hence if you have `DynamicResourceAllocation` feature flag enabled,
your DRA Pods could be affected by this bug. ([#125645](https://github.com/kubernetes/kubernetes/pull/125645), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling]
- Fix endpoints status out-of-sync when the pod state changes rapidly ([#125675](https://github.com/kubernetes/kubernetes/pull/125675), [@tnqn](https://github.com/tnqn)) [SIG Apps, Network and Testing]
- For statically provisioned PVs, if its volume source is CSI type or it has migrated annotation, when it's deleted, the PersisentVolume controller won't changes its phase to the Failed state.
With this patch, the external provisioner can remove the finalizer in next reconcile loop. Unfortunately if the provious existing pv has the Failed state, this patch won't take effort. It requires users to remove finalizer. ([#126045](https://github.com/kubernetes/kubernetes/pull/126045), [@carlory](https://github.com/carlory)) [SIG Apps and Storage]
- NONE ([#126127](https://github.com/kubernetes/kubernetes/pull/126127), [@cji](https://github.com/cji)) [SIG Node and Windows]
## Dependencies
### Added
_Nothing has changed._
### Changed
_Nothing has changed._
### Removed
_Nothing has changed._
# v1.28.11
## Downloads for v1.28.11
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes.tar.gz) | 22bd5efcd44229aea2e0d722042b0b21a05c2d0384ab7d74d983769b378e260cab6c894d685a9b04568a0ccccb098fd0a5897fad3792d429bf663f06c438f3ce
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-src.tar.gz) | cba35f33b44f2a6fb25cbfdafb76e5e060e346e045cb4e6d6ee277ec511194f913703e1a6af12841d4886accee47b282d83a54267e41d6894696dcb1cd698d10
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-client-darwin-amd64.tar.gz) | 98b74fea78336f1fcb06166c6c9f87f86a793871aaacc5746521373d2a15273909246288d9abc65b6a6316d6ec1076a6c11f5b80e22731e31ba70bde4dca789c
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-client-darwin-arm64.tar.gz) | e178945f1b6edf6d5aae533f4701bccc87aab1dd2c4defbf768c667b204914c380d1e1ab1c7e9979968d211222a78e05344081011ac388296f2d014c0ddb6a39
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-client-linux-386.tar.gz) | a168e4dcb5b8b79ce549ddc535fcf00d17983ab85d9863d3ccec0383642c78472eff4df7b73f1ac5e35f96d2a37aed9734c4e0b2dcc0ab93cb1e56a2b1136eba
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-client-linux-amd64.tar.gz) | ea5caa7f7d15cedda291d878a9863db4c0edd3e65ede396eaf4001e8d0fc305ad7891e956ce8702f31b2703135de8972b02d2212266dab3c90ae5992ebea5f97
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-client-linux-arm.tar.gz) | 209a0d8ff652f8fa8fa5c04ff31d280034fb75aac3cfb1765da91b424e162c04fc249961e727c396fa569ada76542c3f0989f63b9ce8e450722cab8294854b20
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-client-linux-arm64.tar.gz) | 0ed63848206c45715f223c4f4dc32ae7ead54ca1a1f9386b0dad24b0661fbe200a532af11a14b35f562c0a2396244e920c400b5f30c05e047c02c16308aabee8
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-client-linux-ppc64le.tar.gz) | 5ba72eddf283a7d0868be0c1a3d4c4415c0907cd6e1cb6de0002b814012d41670f6ffd38fae1e24fdb77ddd28aeee9af48de70605f8560abef77bac09e9de39c
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-client-linux-s390x.tar.gz) | aa192d6fbd4e9ce6c6b047ab13b64d05d0b5ff82fe16bc761145a4e66deb6758a426dcbe65bca5b8036bd9fce39ecd8eb0e43a66ac873c1de59a2997eb9d936c
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-client-windows-386.tar.gz) | c4e91b712d4741517405f96d085419733004204bb89d3456da844efa9fa8b9a4f8415412f3b7d7fd8ad1e048b153061f46cf46eec227ff837c2bade245e1b13f
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-client-windows-amd64.tar.gz) | 8e8c6ef13bfe265ede104eb0e147267dce2a2cc7835aab2f585fed95fc108d5a12b11cdeb06ff436a09efe65c1d11a1b6c30aa0465e052eea80edc30869a03cd
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-client-windows-arm64.tar.gz) | f6b3a5682712849986e322314b956974339b840d4d763e5a9f2bac7007aa9227bbec82296bcd06c97ec272787b05b7680ff2d2f260e2e2b25e944aa08f11c61f
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-server-linux-amd64.tar.gz) | e59ef3c5e55979a510a8b9a8827f80f753054125280633abf3b7fa4bd13cbbc856fdfc0c4c9fa17889124e38e862a4ea0e5c01845e685ac594016f7d177b6c48
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-server-linux-arm64.tar.gz) | 2641dd6147ff8316384266f607c1865bf7c91aaefa59ff6453fc12ffc577301b523d24b93abd9e2be7d04a88ff0a3b5cc78342f395d81419eba442b4d4c0bf5d
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-server-linux-ppc64le.tar.gz) | d83606ceb82d54d87820bcc20cc0f2675daada6e18a7194ea1a8998fb5c921270b6dc1fbe9e873bd404766e9bcfcea5824d5bd90f3bd338e2617144d7db0cb4b
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-server-linux-s390x.tar.gz) | 943dc40980eb501ec00b2ad73c24df562ddfc6773e55752ce4b0906af9ad9fd9c32d6f4102b55a72f6cb9b6d926024e4e39ba70d4e5040e2960a86fa53c1fd91
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-node-linux-amd64.tar.gz) | d901ee1471a2b0f9b7381eb310cbb4b9ce9b8e3dabec3eac32a9c1c451bf7a6fafaddad1ef01a6447384937a9987fcefe7cac6774f64f600859688ffc1d15f69
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-node-linux-arm64.tar.gz) | b3b85cbab0c05410ed3320a18522683418a91c7559037111737d6cf7d26584be6328881f28f0cdf4a63de115a25efe435a394ebf6a2872431a1c2f09c7df3a3e
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-node-linux-ppc64le.tar.gz) | 0fc8531a38788685294937efcd5064eb73d9513add61293b3697fd8ed701cf73a2ceb023f2d9bd7edbba12ad5c3b1d9d2376ad3e7983eb97ce1801c60cb31a68
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-node-linux-s390x.tar.gz) | c791a760f9b59e50aade23db9ad2e17fd715bf1ba65c66f31ca82eb02817b3110a0d0ac4b98abd18c2e01a83fb813e8d1bb45dfc0c2a5e28e130765e94740c45
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.11/kubernetes-node-windows-amd64.tar.gz) | cd35d82e1f8f2e2a7828accfd0d59363feb1bd9f0cf14b68a428da28b39515e60949732b052a145a79770a8e901ef5f59b844e1bb1879405484fadb1d13ab23a
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.11](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.11](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.11](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.11](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.11](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.11](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x)
## Changelog since v1.28.10
## Changes by Kind
### API Change
- Improved scheduling performance when many nodes, and prefilter returns 1-2 nodes (e.g. daemonset)
For developers of out-of-tree PostFilter plugins, note that the semantics of NodeToStatusMap are changing: A node with an absent value in the NodeToStatusMap should be interpreted as having an UnschedulableAndUnresolvable status ([#125308](https://github.com/kubernetes/kubernetes/pull/125308), [@gabesaba](https://github.com/gabesaba)) [SIG Scheduling]
### Feature
- Kubernetes is now built with go 1.21.10 ([#124831](https://github.com/kubernetes/kubernetes/pull/124831), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing]
- Kubernetes is now built with go 1.21.11 ([#125368](https://github.com/kubernetes/kubernetes/pull/125368), [@cpanato](https://github.com/cpanato)) [SIG API Machinery, Architecture, Release and Testing]
### Bug or Regression
- Emition of RecreatingFailedPod and RecreatingTerminatedPod events has been removed from stateful set lifecycle. ([#123809](https://github.com/kubernetes/kubernetes/pull/123809), [@atiratree](https://github.com/atiratree)) [SIG Apps and Testing]
- Improved scheduling latency when many gated pods ([#124851](https://github.com/kubernetes/kubernetes/pull/124851), [@gabesaba](https://github.com/gabesaba)) [SIG Scheduling and Testing]
- Kube-apiserver: fixes a 1.28 regression printing pods with invalid initContainer status ([#124910](https://github.com/kubernetes/kubernetes/pull/124910), [@liggitt](https://github.com/liggitt)) [SIG Node]
- Kube-scheduler: fixes a 1.28.10 regression that can lead to a scheduler crash when processing pods with affinity that doesn't match a real/valid node ([#125042](https://github.com/kubernetes/kubernetes/pull/125042), [@AxeZhan](https://github.com/AxeZhan)) [SIG Scheduling and Testing]
- Reduce critical section in watchcache to fix kube-apiserver scalability under heavy load of list requests ([#122027](https://github.com/kubernetes/kubernetes/pull/122027), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery]
## Dependencies
### Added
_Nothing has changed._
### Changed
_Nothing has changed._
### Removed
_Nothing has changed._
# v1.28.10
## Downloads for v1.28.10
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes.tar.gz) | 364ba90386eae32daf66266d3e225a1ae03f83934554c8a928c44c53833d971f57a0fbc6336ba9b6fd5756cb0824fe984ad8c2575182775551f70b229ad192f4
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-src.tar.gz) | 7930797f68229f1ca6e0992abd7a2d5408e81683065a8b75120281b3252b6f735ae802ed763dad8d04e25896c266c0fe4c55318868c14cb5422133c83b445ef1
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-darwin-amd64.tar.gz) | 49c9916b0c396f25605ed0e8038d6c0655002af2318f49f405abcf658b307c1a8815fb73436365d038e4fdb99ded00ac327f31b7d0ca5d18aca02c38ed8898b6
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-darwin-arm64.tar.gz) | 7232583f1c8efe9c41c7131fff4cba59280413b77905c5ba871313f551e647f73c1f7d9fbc0837b321331cd6ae6b275dc4d2518441f6929463dc720e6fc07b11
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-linux-386.tar.gz) | d9e8999f7bdc53f0211f78e24c548f04b02de75a1dede8a1f91ec53166c262baf5def6d32eccd48daef9116c8425cb8b0a9a0ffe083221a6deaba2ed3eb181e6
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-linux-amd64.tar.gz) | bbd684de821e294e4f39c9a577669a711ca7f1b47251dd5cae02dab62a4bc7347545b02acf61e97e0b4bcfc025710f186816a35470bdce6184a686c09216ea90
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-linux-arm.tar.gz) | 84415f3bf03dae5ea72b5e02ef35cb657e564045b1cbedb8885b2b6006dca697530074aef28be946c36962a34c0ab105f56838afafc817d41572c70330237daa
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-linux-arm64.tar.gz) | 196621add12081126edfda3a272109e8e2dcbb057732350086938b96b1cf32ae115b117b038da25565e3e3f9d22e05a123797115078e002470e4b91a6ae6ad26
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-linux-ppc64le.tar.gz) | 62df1b45d8cd2b25f7cecd2308af7090aaadf2991044b0fa7a0d1a7b46b084a8194f5520f8b32be795e3fa6b19cd4fa1e983e3d0e916295634611a2f5cc36067
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-linux-s390x.tar.gz) | 2bdf394382f78c454c73e4c3b0f8edc2904edbae994675ae4909edd9772e77e74a26d0e37dbf5cd5541c88066b5cd1e0eff5cc96154ece57132a2546fe08ec7b
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-windows-386.tar.gz) | d92d0afbb1de03bed99865393ee6a5c57c2cae4bd89edd197c117d031acde6c9a7e62646ec8d3756dfee3d592e624597565cad800e77348049fc787c8887144d
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-windows-amd64.tar.gz) | 5f9ee48813f2b098adf47c4ddb384596e757581d13b5384768bb353eb3520a83447f54e4c826628dabbea4541419516b6771f923bbc8db345a2017d0b7b7bfec
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-client-windows-arm64.tar.gz) | 0cbeaa95d1d5cf2cae2ea38bf6d71ea5a63ec7fc364c4066be12c27533dc2ad699ef79fe7ba30eda293c1eb5098426cecb04ca98d7234b89be6a989aff9449a2
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-server-linux-amd64.tar.gz) | 06245a0aea642bc38baa1549f4aaf51079252fc0f0b1346a90aadf5c9c260f1e6df1ad81fb6b00f81380270864e6a192e848cb88923a14016f8e4c7ffd097a32
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-server-linux-arm64.tar.gz) | 4b704f3e8099e783fd317583f32bb2be8c2916f719fbba6ba2760eddf62834819e737eb4bd960b6711a0279e7622d94c165aca67df531ce69a3813f7351fcbf6
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-server-linux-ppc64le.tar.gz) | 384287b9de5d37c4f6c3913751326ac8f6cc5d80a104aec190de10bd0762638308c9df058b2ca7e2d2205dfac9db7d8e418b1110b8bde6fd030bc024030eb7e9
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-server-linux-s390x.tar.gz) | 8dc966852a3518e59651b0e9d3e1a1af7ce6dd8a40b5cabcb8fca57275b35f792293a12c652eeb6f55d20869589753928973367fda3caf6b659c9aa1927b72ae
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-node-linux-amd64.tar.gz) | bffb3266c0c11c4831879ed25443ee5c1ec14b3c444140527119a887c6122f4a9a9829e5746e309e8f415a68efaeb948987f89601fde734f8c5fc889c5d0f27e
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-node-linux-arm64.tar.gz) | b6d179291f827f91b73d4cba1131f3af23000714a121292c3daaa79d15006c9d71749c1de8484b5d4ae8f6ad352c460e704f0a1be7f62d471403341a90826f03
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-node-linux-ppc64le.tar.gz) | 05605bb0ed75a6b53cc8d6d009c9705d41596b680d7b209bfc3154897da580a03b8ba46820a56de86d7f1e1b2d0d981d6efbfc279a216aa9cfccea318a41658c
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-node-linux-s390x.tar.gz) | a43f1feeadef4384f18e8aa045c3996cefa9c7680c37e93364aba3071610908712fa3bf82cd8770c6f26b62ca8705cd162ebac623c9bdd53cbf6f83694aabb94
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.10/kubernetes-node-windows-amd64.tar.gz) | 2280896ca4e762e2ffb363e94ebf881c54b0b4a7b36ee21fe305963646295683bc88aab8b831092736a5480753055cb8d7ce780571901d24ccbb99df0fde9948
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.10](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.10](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.10](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.10](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.10](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.10](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x)
## Changelog since v1.28.9
## Changes by Kind
### Bug or Regression
- Fixed PersistentolumeLabel providing wrong topology labels to Azure Disk PersistentVolumes when the external Azure cloud provider is used. ([#124528](https://github.com/kubernetes/kubernetes/pull/124528), [@jsafrane](https://github.com/jsafrane)) [SIG Cloud Provider]
- Fixed a bug that a pod may remain unscheduled if any PreFilter plugin returns nodes that do not exist. ([#124764](https://github.com/kubernetes/kubernetes/pull/124764), [@chengjoey](https://github.com/chengjoey)) [SIG Scheduling and Testing]
### Other (Cleanup or Flake)
- For apiserver_storage_size_bytes metric, we are renaming the label for etcd to be "storage_cluster_id" instead of "cluster" to to reduce conflict and be very specific. ([#124293](https://github.com/kubernetes/kubernetes/pull/124293), [@dims](https://github.com/dims)) [SIG API Machinery, Instrumentation and Testing]
## Dependencies
### Added
_Nothing has changed._
### Changed
_Nothing has changed._
### Removed
_Nothing has changed._
# v1.28.9
## Downloads for v1.28.9
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes.tar.gz) | 6445c7b17f50f2244f1fb39a64662db10252ec6c054379ac1119f7c0ee96b1a97aae1d1f663164e1eff89f9d6c3b3089d81702e85e8c4fed7f835bf53db1070e
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-src.tar.gz) | ba7ae8b833ebc21f384dd36e5efe61b12c082342314097542da0326fc19a4d54a3cd84848be60c85bf3675718eb213216d503ca8f088084e2d77b92cc1848c6a
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-darwin-amd64.tar.gz) | 90d5663170f8bedca8c95bd71653fcb1a2e1c2a7d86b765f8c46de2531447c034560900fd9a31596b4fc2606485c0923b0496902ae9c2c1e43572243596be924
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-darwin-arm64.tar.gz) | 207efb9097bef48895f6e03a9c3054b376d31a9f649f31b2c5bab18a26571dc5713f1a23bfe8cd546eabdc21765c219fab313e6a26866f58fef3647052ce6ca5
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-linux-386.tar.gz) | 366bc0ca6b8b6e6887a57f3b75b21da78d8688dc7c3adefdf5370eda7a49ad0251c41d06ef68e71e7841c1307678425200f9b2ccd55def3749994b0f23ca542f
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-linux-amd64.tar.gz) | 5142ad0fa9d709d28e481d22442550eb5806c376382990c5e8637f7846275841bfa59ace19dc5f6276b563003ef5d7d49b06ec223ba352fc040d17a351085336
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-linux-arm.tar.gz) | b77f309567bd3d828499dd7332ec485257df8a8cbc0d4d65f822c68466c2a2d07bab79317f5474826a73950955bc8af9491da215f05bbdd0d53b9367c9b53062
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-linux-arm64.tar.gz) | d89b89fad313764ee3b7aa71e0b87651961e1d5485bab40cc3c0af00e9e422ffe8245501baee4c465e7cfdeb446721a28d075ce53f726ab38cdfa5aff554ef8e
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-linux-ppc64le.tar.gz) | dfddba1e6db1702b8b80df9bdeead04cd72db47f84d615adc2090c851543d981b3cc9970e68e832ea73d13015ac8113ddef7247828b30da906e75129bb56a17c
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-linux-s390x.tar.gz) | 0957b71eba14a1728accd1e917e81b2cc95cdb4523a519b21fe82cb15885dedffbe49df2dca75059ba7590243c557d1948339a5fd10b67f141fd066606b35b57
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-windows-386.tar.gz) | 12179c49f2fa31970edc3b00232b69d431500200a2f3945a3fb4ce04d458c825b1e214f9ced1c7bb06777b79ca19f86444a1bee5f2f35b60f4b3407f4fa861d7
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-windows-amd64.tar.gz) | 6666378dbc9a43f62bfd69ef81993c4463ef1c8862dda1b40b5b18a90a81cfb2c26f19e15d8e3e019ab1ac8140cc15e11cd5d308c172f949264df69ee335047e
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-client-windows-arm64.tar.gz) | c74973f02e46c6c21a50b9b08c7211e475a8b29ad375feb84d5c36a9b8716052f5daec77e7a6b138b045ff88701740f357c7654bc48a339debae88036bc8ae0f
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-server-linux-amd64.tar.gz) | 9672e1921f858b3e77d85d8d915ba634b6693aa65ea223fc1eba0ca97e893dc391f691b8a35fb9c17b0f07ff0f6f37cae99164c2510e36a5fad6a3cdcf33a140
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-server-linux-arm64.tar.gz) | 668f237dbb96fa6b50f40c5452ea02c9db19c8fc07e73818d447f72b854f4864e6fcb119529439c51aab9e3233559eca7cfbe0d65d6f733f2281380c08c8a3cd
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-server-linux-ppc64le.tar.gz) | 222aefa46ce11f3345f72e1ff058da797d2fc3ccd08e5a9a8d4438f7b0262e4abb87cb6d7d719b30105a574fc5e61c9378f6fd1ccb16cfe7ccf5db5e8e0f8299
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-server-linux-s390x.tar.gz) | 74d30a38d00843d4b90906aff0aeee067effb64f6e3e244b6fd730a016de39c5754b20127b12239ada4f23099c6f1bcb2a619b084e8a3a5478b7e9fb8465e4c8
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-node-linux-amd64.tar.gz) | c215d09bd69bb71ecfc81d6a4605e16c68fae940ab62880b9e3a60e84805897c1bbd29fb98fbc3908629809b7396fe6c00765d6e59c44c85666bf70371aa6b4f
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-node-linux-arm64.tar.gz) | d19a290d769491fe1d97cb416aa481bdfb7d4831a4ceec35abc90d5035f8cd529fb3e4653b1ec71cf8d0a38ce10d6e1e9d054bdee7d243cc01fcb44de94ddabe
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-node-linux-ppc64le.tar.gz) | ab00c8323ec13870a270beb0d172b8c3371c69b234a422979ac5acb68349f46dc87a65fd734305ea940985e46d60a4c4a4a2886076a31ca4a67660506582076d
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-node-linux-s390x.tar.gz) | 60dcdb46e9a0b35505e06725eac88b590e62a97ad978573dfc98392a57538ff0d1c8ec15449cc8f47747c97bd29bda75302599443f7f969eb99eba9cbd78c27e
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.9/kubernetes-node-windows-amd64.tar.gz) | 7c2d2f8fdefae24583de5eecbb61165196508f91db57ec7d03eb9c61c02f61b440b1472e0d7619dbd81d94c2aefe04613db59e25e12cd4792cec86aace76d3d4
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.9](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.9](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.9](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.9](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.9](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.9](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x)
## Changelog since v1.28.8
## Important Security Information
This release contains changes that address the following vulnerabilities:
### CVE-2024-3177: Bypassing mountable secrets policy imposed by the ServiceAccount admission plugin
A security issue was discovered in Kubernetes where users may be able to launch containers that bypass the mountable secrets policy enforced by the ServiceAccount admission plugin when using containers, init containers, and ephemeral containers with the envFrom field populated.
**Affected Versions**:
- kube-apiserver v1.29.0 - v1.29.3
- kube-apiserver v1.28.0 - v1.28.8
- kube-apiserver <= v1.27.12
**Fixed Versions**:
- kube-apiserver v1.29.4
- kube-apiserver v1.28.9
- kube-apiserver v1.27.13
This vulnerability was reported by tha3e1vl.
**CVSS Rating:** Low (2.7) [CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:L/I:N/A:N](https://www.first.org/cvss/calculator/3.1#CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:L/I:N/A:N)
## Changes by Kind
### Feature
- Kubernetes is now built with go 1.21.9
- update debian-base/set-cap to bookworm-v1.0.2 ([#124198](https://github.com/kubernetes/kubernetes/pull/124198), [@cpanato](https://github.com/cpanato)) [SIG API Machinery, Architecture, Release and Testing]
### Bug or Regression
- Fix pod restart after node reboot when NewVolumeManagerReconstruction feature gate is enabled and SELinuxMountReadWriteOncePod disabled ([#124141](https://github.com/kubernetes/kubernetes/pull/124141), [@bertinatto](https://github.com/bertinatto)) [SIG Node]
- Golang.org/x/net is bumped to v0.23.0 to address CVE-2023-45288 ([#124179](https://github.com/kubernetes/kubernetes/pull/124179), [@MadhavJivrajani](https://github.com/MadhavJivrajani)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
- Kube-apiserver: fixes a 1.27+ regression in watch stability by serving watch requests without a resourceVersion from the watch cache by default, as in <1.27 (disabling the change in #115096 by default). This mitigates the impact of an etcd watch bug (https://github.com/etcd-io/etcd/pull/17555). If the 1.27 change in #115096 to serve these requests from underlying storage is still desired despite the impact on watch stability, it can be re-enabled with a `WatchFromStorageWithoutResourceVersion` feature gate. ([#124006](https://github.com/kubernetes/kubernetes/pull/124006), [@serathius](https://github.com/serathius)) [SIG API Machinery]
- Kubeadm: fix panic in the command "kubeadm certs check-expiration" when "/etc/kubernetes/pki" exists but cannot be read. ([#124124](https://github.com/kubernetes/kubernetes/pull/124124), [@carlory](https://github.com/carlory)) [SIG Cluster Lifecycle]
- NONE ([#124326](https://github.com/kubernetes/kubernetes/pull/124326), [@ritazh](https://github.com/ritazh)) [SIG Auth]
- OpenAPI V2 will no longer publish aggregated apiserver OpenAPI for group-versions not matching the APIService specified group version ([#123625](https://github.com/kubernetes/kubernetes/pull/123625), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery and Testing]
## Dependencies
### Added
_Nothing has changed._
### Changed
- golang.org/x/crypto: v0.16.0 → v0.21.0
- golang.org/x/net: v0.19.0 → v0.23.0
- golang.org/x/sys: v0.15.0 → v0.18.0
- golang.org/x/term: v0.15.0 → v0.18.0
### Removed
_Nothing has changed._
# v1.28.8
## Downloads for v1.28.8
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes.tar.gz) | 66998a96c8af53a8249708f5b25e77aae10fcf07e863d8c80145759dc7813d944a4bbd1b3a2b3676f33d2daf89d94bfe79475efe77694bb39b804dd8946405e8
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-src.tar.gz) | 02177b6c0517146bec036ddbad8575f742bbf8b328a604bc30f5e16398dee22f192b6840370d9e345492f919de98a0978c55ea0890587cb91afe77109a7efe8c
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-darwin-amd64.tar.gz) | d5cde8002471c38a70417e708c6d355498e770de513dd8cbc5364036498eed01d2a2b69266a91ad132bfaab4fa0082986af7e3d9907d906459093eb30797bd84
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-darwin-arm64.tar.gz) | 03895f6d43f07cd83225100ac633830bf96a4a7f8bf1cf4175ab11bb4c98696efefa67ff1ea6333d00fe46bd8dc329dcb5a5355f97a616ad3666d9b21d8d28e7
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-linux-386.tar.gz) | 072894bceb8dbc4b443be095d0446d33b5ab235d312a4a571afa81411880bb882a352d4b023e5edf96be7fbb66f367a37056ce0cb5b2c3f54e767cfe45d111bd
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-linux-amd64.tar.gz) | 0812400f8285dc6ead6d25b332ebe3355edd1619b092d1af2445c578553b09a435af7627322eb50d07d0f2d82fd29f4441ceb5ba0b4c4bb990db2ebe197d3e0b
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-linux-arm.tar.gz) | f08d1a33a84145f2493b0d4d85a85a2331d3753b93836166faa41e9222eca1cfdf0e3da624654d64a810bf16159ac2524d4be789ee85d6255ea1a3fde12b712f
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-linux-arm64.tar.gz) | 1ae4bd65c4f3483797c9b7dd7e1e8727b22a10dc09c877a9c2d468f615d1cdc915ec9f372832025fa7153cdb8e302d239738671a32a3ec715e2b7aa6d307dbd1
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-linux-ppc64le.tar.gz) | daf0b71a340f7afe81cd0c49d42f9a6e259fcb0b5bcb60f7b0ddfe0137d7a5b9a80569aa38b1e6eb59d094b2bd69d64487d7df8021fd5720ec216ddb615e37d9
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-linux-s390x.tar.gz) | 61e81b992aea5ecbfb395b9ff26b66a209530dc3155d3275ff977833d4d8798972571a2b01cf6eee8389b75a9d03aee555c8c211114aba601e1864f86b8006da
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-windows-386.tar.gz) | 5c8890d079280deb489905e9bf8b1ef5df9e1267eda9dd0c691bca8f1eb2395629a5bea4d4e2d9d71b89af422a515e226aab50724ee673f298e15aa311926738
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-windows-amd64.tar.gz) | 63a47818e2d45ffa0e598bc7c0915e140121fedb7cde578fd00332118401bbf495aaee811da11f0cd9075e58a94fb1eda40936a7c12fbfae99e06a4bd04ed0fd
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-client-windows-arm64.tar.gz) | 287a83b2f19f15ebef9488d65c7509a0423345873671a748f77ea71acdae1d44cb83c7d638677e248907eff14e4fa6612947e7b7f63c1eaabaecf5d607f3f49b
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-server-linux-amd64.tar.gz) | 3cb91a04cff7cc54d40e3cdeb9bd2b6ba57a54223de1aac82db33afcac47b7ed25a7267984937a7fb99e9eac23080a1879ec392978def7d70d327cff208eea2e
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-server-linux-arm64.tar.gz) | 6bd7d6654ab5454cada91454189b250c693264b0fe66dd2bcb7a80d7969785ecfecf90a8b550d1e9e576316c4275bbbd91a445819ec20a710eddc3c611fec3b4
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-server-linux-ppc64le.tar.gz) | 12058f01fcfa5c450c95f5f00149d095db7703bdb3736a6ed3fd33e8af3c60b702cfc4ae90cc79bbe0bb60de8470f9969eed4135343d527e15f852c3a986ab69
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-server-linux-s390x.tar.gz) | 3b37e695260f4305f916dfe53a4d6235ca4c4a559e0006e2c38e1832132e7b65c231f5e7daa6d7507f6d1be303430bd38f8425e1623a10389724c576054b4f7c
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-node-linux-amd64.tar.gz) | 8739961e2657ac011ff9112d85d1958a6fe6f6c8d358ae5e882c526f40d49f95ebf3b90e35473fdd50981f952951714a2d0ed722efbae886586a36e33fb7d79f
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-node-linux-arm64.tar.gz) | 85a5d9971e6ed12e51f274259dc39223599c42a0f81f546f19ac5ccc60caae46e2feb850d6fd5659aed6e2411127109488833d270ff42344dbaa6b30ffecf511
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-node-linux-ppc64le.tar.gz) | 11e284538091e43cd06c5ecf1c038f75242225d06ea95bd0c395bef2f8964a1037ee5e89bf74892d011e2a61d86f7d16aafb691e43b22feaa189fd53c26f2639
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-node-linux-s390x.tar.gz) | e89d208a8349e5d5208473bbf1aa67285039d18ff87b0d32555baec13c29ffee529a7a8fed36363fa9adc9825568c98d35a5fd8bab74803df3c1c1c1846e8fd1
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.8/kubernetes-node-windows-amd64.tar.gz) | 1dfa21cab0fc5dba81752f541801eab2d35da875b4a4688fd0b0b7d864a328f7fd60fdfb0488cdb314238102719aa9ba8bc7e7c539c466ff43bf556c1714b080
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.8](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.8](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.8](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.8](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.8](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.8](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl) | [amd64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-amd64), [arm64](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-arm64), [ppc64le](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-ppc64le), [s390x](https://console.cloud.google.com/artifacts/docker/k8s-artifacts-prod/southamerica-east1/images/kubectl-s390x)
## Changelog since v1.28.7
## Changes by Kind
### Feature
- Kubernetes is now built with go 1.21.8
- update distroless-iptables to v0.4.6 ([#123772](https://github.com/kubernetes/kubernetes/pull/123772), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing]
### Bug or Regression
- Fix error when trying to expand a volume that does not require node expansion ([#123055](https://github.com/kubernetes/kubernetes/pull/123055), [@gnufied](https://github.com/gnufied)) [SIG Node and Storage]
- Fixed a bug that an init container with containerRestartPolicy with `Always` cannot update its state from terminated to non-terminated for the pod with restartPolicy with `Never` or `OnFailure`. ([#123710](https://github.com/kubernetes/kubernetes/pull/123710), [@gjkim42](https://github.com/gjkim42)) [SIG Apps]
- Fixed cleanup of Pod volume mounts when a file was used as a subpath. ([#123052](https://github.com/kubernetes/kubernetes/pull/123052), [@jsafrane](https://github.com/jsafrane)) [SIG Node]
- Fixed the disruption controller's PDB status synchronization to maintain all PDB conditions during an update. ([#122056](https://github.com/kubernetes/kubernetes/pull/122056), [@dhenkel92](https://github.com/dhenkel92)) [SIG Apps]
- Fixes an issue calculating total CPU usage reported for Windows nodes ([#122999](https://github.com/kubernetes/kubernetes/pull/122999), [@marosset](https://github.com/marosset)) [SIG Node and Windows]
- Prevent watch cache starvation by moving its watch to separate RPC and add a SeparateCacheWatchRPC feature flag to disable this behavior ([#123694](https://github.com/kubernetes/kubernetes/pull/123694), [@mengqiy](https://github.com/mengqiy)) [SIG API Machinery]
- Restore --verify-only function in code generation wrappers. ([#123261](https://github.com/kubernetes/kubernetes/pull/123261), [@skitt](https://github.com/skitt)) [SIG API Machinery]
- Updates google.golang.org/protobuf to v1.33.0 to resolve CVE-2024-24786 ([#123764](https://github.com/kubernetes/kubernetes/pull/123764), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
### Other (Cleanup or Flake)
- Build etcd image v3.5.12 ([#123069](https://github.com/kubernetes/kubernetes/pull/123069), [@bzsuni](https://github.com/bzsuni)) [SIG API Machinery and Etcd]
## Dependencies
### Added
_Nothing has changed._
### Changed
- github.com/golang/protobuf: [v1.5.3 → v1.5.4](https://github.com/golang/protobuf/compare/v1.5.3...v1.5.4)
- google.golang.org/protobuf: v1.31.0 → v1.33.0
### Removed
_Nothing has changed._
# v1.28.7
## Downloads for v1.28.7
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes.tar.gz) | 084b28ba78ad4cc6da8422535222ee2232a6fef0f5f7eb5cd0aa1fd6b53433c4a4b2efc0002020c6cfae91ff8c194e0e0a55c876f3e563d33ff8ae4dcfaccb30
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-src.tar.gz) | d20000cd269a569013c5cac3761e03f5c2f99094f57b43e0f531a752b355fc7348efbf93ce75185659a2b2a5be65fed5cfa9948cdaab9d359f571899eab74530
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-darwin-amd64.tar.gz) | be563c6e071585fcfb0d627ee8e2c1cc2ab0375e0e6b3ca5fab758dc085825a399169e1e309d008c853a493e3a1b899ff7bbfd491db082f7becf98d80a31fe87
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-darwin-arm64.tar.gz) | 49d1ef441b7de249ca988608f865d9d9d3e565c437888371ac2901cee6e98f607602cf8183f54bbf9ac77bd0d50bebca151e172b4c764ae0f6e0523bdddc0111
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-linux-386.tar.gz) | 693f2fc3288f9697c831b9d52c507277c3ef38afe8136492755c1928e489c49cd7fa78c482c91421199769b979d19c014faf36a7c558ecc6eed245fbd03c63cb
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-linux-amd64.tar.gz) | 8dc1da1b4f447c0d3d196da83f9560d685f311c2f3a10ca2560af8f328dd1b390547b9ca49a3754e92be4b77b086e3db2a888d8f69afb696d7e6f0b5d5958bf9
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-linux-arm.tar.gz) | a00d191feb238cbd845c8f0c641891d043b182e45c75466af8111bbae51e8e81233545fb431149d23b0525976037474421125031b9b05c9730c983f0ce804937
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-linux-arm64.tar.gz) | 340105ecd45b0c7904552849310e6e681f3e209583d4e54298ba7a2f2f8ec205acd5b17e119e0cf4c58bb993c1527610a4ff30c7b584c2f0239a93cdb6ba3357
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-linux-ppc64le.tar.gz) | eaa9ff433b487b7f42d38e322b8f901cca30f4d22ead2cdb9c48aa13c44c48ba4059c1913ef72be9b812a7792d58c0b6197be87d64a29b6bc0135cce8f43578a
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-linux-s390x.tar.gz) | a0d7471236cba0ac9644916e09b92d42f2c884ea277e2b368061b68e47d6c357fc0cc79c7aee79a19862168072b49bd9a238c3db59bc1f8afd4d0cf76e7ea725
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-windows-386.tar.gz) | 24ff9a4ce3715d293a2895e178fe9acc2ac393ec089e75eec343f541bf48b7d03c7e43d0ca09fbe8694914ed2f05f8d86e5d113bff1433f9744a06b931a7cef4
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-windows-amd64.tar.gz) | da69beeb042b50279e3185d9b92f9738a8082c40f073e5b34b585ae6f11268d69277104ff17f572cfb5a0065ec86bb18e2882eab63fd0c9d9aee18c45402a5e6
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-client-windows-arm64.tar.gz) | 1e3b2e84638e75c70984f1366d24cc68f9eee0a9ee37008f22686173ca886ce538562252537753f929a4d242a91134efeb90d75164074a202b90845394337cd0
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-server-linux-amd64.tar.gz) | ea566c66916cb6b892dec2151850bae766a5593e3a4c1cc39b1ff945fffd843611ea980a01cc9dea4bd69bc1a1fefc6e31284937e8d88c19f7d8e183917bdf8f
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-server-linux-arm64.tar.gz) | 157b2ccde60b2a197e707b504affea22717f32fa01b7727bf7bd8e2709c728eb09e9dc918b11beb01908298f1eba2a6bbf8939e12dc23987031bedc6d6b8ed96
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-server-linux-ppc64le.tar.gz) | 8921192dfb3e8fc375cf0e0dd572799f4a0fb59ffe3722092779fadf04d3a00aaba5d478965bf613a0dc6be156474e615cd480f695931c7300e24851f993e833
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-server-linux-s390x.tar.gz) | f5b7b47bf826157e847cac806104f36848483682926ad3c168cbd273c3a573e582fa686b36c9016d4e593957672566c20d2f8f24df30f1f2863ebbb2d60c96b0
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-node-linux-amd64.tar.gz) | e1cfaf7e2a8f264fdbf6424d50732c8806f2a3a64cf3ba34f06a48fbfcfdcb6c255d9873285361fabef4d8290eee373e5933ec67cf1145b00042ee547e65ba8f
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-node-linux-arm64.tar.gz) | 984f5d37ef39c21358d3e96c27e8bf5e451b41cb268cb81c65fb3a54e0ff55284e8e9ba668f99a70daa4ab1cdaa54f67f6c421ff283948906551529c372f67b5
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-node-linux-ppc64le.tar.gz) | d03ae5a3952246dbba92093e8a38d0ccf46fa345717e248427d49bca04ea66b1985a2bbdc442aad87e81d8f619cfe060191cf7d7ec97f91a8590d98dda538f2c
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-node-linux-s390x.tar.gz) | 94420b4279f825bc6e412d51c6076c07fa44a06c55f6bea8a5a028a762b581d87dc94ea0d20369f66410d6340d4623f9c1cac0b68d27bbc921b80f28de39a0e2
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.7/kubernetes-node-windows-amd64.tar.gz) | 7ae84dffdacb29d67f9a5d655e96e68e0adb5d8bcb329e5dea6c9a603a3cc5b72825077eb535ba6a9a34691c79524ef2ee7745f0b15be97f7e97d8ddd0cfd14e
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.7](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.7](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.7](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.7](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.7](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.7](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.28.6
## Changes by Kind
### Feature
- Add process_start_time_seconds to /metrics/slis endpoint of all components ([#122750](https://github.com/kubernetes/kubernetes/pull/122750), [@richabanker](https://github.com/richabanker)) [SIG Architecture, Instrumentation and Testing]
- Kubernetes is now built with go 1.21.7
- update setcap/debian-base to bookworm-v1.0.1
- update distroless-iptables to v0.4.5 ([#123228](https://github.com/kubernetes/kubernetes/pull/123228), [@cpanato](https://github.com/cpanato)) [SIG API Machinery, Architecture, Release and Testing]
### Bug or Regression
- Fixes a race condition in the iptables mode of kube-proxy in 1.27 and later
that could result in some updates getting lost (e.g., when a service gets a
new endpoint, the rules for the new endpoint might not be added until
much later). ([#122757](https://github.com/kubernetes/kubernetes/pull/122757), [@hakman](https://github.com/hakman)) [SIG Network]
- If a pvc has an empty storageClassName, persistentvolume controller won't try to assign a default StorageClass ([#122704](https://github.com/kubernetes/kubernetes/pull/122704), [@carlory](https://github.com/carlory)) [SIG Apps and Storage]
- Kubeadm: do not upload kubelet patch configuration into `kube-system/kubelet-config` ConfigMap ([#123107](https://github.com/kubernetes/kubernetes/pull/123107), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle]
- Kubeadm: fix a bug where the --rootfs global flag does not work with "kubeadm upgrade node" for control plane nodes. ([#123097](https://github.com/kubernetes/kubernetes/pull/123097), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle]
## Dependencies
### Added
_Nothing has changed._
### Changed
_Nothing has changed._
### Removed
_Nothing has changed._
# v1.28.6
## Downloads for v1.28.6
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes.tar.gz) | 63f60784a96277b7e06524123d483534d65bffa5416cabbe6b48659bbd4f901ffc8cbf9708c0edf2bd37eb09ea2cc5e77813de7a1f274bc5a7f467e949ce0fd8
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-src.tar.gz) | 33410e98750825ce3472a0b4add5e5b2ec42c3735c405d045fffe8b71e0345ef3a335cda142a896b49190e0f667d33db909f851dae00059d406d1b067e645bfe
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-darwin-amd64.tar.gz) | c02f0d2c96bd7d62f699f21437b2b49966ce1fd67442cc847b3e95a3233144983627e3d69cc20d0250b08f197481e5d9a4f49988174957a9ebc7bc3178cd64e8
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-darwin-arm64.tar.gz) | fc4aa1681b2897e69a0b2924a96ac82b46443b98b5cc26f5536b648509735dc21f09d29b68506153c05c915ad519979d6e751023b46fa4e2baf1b90dbc1bddc7
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-linux-386.tar.gz) | d46bdc7005c941baaecc5f770b348ab1d80701b4255b34cd4cf887bc722bdb4826524d95d3c7da6052e7506e9f8ffc40c2674436177f7771227cdfbcc1c1827f
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-linux-amd64.tar.gz) | b81a92ff7ead2de98f86b3da3976917cce45e576c6368c564f1dbe20a2e0cb99fa51da3aa5cc16805d57abe856f82cf3318c1425d6d05557d18b856e8550a06d
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-linux-arm.tar.gz) | 9ce9c439328a93cf4c69236ca6fea2834cd31bcb3752e535be15ec63cc79f4bb1645a79bc9cb900b8d97ff65920cda9b95de95b81af3b70e3d436b17827d4af9
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-linux-arm64.tar.gz) | 8618c45c20ff4b0c839731e645c9f081363f25ef3b8e1436aa7643db8cbf03749a20d22ca0cd3e98cf679b74eee7cb2d3ba9b80d0d817dcb41dac6322fdd670e
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-linux-ppc64le.tar.gz) | 38ef6b5c0287f267059f71ede0c22f30fa2291e5d1a18ad63cb540de0df51a87b66f3aa04567208e4ce4b298f99d2de72e6010101b3075e8896b5c5189e77bfc
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-linux-s390x.tar.gz) | 94b46b7bbfadeaba05ec086cc7031be41d697b8377ddab419177ab39518ed59b7f2f8e6ee12256bd2582030788ac126009d82f409f5db5d8c51bbf6e3a8f4f3e
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-windows-386.tar.gz) | 57b55ba7dd596dedfb09343ce5f8ea14783511bf30e7b462c7df06f27abc4f1cff2d479c56d7de1d3022567edcfb7c923465f9e651759ef429f15acb595b9530
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-windows-amd64.tar.gz) | 84d90c155b9ffc1748f225a9c47a04a48cdbb8106c9fcaa33d0893bdd5590da089a81943754012bb932d4ac5eb645696eb589d60e8747f561ecf319bcadb220a
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-client-windows-arm64.tar.gz) | c5d6d1081bbb0410faba25fca0d21c8a9417df958b953c23ec41c31266f43ef92aa34d8b838e351442d956089ea387abf6b09a4279a2a91e37c592f45427434e
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-server-linux-amd64.tar.gz) | f01b166c4581ff933972be0c82ee168dee26724598141a5855bf26a663c52771e3476a1999e65680940b5436871e887432f93525d83924b0f39a706e8e14a5b7
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-server-linux-arm64.tar.gz) | ffbb00d02764f301bdf4e5e840443a1146c729c379462194a24e9596cb66f87469373cc6df9293f7a36cc246b90a1dd9e637c8ed6fb657ddda9d69c40e70c809
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-server-linux-ppc64le.tar.gz) | 1d6aa422e06174018ae86af755e80fa2b1b4f3c114f051183bdd04479fc47a17769e34d6a09525567e1afa577069895716637f805b5af15ac32bd80ee79b48fe
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-server-linux-s390x.tar.gz) | 256104b0d9cf882fd8702f7b0a2c20533bd4ee7035db03caa8eeb2b54c0aa70742eb7c934b6d9e8def7108fe991e10a5d0447e02f25ef65f2feadfa13e5d3de6
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-node-linux-amd64.tar.gz) | f71b2b1193fc1316f66203f1c8d862ebfff59783c90a582d06553273b25ae70761eb971e35bbc45164a38cd5177196644c06174d2d6a3db108c292bd47d165ab
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-node-linux-arm64.tar.gz) | 86ea03b352369f5c0d24002bec322640ec69364031eea827667a4436c2357d2e35ca92cd3489d59fe045c9492bae76336d086e575db2d96f5db668e5010a608b
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-node-linux-ppc64le.tar.gz) | 0ce43b5f3e1df59e3f4e5673a16e23314e2c4f3d30639b965120cafebe709b812401c4db3397e9785bc46c3aac0ad92d9b315cce084cf42814437f8ad9767525
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-node-linux-s390x.tar.gz) | dac0e90c901513f9006cc5ecc363c8241e097e32ee17ef792bcd698b3c64394ecf32b861484fa7365d5bb4c5cb6fa61e7526a890e392302be53d2f49758e642e
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.6/kubernetes-node-windows-amd64.tar.gz) | ed00414a085ae0c986040c97d3c1d2631e353795e420d13feba1ced16b9977a17caaa24f9e563adfbd9128a60f5cebc206ae304ca4dfd928219fd9cc71b20219
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.6](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.6](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.6](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.6](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.6](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.6](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.28.5
## Changes by Kind
### Feature
- Kubernetes is now built with Go 1.20.13 ([#122712](https://github.com/kubernetes/kubernetes/pull/122712), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing]
### Bug or Regression
- Allow deletion of pods that use raw block volumes on node reboot ([#122211](https://github.com/kubernetes/kubernetes/pull/122211), [@gnufied](https://github.com/gnufied)) [SIG Node and Storage]
- Etcd: Update to version 3.5.10 ([#121805](https://github.com/kubernetes/kubernetes/pull/121805), [@mzaian](https://github.com/mzaian)) [SIG API Machinery, Cloud Provider, Cluster Lifecycle, Etcd and Testing]
- Fix: Mount point may become local without calling NodePublishVolume after node rebooting. ([#119923](https://github.com/kubernetes/kubernetes/pull/119923), [@cvvz](https://github.com/cvvz)) [SIG Node and Storage]
- Fixed a regression since 1.24 in the scheduling framework when overriding MultiPoint plugins (e.g. default plugins).
The incorrect loop logic might lead to a plugin being loaded multiple times, consequently preventing any Pod from being scheduled, which is unexpected. ([#122368](https://github.com/kubernetes/kubernetes/pull/122368), [@caohe](https://github.com/caohe)) [SIG Scheduling]
### Other (Cleanup or Flake)
- Reverts the EventedPLEG feature (beta, but disabled by default) back to alpha for a known issue ([#122719](https://github.com/kubernetes/kubernetes/pull/122719), [@pacoxu](https://github.com/pacoxu)) [SIG Node]
## Dependencies
### Added
_Nothing has changed._
### Changed
- golang.org/x/crypto: v0.14.0 → v0.16.0
- golang.org/x/mod: v0.10.0 → v0.14.0
- golang.org/x/net: v0.17.0 → v0.19.0
- golang.org/x/sync: v0.2.0 → v0.5.0
- golang.org/x/sys: v0.13.0 → v0.15.0
- golang.org/x/term: v0.13.0 → v0.15.0
- golang.org/x/text: v0.13.0 → v0.14.0
- golang.org/x/tools: v0.8.0 → v0.16.1
### Removed
_Nothing has changed._
# v1.28.5
## Downloads for v1.28.5
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes.tar.gz) | 62f0f175e544a31e685d1371c5a06af158be6a539aa0f33c2adb0765c995592e8182a29da3f92a326b98d2ec09b3202d63a73289bbfcb17cbb1e6e6961329c34
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-src.tar.gz) | 18f493ac18499c4e0e50b72bc3f01670e849d02f8b4bc70b9deefbbabb6119314105196d51d4fdcd9de2c66054ffacbcceb41ad19581e3d2a2d893ef2d97201d
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-darwin-amd64.tar.gz) | 4167663b3bf355a710cf84abd72df0c40a99c230c90711cf6d374d606022eea5d93137b1139530696a4678c3714bb61cfe5ac2fc599c7b4a4ad79486e7427bf9
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-darwin-arm64.tar.gz) | f41cb16a674312206d6a5ae0db9670f5873e1ab946b9bae65afc5899a6b4624af12b1c4a6eb08330cc7147572b6a6d097d864244a308fb2a55a5099c013fc9d5
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-linux-386.tar.gz) | 87f205d969374cf790a6bbef32d9a76edf83babdb9a33e50009e25be4281968d89340d24144bfa1ab73bb589fcf86fef604e567d6405c30deb2fe84a0246ef27
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-linux-amd64.tar.gz) | 9cd61a97b37cb27cc565f5a2cebd6086b86148c5759eb0e6a0c03e7be4b701bec407c46a65633c51a00a7aa74733c2fdd082b9da3382d38525e2e5b8dbb11b77
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-linux-arm.tar.gz) | cc79b49bf4478032bcd086167959141648e7805173636f8779a0a0242db2a5cbdb96131ec2eaa823db89c022a0e4b88cfd0a292a99ceebca3bbbe56c20410c57
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-linux-arm64.tar.gz) | 84f4074804e49764b7f911d7f23c6814df18f47cddc4d56d0b951cec12a82d27602744a73888314473639051bc4f1318cbd53ecd79be009f8d01d9ab4f71748e
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-linux-ppc64le.tar.gz) | 31f0dea0b968e9b0df3199943a7684795db1bf9168f2c65eaf6b2c77932eb2f87c7ccc1e2988c087c13d908671fcec6def9d28b7c08e23b7943c96785abadd74
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-linux-s390x.tar.gz) | 71216d98b1f2f663d47814af5893456a8e39639af2d571eca23bd70e670b9809c81d6845a681d9fb4609943a51b9058126b3ed9f0c9c6ce363cdb9835ca21041
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-windows-386.tar.gz) | d5c04c16229332ceeb77cc61bd8e8508f886f713c8ad97674ca16039033e8695a048216835bc633456c561748b3c571d4a0c6e6eb34ff542935c78536a9e268e
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-windows-amd64.tar.gz) | c7d453bc83d88c0ae2db9561e14172a4de1fc572ffc09699528a363b835e61a2426e7fa403afe549c6f437cd64cd789354cf64232b66bcfe507019b0106ada29
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-client-windows-arm64.tar.gz) | 09045b94d4efc0d184eb3dcab2a162908838109d7c2661202c6aefb5490275e99e79b994400a495c8264ff65b6b0423bf6e0a84eee579aac872abcda7d60a0aa
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-server-linux-amd64.tar.gz) | 22c6e2f42fcd5f7b9509803388c75013e6cafad815673ada33ed112bcd2798b5973ac518975479a1b0a4c7d0ea85b647273e9d8bf3296373839259592c7c7bc1
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-server-linux-arm64.tar.gz) | 162c982de71d05d09e2396de6121fab39f4cca72b75e69a6d43b1cbc14d852711c6ac0ed3458c96f1e34fe7af222038355b9389cc814877d60b9dce113649d24
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-server-linux-ppc64le.tar.gz) | 1bebd732647659ad57b25d2cff977e3c6a9f2cc38752eafd6c007ce174b3eafeb0a2a9ed6b1838df794a43bd782bc0ea65245df2058fab1b11e43204aaaa1cdb
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-server-linux-s390x.tar.gz) | 5c2254c6ff0c779cc42234c66d0ccfb12a6c5e81bc1c82feb476cf375fa54f0304a34372d9af83b778495ab94dc560886b8d3f44d1fb6768727ea33960c61ff4
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-node-linux-amd64.tar.gz) | 07aeddf94fb7172646a83a6304319568c5ab897f172cacf7fcdd0f5d7978fa85d9a32848d7b342dd27ddb1ffd9a38eb3d0cbb020fb93c8651cd28b30c5d631d0
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-node-linux-arm64.tar.gz) | ca26eb4f25ecd2ee0d4cd9d7a3813baf350fe3e5ab38ec02b1d06db8481cbf0b04f016fc15af8af7ddf789e6f42daa9ab09531d983020bf04b2f2fdd338cf15f
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-node-linux-ppc64le.tar.gz) | f850f0449e50ea3d8655edd9cdbb6447dce15d01919b023edf3ce910b03bf09f04327cb22707d931ac4391a6910a83b461f4b5b57f0e80ecf4c130861298df98
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-node-linux-s390x.tar.gz) | dff3115be5882177dcc774f3f277c64cf0658ad35d23a2f2323e2a952df69fb644b57cf26b2b5ccf84a1cd56ad80762494be236522c266fdce2790aa48671a70
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.5/kubernetes-node-windows-amd64.tar.gz) | c78383266ed43429600acc8a46bcc6c77437af7c758b9c530716d34d33f2bd9667a13491324b8b3f733e2ea63fc6047ab639f8126956291a2fc8124a37b024cd
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.5](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.5](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.5](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.5](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.5](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.5](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.28.4
## Changes by Kind
### Feature
- Kubernetes is now built with Go 1.20.12 ([#122216](https://github.com/kubernetes/kubernetes/pull/122216), [@xmudrii](https://github.com/xmudrii)) [SIG Release and Testing]
### Bug or Regression
- Fix panic if there are more terminating pods than active pods ([#122267](https://github.com/kubernetes/kubernetes/pull/122267), [@kannon92](https://github.com/kannon92)) [SIG Apps]
- Fix: statle smb mount issue when smb file share is deleted and then unmount ([#121851](https://github.com/kubernetes/kubernetes/pull/121851), [@andyzhangx](https://github.com/andyzhangx)) [SIG Storage]
- Fixed a regression since 1.27.0 in scheduler framework when running score plugins.
The `skippedScorePlugins` number might be greater than `enabledScorePlugins`,
so when initializing a slice the cap(len(skippedScorePlugins) - len(enabledScorePlugins)) is negative,
which is not allowed. ([#121667](https://github.com/kubernetes/kubernetes/pull/121667), [@kerthcet](https://github.com/kerthcet)) [SIG Scheduling]
- Fixes a kube-apiserver log volume regression bug in default 1.27 configurations (introduced in 1.26, activated by the AggregatedDiscoveryEndpoint feature enablement in 1.27) ([#122096](https://github.com/kubernetes/kubernetes/pull/122096), [@ritazh](https://github.com/ritazh)) [SIG API Machinery]
- Fixes a regression in kube-scheduler memory use in default 1.28 configurations by moving the SchedulerQueueingHints feature gate back to disabled by default. ([#122291](https://github.com/kubernetes/kubernetes/pull/122291), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling]
- Fixes an issue where StatefulSet might not restart a pod after eviction or node failure. ([#121389](https://github.com/kubernetes/kubernetes/pull/121389), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) [SIG Apps and Testing]
- The scheduling queue didn't notice any extenders' failures, it could miss some cluster events,
and it could end up Pods rejected by Extenders stuck in unschedulable pod pool in 5min in the worst-case scenario.
Now, the scheduling queue notices extenders' failures and requeue Pods rejected by Extenders appropriately. ([#122045](https://github.com/kubernetes/kubernetes/pull/122045), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling]
### Other (Cleanup or Flake)
- Bump distroless-iptables to 0.2.8 based on Go 1.20.11 ([#121976](https://github.com/kubernetes/kubernetes/pull/121976), [@cpanato](https://github.com/cpanato)) [SIG Testing]
- Makefile and scripts now respect GOTOOLCHAIN and otherwise ensure ./.go-version is used ([#122075](https://github.com/kubernetes/kubernetes/pull/122075), [@BenTheElder](https://github.com/BenTheElder)) [SIG Release and Testing]
## Dependencies
### Added
_Nothing has changed._
### Changed
- github.com/cyphar/filepath-securejoin: [v0.2.3 → v0.2.4](https://github.com/cyphar/filepath-securejoin/compare/v0.2.3...v0.2.4)
### Removed
_Nothing has changed._
# v1.28.4
## Downloads for v1.28.4
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes.tar.gz) | 9b1aa58395d4fe0efd75382dc4fac3c3203570f80f71e5a4e354983b597d4af442475bbcc65a7fccb2c2faa874954b69d09122c84e5ee78caa2ddc4cd8b82b26
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-src.tar.gz) | 70b929f5ab11b0bb14d0e7870a2f4e3b4b1c5d52016cec4560198a339fe3363c6e9de0ebc50b643ba12d569bd527737141c36d8abccb559596e772624a8219cb
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-darwin-amd64.tar.gz) | ed701ec7c67260bfb5cb910ef890732d171e72f3abff552d321a014ae59f1f45fa1339949c0711a9e82626eed5d55916489c8e339381e120c4a9b63970b8b3c0
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-darwin-arm64.tar.gz) | 43b6c1daa9b5597281536a4875f6f63258111f1a6fa1a38c41f95d07309991dba4f2eccf2321db5ea126d0f51ace2ed390ed10919ce7280f134078aebfc54d1a
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-linux-386.tar.gz) | fe0299567d47edf8625ea84b12233e07559cacae0fc3797afb249c86171b803125d2c34c15bdbc4a25a05646c8480bf9943b8dd9a2583cbedf70bccef309113f
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-linux-amd64.tar.gz) | 8da3b845e44ecbf94f2766d18dcccd2cd55f5645038c5ca50ae2163989cec5a330fc6c4b55780e4986f2b619edacd21c0236a5637b9623cb452b97b590a2f483
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-linux-arm.tar.gz) | 0e51666c37cc3e151ec67457a9d51820d6b7a2fe52d6bcb6c187b8a7461d64e6279c762ce34a272baa05c0feae47859df390e2aceb6f8396ba471a103456b9b5
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-linux-arm64.tar.gz) | 57293627a6f4b2bd45db5101f81a8b8d6e4986f5a11d24f4f782c87b5867b8b8015654b8355dd51292cb663b1719d34282259a514d566993fd9c16db95768a7b
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-linux-ppc64le.tar.gz) | 789fc636d69e58c41fd2fd83f07ed66bc96ea2e929114d5b7fcf2522a3a5c61b49c56986ac866d093224f5ef6878e9dfb6ffbe8674df5228b869864ae140680b
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-linux-s390x.tar.gz) | 99f3cd73dbfdb0bcade8552c5adc050b67a416edf68f35bde0e62fd93fcc88d160fa416d23ee72f0ec2d7a375f990d218c4fe8b69ebc0923bf68c4cbd992ab10
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-windows-386.tar.gz) | 9a60ead2cc4333138936a561a6d07671f8b17b865733d98f8d64cd91f0551897adfe1a1ca660dc2bf474375128ecdc4654983eb63285be50656b59a54ba7bf77
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-windows-amd64.tar.gz) | ba174234eba338f563a4ba6fb66d12db62df272d5fdf2c68fd12d7660c88036f5c35881705f07e4b0bad58dad2d49465643c40547d14a718f5a2098468e233d0
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-client-windows-arm64.tar.gz) | 7e830b5aadef11109ff3bfb24d57455ab9292f51e37f8c60d84c09ccd9adf4db2cd338c7402d8c03242db5d96877275dddf56f45531392ac7f7071d85bb1ec7c
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-server-linux-amd64.tar.gz) | c7d7016860ff44c15322040a5764d4acbde32ffebeb84802b9be820b4be22d9e0a1f8c2ee4547dccebaf133acc22f624d46be0de567ac5f98eb97303bbd5d7a9
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-server-linux-arm64.tar.gz) | 4f1d2299f0d0ca52ff4a793d41b8bcb4c50fc9fe6584559a92f88e91966fc6d47b78cff407076e9169d6a592cddbadebeb2348c7c96192eb5fba0f71818a3752
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-server-linux-ppc64le.tar.gz) | 1f86ee121c0f91f7cd0e0006fea01248b2a3afd49c748da99a85774a5c8dc0b98a2a4b3668186cb59fd77321287b8a745e6feb2d29dfc0a178795361b5b8a4ff
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-server-linux-s390x.tar.gz) | 98be6cfd43643ac86771ce36856925e7827cb57bec9482a33c23b6ee04c6a208e9f359027f762a825c8b62163dbf1786ec64ff51dd47b495d6f09cce6423d0e1
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-node-linux-amd64.tar.gz) | 845afdaa584d917cd29d46963bc337560eecb59c97b9ce0f664e3bd9ebe9c803fb239fcd6af69b852c2f553480d09b2b0a34db109d9fb8ea315a0df0fbbce0d0
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-node-linux-arm64.tar.gz) | cf17d5cce0519fff510a4599e014f1972e3e34d7f2f453c75d556296cdffc550271c76c1742ced1b5890ffd934c49c6aee67b769c0f758c914a7e40a402ccd3a
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-node-linux-ppc64le.tar.gz) | 8338669995d7179807a3cd82f6788830907440d19d91f0d5d8572f8a84a2fda101f878548fa2332736ee7e09c8d70af7402a195b54019a15ff95850751420edc
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-node-linux-s390x.tar.gz) | 65e0742ca66a62ab971fc11d5e5c19ea776fedea9f3648b5dac7689005f951ec63dc0455ef2d004aeada094ba1922d0c827e6af678f2ddb68134f1a936daf258
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.4/kubernetes-node-windows-amd64.tar.gz) | a147c116fc4d6c49e624a2ea0a59bdebbe61e9fa7f28743cd543459f61b6f22f7c77d0f4926a46e3d116fc30c39b2df89395d4da44f5c6a443bda35392139f51
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.28.3
## Important Security Information
This release contains changes that address the following vulnerabilities:
### CVE-2023-5528: Insufficient input sanitization in in-tree storage plugin leads to privilege escalation on Windows nodes
A security issue was discovered in Kubernetes where a user that can create pods and persistent volumes on Windows nodes may be able to escalate to admin privileges on those nodes. Kubernetes clusters are only affected if they are using an in-tree storage plugin for Windows nodes.
**Affected Versions**:
- kubelet >= v1.8.0
**Fixed Versions**:
- kubelet v1.28.4
- kubelet v1.27.8
- kubelet v1.26.11
- kubelet v1.25.16
This vulnerability was reported by Tomer Peled @tomerpeled92"
**CVSS Rating:** High (7.2) [CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:H/I:H/A:H](https://www.first.org/cvss/calculator/3.1#CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:H/I:H/A:H)
## Changes by Kind
### API Change
- Retry NodeStageVolume calls if CSI node driver is not running ([#120330](https://github.com/kubernetes/kubernetes/pull/120330), [@rohitssingh](https://github.com/rohitssingh)) [SIG Apps, Storage and Testing]
### Feature
- Kubernetes is now built with Go 1.20.11 ([#121812](https://github.com/kubernetes/kubernetes/pull/121812), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing]
### Bug or Regression
- Fix 121094 by re-introducing the readiness predicate for externalTrafficPolicy: Local services. ([#121116](https://github.com/kubernetes/kubernetes/pull/121116), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider and Network]
- Fixed a regression in default configurations, which enabled PodDisruptionConditions by default,
that prevented the control plane's pod garbage collector from deleting pods that contained duplicated field keys (env. variables with repeated keys or container ports). ([#121379](https://github.com/kubernetes/kubernetes/pull/121379), [@mimowo](https://github.com/mimowo)) [SIG Apps, Auth, Node, Scheduling and Testing]
- Fixed the issue where pod with ordinal number lower than the rolling partitioning number was being deleted it was coming up with updated image. ([#120731](https://github.com/kubernetes/kubernetes/pull/120731), [@adilGhaffarDev](https://github.com/adilGhaffarDev)) [SIG Apps and Testing]
- Fixes calculating the requeue time in the cronjob controller, which results in properly handling failed/stuck jobs ([#121327](https://github.com/kubernetes/kubernetes/pull/121327), [@soltysh](https://github.com/soltysh)) [SIG Apps]
- Service Controller: update load balancer hosts after node's ProviderID is updated ([#120492](https://github.com/kubernetes/kubernetes/pull/120492), [@cezarygerard](https://github.com/cezarygerard)) [SIG Cloud Provider and Network]
## Dependencies
### Added
_Nothing has changed._
### Changed
- cloud.google.com/go/compute: v1.19.0 → v1.19.1
- github.com/cncf/xds/go: [06c439d → e9ce688](https://github.com/cncf/xds/go/compare/06c439d...e9ce688)
- github.com/envoyproxy/go-control-plane: [v0.10.3 → 9239064](https://github.com/envoyproxy/go-control-plane/compare/v0.10.3...9239064)
- github.com/envoyproxy/protoc-gen-validate: [v0.9.1 → v0.10.1](https://github.com/envoyproxy/protoc-gen-validate/compare/v0.9.1...v0.10.1)
- github.com/golang/glog: [v1.0.0 → v1.1.0](https://github.com/golang/glog/compare/v1.0.0...v1.1.0)
- google.golang.org/grpc: v1.54.0 → v1.56.3
- google.golang.org/protobuf: v1.30.0 → v1.31.0
### Removed
_Nothing has changed._
# v1.28.3
## Downloads for v1.28.3
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes.tar.gz) | 98fd6d3713e8708e7664adf7e9fcae73b570ec0e45b40aa9e8344eb9301b5b82c103e263347bf6996813ef6c8df302727754b955c20afa1c69f3784c0a2432d5
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-src.tar.gz) | 1568c2f0464dd4c3c99e636dcd8ff6ec7716ae0c7e2c6bcb0b98cf30006f282bc011a2296a449026886f84ff7d37963f59e4cc5afdf45ec8d392b7d71a738f55
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-darwin-amd64.tar.gz) | a49da64f8408cd91e082ef199daf5f1d84460620a78c8f9a65ee0b1905a02b4f4ade2abe95e342291c4ea341be2dccb53cdd9b7f05ee79c33772c786f36e116f
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-darwin-arm64.tar.gz) | 5565934425a12c8a38e2270839624dcf617346ceca07c2b5f8fda25940c6361b6ec948babb2d02d855edb2fadaa57c12856a8f7fc67a34a606710486b326a4ce
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-linux-386.tar.gz) | 58f0a7342903350e25acca29ffb59851fff47c49e66a4d5f27e73b49baa570596741dc1989a53f0a84361d5dcc1f41a3bb3bf8369ee7c7ac85275056bd17e59b
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-linux-amd64.tar.gz) | 0d5e1d09eb0008a67b1d59aa63e8b6e7e7230ba1ab32cdac8a722188d166f5dc9008b595947c42aff8a410596ece0a4346cd19ac9ab3a2913cee0eaab127b238
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-linux-arm.tar.gz) | eae7aa6f40b94dd3d098f3a5c788e1b9dc3051a055b6bac64a602c9ebfcc70645231269c0322abf94f9fd65348c16c0be78385d324e80e00ef4843b0f82cd49f
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-linux-arm64.tar.gz) | 95192e41d73e4b0585b8fd54e4c79c92c1ed9d37b80dfa8c8d3a1e289b5a7c32d67de4f5fce193e5f0fc82867f7c6f73d75fdc516ca437c236930eff90106088
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-linux-ppc64le.tar.gz) | 3e4289893cb3f8492d99b77d09eddaa3d55ee2cbf6a70c7ff1c9ce38f6b744c62c545a55d01d43d716876c648f628044945e90c361a82cc1a008ae808b29c92b
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-linux-s390x.tar.gz) | 8a2b4a3180752612dd636c26853e9b23fb7a58a8b631f6f29172c3002ebdf7086d7def21a187ced179fda4d08a613390dee1a0dae46e5d136c9cb8813da54049
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-windows-386.tar.gz) | 57c0c4dfffe7e81d5144ee672d200e8204aab635701b2418e6cdb6eb130a65bbe08f7651dfdad7ae047818a33fb37aad2b7d6bfbb2d853c35108f1462c3cfa27
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-windows-amd64.tar.gz) | 3607e1ce781dcd636de8bc7f470257675c6e333bb6a56948016463b2581b2ed4bad1f7b19edf5e0fccaa767d4cc57fdc196f0ce18182001901f0084bd8c5b98b
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-client-windows-arm64.tar.gz) | a1fcacceccf712c752521a505e14aed113c75f40b690293e9f5411c5bfaeab9946cd2f067cdf7c4e5f57407c104feb9f4fa6a74c763653c460b89cae4a0d317e
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-server-linux-amd64.tar.gz) | cca2f7a1aa100c2ccc789536f27015848a45c7261523e605a2dcd0d49a06db85320706725c7f34cb9e90402f6d3349798a6d62c160e6811acb7c5bccd54aaff2
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-server-linux-arm64.tar.gz) | b5b2705a45d0ce2bf7bc3b2a5854796497b5b88a77aabae162fd6fb9e20c1fa71ba620c3183d4098a4f2f9f029406e3c5a36c9d21ac2471e51029fbee984c3db
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-server-linux-ppc64le.tar.gz) | 9b8637025481cb4774491634b62d03f94e152728710fe2e3b08a5f895d1d2902ec287f73a07ef441e16c5af8f3fc20659eec478f12f816948d2229123378a0f2
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-server-linux-s390x.tar.gz) | 78b2eb80422e23e7124f54103b83438d1ef18f25dd7c0d598f42f2aac5982f49165fee7da5a0d407ac81dda11161bc6a676cb4ce39fdcd5b00347662848f7428
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-node-linux-amd64.tar.gz) | fa04b8aaaa0c8cf9eec44f5a7b623547ab4821db45c6cd8c877b2eb0b6419c5f5ee5f2181af5bad9d1017811e1ea7b78362e1d0d6ba455e5c0cf899f2ce7d996
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-node-linux-arm64.tar.gz) | f18549315c58a86b4ec96eb04ea3dc10e3c01f9b835f721d8d20de4053e345e2789d7a4c211422a10f0f2b0ec3f12535766f61aabf174f400454434d4534c649
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-node-linux-ppc64le.tar.gz) | 03bbf9a6053a42fef41048470cd4e8956bb0cd6b3c407ce86f5192f2a2c95c97ad9c9823d7e257882bf4cf2489b5225a3f49e7948ff7f1e758d4084faef0c1b8
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-node-linux-s390x.tar.gz) | 2495e2810f763174102884d25e932d2dcd6984adf1bfe6733837c23174b20ee8fe47d6a9961ccc86fe233a161544d153a462e71badf6acc8d4f89513d82bcd37
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.3/kubernetes-node-windows-amd64.tar.gz) | 0e29716b0e16bd3aa3e85d85fb6ddc0de895a7ad4b0d6aacc5503c93b12f67139138f92b2b0dbf91001f32a3c9ac31a0f9116cfff3701d3546ac204f58791c0a
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.28.2
## Changes by Kind
### Feature
- Kubernetes is now built with Go 1.20.10 ([#121153](https://github.com/kubernetes/kubernetes/pull/121153), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing]
- Kubernetes is now built with Go 1.20.9 ([#121025](https://github.com/kubernetes/kubernetes/pull/121025), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing]
### Failing Test
- E2e framework: retrying after intermittent apiserver failures was fixed in WaitForPodsResponding ([#120559](https://github.com/kubernetes/kubernetes/pull/120559), [@pohly](https://github.com/pohly)) [SIG Testing]
### Bug or Regression
- Adds an opt-in mitigation for http/2 DOS vulnerabilities for CVE-2023-44487 and CVE-2023-39325 for the API server when the client is unauthenticated. The mitigation may be enabled by setting the `UnauthenticatedHTTP2DOSMitigation` feature gate to `true` (it is disabled by default). An API server fronted by an L7 load balancer that already mitigates these http/2 attacks may choose not to enable the kube-apiserver mitigation to avoid disrupting load balancer → kube-apiserver connections if http/2 requests from multiple clients share the same backend connection. An API server on a private network may choose not to enable the kube-apiserver mitigation to prevent performance regressions for unauthenticated clients. Authenticated requests rely on the fix in golang.org/x/net v0.17.0 alone. https://issue.k8s.io/121197 tracks further mitigation of http/2 attacks by authenticated clients. ([#121196](https://github.com/kubernetes/kubernetes/pull/121196), [@enj](https://github.com/enj)) [SIG API Machinery]
- Fix 1.28.0 regression where adding aggregated APIService objects could cause apiserver to panic and affect the health check ([#121040](https://github.com/kubernetes/kubernetes/pull/121040), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery and Testing]
- Fix a bug in cronjob controller where already created jobs may be missing from the status. ([#120649](https://github.com/kubernetes/kubernetes/pull/120649), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps]
- Fixed a 1.28.0 regression where kube-controller-manager can crash when StatefulSet with Parallel policy and PVC labels is scaled up. ([#121184](https://github.com/kubernetes/kubernetes/pull/121184), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) [SIG Apps]
- Fixed a bug where containers would not start on cgroupv2 systems where swap is disabled. ([#120924](https://github.com/kubernetes/kubernetes/pull/120924), [@klueska](https://github.com/klueska)) [SIG Node]
- Fixed a regression in kube-proxy where it might refuse to start if given
single-stack IPv6 configuration options on a node that has both IPv4 and
IPv6 IPs. ([#121008](https://github.com/kubernetes/kubernetes/pull/121008), [@danwinship](https://github.com/danwinship)) [SIG Network]
- Fixed an issue to not drain all the pods in a namespace when an empty-selector i.e. "{}" is specified in a Pod Disruption Budget (PDB) ([#121131](https://github.com/kubernetes/kubernetes/pull/121131), [@sairameshv](https://github.com/sairameshv)) [SIG Apps]
- Fixed attaching volumes after detach errors. Now volumes that failed to detach are not treated as attached, Kubernetes will make sure they are fully attached before they can be used by pods. ([#120595](https://github.com/kubernetes/kubernetes/pull/120595), [@jsafrane](https://github.com/jsafrane)) [SIG Apps and Storage]
- Fixed bug to surface events for the following metrics: apiserver_encryption_config_controller_automatic_reload_failures_total, apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds, apiserver_encryption_config_controller_automatic_reload_success_total ([#120544](https://github.com/kubernetes/kubernetes/pull/120544), [@ritazh](https://github.com/ritazh)) [SIG API Machinery, Auth and Testing]
- Fixes a bug where Services using finalizers may hold onto ClusterIP and/or NodePort allocated resources for longer than expected if the finalizer is removed using the status subresource ([#120654](https://github.com/kubernetes/kubernetes/pull/120654), [@aojea](https://github.com/aojea)) [SIG Testing]
- Revised the logic for DaemonSet rolling update to exclude nodes if scheduling constraints are not met.
This eliminates the problem of rolling updates to a DaemonSet getting stuck around tolerations. ([#120785](https://github.com/kubernetes/kubernetes/pull/120785), [@mochizuki875](https://github.com/mochizuki875)) [SIG Apps and Testing]
- Sometimes, the scheduler incorrectly placed a pod in the "unschedulable" queue instead of the "backoff" queue. This happened when some plugin previously declared the pod as "unschedulable" and then in a later attempt encounters some other error. Scheduling of that pod then got delayed by up to five minutes, after which periodic flushing moved the pod back into the "active" queue. ([#120334](https://github.com/kubernetes/kubernetes/pull/120334), [@pohly](https://github.com/pohly)) [SIG Scheduling]
### Other (Cleanup or Flake)
- Fixes an issue where the vsphere cloud provider will not trust a certificate if:
- The issuer of the certificate is unknown (x509.UnknownAuthorityError)
- The requested name does not match the set of authorized names (x509.HostnameError)
- The error surfaced after attempting a connection contains one of the substrings: "certificate is not trusted" or "certificate signed by unknown authority" ([#120768](https://github.com/kubernetes/kubernetes/pull/120768), [@MadhavJivrajani](https://github.com/MadhavJivrajani)) [SIG Architecture and Cloud Provider]
- Set the resolution for the job_controller_job_sync_duration_seconds metric from 4ms to 1min ([#120667](https://github.com/kubernetes/kubernetes/pull/120667), [@mimowo](https://github.com/mimowo)) [SIG Apps and Instrumentation]
## Dependencies
### Added
_Nothing has changed._
### Changed
- github.com/vmware/govmomi: [v0.30.0 → v0.30.6](https://github.com/vmware/govmomi/compare/v0.30.0...v0.30.6)
- golang.org/x/crypto: v0.11.0 → v0.14.0
- golang.org/x/net: v0.13.0 → v0.17.0
- golang.org/x/sys: v0.10.0 → v0.13.0
- golang.org/x/term: v0.10.0 → v0.13.0
- golang.org/x/text: v0.11.0 → v0.13.0
### Removed
_Nothing has changed._
# v1.28.2
## Downloads for v1.28.2
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes.tar.gz) | f6e13f2632697aab3ce6230d777240dd3d9c23b65eba7ff7d1df5d330e4dd926f8f439b77d823f8d08f44ddcd7eeca476af6d83eaa29cf623e86f2e4f315074a
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-src.tar.gz) | 3c21536962251eb199e4b0f42379cdfa172e826b10a28d8946df23bb8bae5e12d09647448d1f7a9c7146166178dd38398dee308dcc1e604000be908e1e0bbe89
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-darwin-amd64.tar.gz) | b85d0f2da76708a934cb8cecdf08a2d7c146c8f8209f49deab82b01c15842cf3c0631e01977af20230e69d478dfb21b5bf6acf9fc985d9ae27d1126f7a9f1112
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-darwin-arm64.tar.gz) | 0e556a34b3c659c45a9368b8b0d709831bdfa6562adb48dd5c924085806f3e1b76d4ba3b5dd719bc2d126f1bd640ddc94ddbd37515168a4de1a358b4605e031a
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-linux-386.tar.gz) | a091434bc89f762655fb76e301c9287297ab48079118eb045589d6ef246fcda307a1799732178c568ea8c64211b5bdae3ba7a836bdd60d1c61bb7dcb7b7ee324
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-linux-amd64.tar.gz) | ab534cd06d8cc89d1288590cfae98415facaa7db2f481d8f6be0a20574d2990cc55348cf98386c34df7788aa80ff018fc844816a2b605bcb350a82d752738fdc
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-linux-arm.tar.gz) | 31fb2570bd4dff5ad9f6525e33fa80847ee35d2804a1c81af8ce27855a1b4d8267bea3b522ac90bd49bc5c6a9a9fd9388a14899ebd48baac214a98644f357e02
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-linux-arm64.tar.gz) | 2304f6888752dd22e898526df091b66aae85835690e922a8e017d57e077dd1c8fcdfca16fc5aba94e9fb51ad800832305a574aee24c6cfa5d37278c8e28a144e
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-linux-ppc64le.tar.gz) | 5dbb4fefd197b14ccdc3b82d088b0e09987b1a8afbf47e03abb6707c455716658ed95d7e2dce7e5d7e12981febd53eebd0ed6296d282d2ce317c191a2c8116e9
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-linux-s390x.tar.gz) | 455e436244ff306604eb5d8a230a24186799ef5c462b7f278bb2d62e36245639db7a0d89241682dc201351a4c648ba788acdd7ac73a486cd6db426e1079ed87b
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-windows-386.tar.gz) | 96a4cf768fb59ad626a383bddcc9bab433f9d309ba3b06a8c9d927799a5ee6c4645412d22a190255c9a3e7104bf6d914f5976fcdd39876bdf1598c72dabb68a7
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-windows-amd64.tar.gz) | e7075cf3f0103edd22962fdb5a9adb4f80249a4adc8309794fc15f2f3d8e934feee6adc47c1724cc3a1c497d9cc0d4afbe0d66511042670cd2f1da1d82c70ba3
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-client-windows-arm64.tar.gz) | d230193bc1f73f6834f0a9e919b673b7fcd645343773d9cf05b33ce95d81d5ed6f8efa1620e7099e8291beab47a2f3e058dde25a96fd0873b2de2ab28e7c3b7d
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-server-linux-amd64.tar.gz) | cd54f2f48733806208d3b1585ed307bebf91893e250a1eb3e18355c9a9e6d1f75a70966cb66165be2fbb8566e5368b14f66f63a8929fbceca30aa73bfd491441
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-server-linux-arm64.tar.gz) | e879568fb40ac54b897ac52e39aa3077278f9ea502c4b7d639a247d503be85db623c264cc30300016ac1651db2c93bc82420decb5e904af396f54e40d3aa33ff
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-server-linux-ppc64le.tar.gz) | 212ef04a6d443f239fa6ecb34df5f4fc93f172dab5a5d2931a0554eafdd3901e8504f6e8918712c44f843f3be55adb83c241839d60a9de1d4b6988035f47dc6d
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-server-linux-s390x.tar.gz) | b7efe45fbb811ee09e0c8daec0b608546f065c389e2b480c0d5178ea778690f11fc3d880a84baba2331bb30042898a616b5a4f7934456543aaa9b3fc0169f923
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-node-linux-amd64.tar.gz) | adafa3beb4525d898a602ec87f1d2b35ad89f71f6b89aad81ee0cdff97a2916ef323a21b5007d61bb0453e12d3058ad21861184049eb22c26b86be4dc268e7d9
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-node-linux-arm64.tar.gz) | c9ece93728bc2712004d508c37e692add5cc358437ee0b209ef062b81db137cccf85bf9f4560c6436c0b83193a4ba23fa28e06dd432281b946f235aa30e88842
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-node-linux-ppc64le.tar.gz) | 85918f6235563c10d58ae18e0a1918011492c4f812a753ba185447b6080aae88ccf94c6008519694fb9fca4b2a9cc02dec6ae91f1ca55a489b383d85c8f7fa9c
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-node-linux-s390x.tar.gz) | ea59ee997d3df3d89405be4943dbe8f76cc8066e98173db30f0f83ad06a9c60c73217004f837f14310bf17228fd0c4e2d2917632ddda3cc3e483c353edc38745
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.2/kubernetes-node-windows-amd64.tar.gz) | b8125003fcd8fa7e89d7cbc49f634cc01cd8fdf198a3325f22e5bb3743923e03e82c3d4a7545b32b0da82c950b8d887997e502839359027f194de4a0a4774654
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.28.1
## Changes by Kind
### API Change
- Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps.
The incorrect cost was evident when the result of a function was used in subsequent operations. ([#119807](https://github.com/kubernetes/kubernetes/pull/119807), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth and Cloud Provider]
- Mark Job onPodConditions as optional in pod failure policy ([#120208](https://github.com/kubernetes/kubernetes/pull/120208), [@mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]
### Feature
- Kubernetes is now built with Go 1.20.8 ([#120495](https://github.com/kubernetes/kubernetes/pull/120495), [@cpanato](https://github.com/cpanato)) [SIG Release and Testing]
### Bug or Regression
- Fix OpenAPI v3 not being cleaned up after deleting APIServices ([#120108](https://github.com/kubernetes/kubernetes/pull/120108), [@tnqn](https://github.com/tnqn)) [SIG API Machinery and Testing]
- Fix a 1.28 regression in scheduler: a pod with concurrent events could incorrectly get moved to the unschedulable queue where it could got stuck until the next periodic purging after 5 minutes if there was no other event for it. ([#120445](https://github.com/kubernetes/kubernetes/pull/120445), [@pohly](https://github.com/pohly)) [SIG Scheduling]
- Fix a concurrent map access in TopologyCache's `HasPopulatedHints` method. ([#120372](https://github.com/kubernetes/kubernetes/pull/120372), [@Miciah](https://github.com/Miciah)) [SIG Network]
- Fixed a 1.26 regression scheduling bug by ensuring that preemption is skipped when a PreFilter plugin returns `UnschedulableAndUnresolvable` ([#119951](https://github.com/kubernetes/kubernetes/pull/119951), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling]
- Fixed a 1.27 scheduling regression that PostFilter plugin may not function if previous PreFilter plugins return Skip ([#119942](https://github.com/kubernetes/kubernetes/pull/119942), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling and Testing]
- Fixed a 1.28 regression around restarting init containers in the right order relative to normal containers ([#120440](https://github.com/kubernetes/kubernetes/pull/120440), [@gjkim42](https://github.com/gjkim42)) [SIG Node and Testing]
- Fixed a regression in default 1.27 configurations in kube-apiserver: fixed the AggregatedDiscoveryEndpoint feature (beta in 1.27+) to successfully fetch discovery information from aggregated API servers that do not check `Accept` headers when serving the `/apis` endpoint ([#120359](https://github.com/kubernetes/kubernetes/pull/120359), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery]
- Fixes a 1.28 regression handling negative index json patches ([#120329](https://github.com/kubernetes/kubernetes/pull/120329), [@liggitt](https://github.com/liggitt)) [SIG API Machinery]
- Fixes a bug where images pinned by the container runtime can be garbage collected by kubelet. ([#120053](https://github.com/kubernetes/kubernetes/pull/120053), [@ruiwen-zhao](https://github.com/ruiwen-zhao)) [SIG Node]
- Ignore context canceled from validate and mutate webhook ([#120155](https://github.com/kubernetes/kubernetes/pull/120155), [@divyasri537](https://github.com/divyasri537)) [SIG API Machinery]
- Kubeadm: fix nil pointer when etcd member is already removed ([#120010](https://github.com/kubernetes/kubernetes/pull/120010), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle]
## Dependencies
### Added
_Nothing has changed._
### Changed
- github.com/evanphx/json-patch: [v5.6.0+incompatible → v4.12.0+incompatible](https://github.com/evanphx/json-patch/compare/v5.6.0...v4.12.0)
- github.com/google/cel-go: [v0.16.0 → v0.16.1](https://github.com/google/cel-go/compare/v0.16.0...v0.16.1)
### Removed
_Nothing has changed._
# v1.28.1
## Downloads for v1.28.1
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes.tar.gz) | 87829907851a0f75bdca725d89c1290ca898fb92c0c6271fc6ddda8c06361bb0131306700b5c7c96d9f083b223e61e3d4cc55479de21cd4de64ab942ce2f91e2
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-src.tar.gz) | 52297b3ef9082933d55973bddd54249dccb0b3aeb6ca131d80a35d6c60c70711042b6373d66fb1bf9e79046a4a366060a31faef0033ae5e29f14df8e5f1c6f87
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-darwin-amd64.tar.gz) | 2a0bdb8dffb4d710ee01ca48437da9f80f8568efb4c9ac7a13b201d307e9f68698b2c102b8fcd2f59e4082cd330229ac5febfb6e99dea2a90c2aa93aa176d720
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-darwin-arm64.tar.gz) | 7824c1907510fda1a91870f55adad9ff4d5e7b01d2cfb9805e7b8c56c24aaa4b6aefd3ad5f374eb6f97056142d5d5ea9ed552347d293789eafe3bc56fcd17326
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-linux-386.tar.gz) | ef812e222d771bb29209d3a66e4168c0f03922b3cc8585151833ce304f30b124e97cec27fa2ae5abe990bf901f27db81835c817e70ae6738d2b58e863854e996
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-linux-amd64.tar.gz) | b5e9823886c8c26c22078cf5cd233612f38240e5ceb3c7bc5c032fbbfee59f6a631b53aa541bf8afc2eba496f5d0476357d1738cf771aaa95661c83d91372b51
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-linux-arm.tar.gz) | fae089315283125f5a30103260319daefaf092aa488175000ad19ff55bb90b62f9b7d3b9fe446b3103087f06fd353d96f521a8f8f33cd2cf3e0887183ff3087f
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-linux-arm64.tar.gz) | a46645ca155fb3a98159678b92f0075a14ea731a0e0feebc1f2f243e2b50f7413272969aabd6feb7a68f1d64de66fb330e389edc326c8ec90457b3c7c9b25783
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-linux-ppc64le.tar.gz) | 45259cabe085abba6d0255bdecd6890c89a098a7dcc8f65013acdaf1471a013d4671c04b047f36c160b7623e476266c375e0c87e3e1d3ba666c9f27a184bfc78
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-linux-s390x.tar.gz) | 72f8a4f15745bec71328bd3ec346cf886b0a56bc409bc0c756828bfa74e6e52e7bf444a40da133a9de899df60c2e44082fd60e5532d9f1fef8c948827e8ef51e
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-windows-386.tar.gz) | 7a2b3d717f586b14720c97642a2f7a8bbb6720f874a4dbb22a8f450598017e3247ed1c1e861993a489fedb10ebd782aa074182061323b0c3509ace10534b9a3f
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-windows-amd64.tar.gz) | ebc759a5164fbe9c292db46c9ed2070d6162a2ba7bc2d246d2538d5d9322ca08d309507e428b954616e0c4fe21db32671300970bb0fbd8286e85461d9dcc4487
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-client-windows-arm64.tar.gz) | a77c2ed43148fe914d483862481dbe70ef459bf56d1c4b7a75e4146b67be41b5d54756b94414ad6013c31a13affe13594600462551a64bc0ce82883b8b44cc35
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-server-linux-amd64.tar.gz) | 5727d958896b8032667111da019b0ea39ed1040f966c6758180f5f42656c5c78965cd61c162d64ae9df830fcfaddf61c1d05433430facb20fe2b96e4e1a4716f
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-server-linux-arm64.tar.gz) | ab3df8c2d10a91ab155b2c615b4ae95650d949294503a3863fd93181417287e50a4b1b3641e48e6f729c2fa666b4d62e620234841c5047ed9789d567cb600a37
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-server-linux-ppc64le.tar.gz) | 4a8a86e10b90944fbb0a80f55b99bba77ff5e82806cb11a13286c9e434523eee3723960ad9615a5d44e74e693041575624f2e82132c17441d9faa4bd21170a59
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-server-linux-s390x.tar.gz) | 12ebaeba75861087d058cbf65b2853d54a802d61408f7fd05480d0f00ebb06240d00b84cbfc445f66478f359deb65a265e0108f7e4f1f82dc664c12be82a17f9
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-node-linux-amd64.tar.gz) | d736a6abfeec797a48944caf926c737fb67e9cb2fa20913b4d5c2b1b12b2aca550a82bb4a20836f15fe8c4054e5fe7aebd3f380845054eaad9774fd68296ac48
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-node-linux-arm64.tar.gz) | c1f7984b4f173c98bc0020348848eee50a42193fb49803229bc79efafc89ea4e9d23398aea96f393265e6ba2705b0c0edb1372b0052f6567b455a12798ec9f4e
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-node-linux-ppc64le.tar.gz) | 94f4fe0cc6185d8ee361b04c49abdbd768863877f54bb641eb02c787f8d6011ad6dfc074e496c1a9004f9fa3143e1d546f3904059ded0866d9d4ab9eca0ab670
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-node-linux-s390x.tar.gz) | e81b885ea5eb00628f67715f72bcd4b9d5d143b0bef4f9a40f592c3ee7ffefb01a35241e4801c3a3e1e353944eb91797c9346e834410478345a781e8b8ae1b40
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.1/kubernetes-node-windows-amd64.tar.gz) | 0b049a819495249820c64f7b728b6abf31548c977a501cd4a9a60ab60bed35c688677db443002735033dbb3a2f6ae13842b07b81c7f72eb3d13f2f900f4d421a
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.28.0
## Important Security Information
This release contains changes that address the following vulnerabilities:
### CVE-2023-3955: Insufficient input sanitization on Windows nodes leads to privilege escalation
A security issue was discovered in Kubernetes where a user that can create pods on Windows nodes may be able to escalate to admin privileges on those nodes. Kubernetes clusters are only affected if they include Windows nodes.
**Affected Versions**:
- kubelet <= v1.28.0
- kubelet <= v1.27.4
- kubelet <= v1.26.7
- kubelet <= v1.25.12
- kubelet <= v1.24.16
**Fixed Versions**:
- kubelet v1.28.1
- kubelet v1.27.5
- kubelet v1.26.8
- kubelet v1.25.13
- kubelet v1.24.17
This vulnerability was discovered by James Sturtevant @jsturtevant and Mark Rossetti @marosset during the process of fixing CVE-2023-3676 (that original CVE was reported by Tomer Peled @tomerpeled92)
**CVSS Rating:** High (8.8) [CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H](https://www.first.org/cvss/calculator/3.1#CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H)
### CVE-2023-3676: Insufficient input sanitization on Windows nodes leads to privilege escalation
A security issue was discovered in Kubernetes where a user that can create pods on Windows nodes may be able to escalate to admin privileges on those nodes. Kubernetes clusters are only affected if they include Windows nodes.
**Affected Versions**:
- kubelet <= v1.28.0
- kubelet <= v1.27.4
- kubelet <= v1.26.7
- kubelet <= v1.25.12
- kubelet <= v1.24.16
**Fixed Versions**:
- kubelet v1.28.1
- kubelet v1.27.5
- kubelet v1.26.8
- kubelet v1.25.13
- kubelet v1.24.17
This vulnerability was reported by Tomer Peled @tomerpeled92
**CVSS Rating:** High (8.8) [CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H](https://www.first.org/cvss/calculator/3.1#CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H)
## Changes by Kind
### Other (Cleanup or Flake)
- Fixes ability to build 1.28 without network access ([#119982](https://github.com/kubernetes/kubernetes/pull/119982), [@liggitt](https://github.com/liggitt)) [SIG Testing]
## Dependencies
### Added
_Nothing has changed._
### Changed
_Nothing has changed._
### Removed
_Nothing has changed._
# v1.28.0
[Documentation](https://docs.k8s.io)
## Downloads for v1.28.0
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes.tar.gz) | `9aaf7cc004d09297dc7bbc1f0149d6424d85717b0f31658997bf9a1eee7343ad1ede25e506e1b85956f6b08393d5c7b58e59de860c2f880d97544fd79dfae9da`
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-src.tar.gz) | `8e9071210316caac1762535d9437a1e7dcbf644ae8852f4d5babd19a743810c3a2ff2b30f009ba47d28a50e7f5691e56f9b273444bc387dbe95be56c71f2c9d4`
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-darwin-amd64.tar.gz) | `e9282dc96a73431ed50ef0c515be2a7953b4a243d42ce817e282006aa0431f4f3909971701c4847ea2dd9f268de13fbca40424eff6316f7697faebc2bc0fcff6`
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-darwin-arm64.tar.gz) | `d0c24710717979494d32e9e518435e0803c297a369a1e5022fb522df6a2d4cf91112ad2a3e583abc85080c8ead9644dbb42a6387518a834e5f3d93ca097d0977`
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-linux-386.tar.gz) | `8f55c1fa60e50c40a81551ab2d6fdf9327a57c445e281105f426ff686395097219766869edef1935b299feb58ab01c9612a1efe3cd1ef06bdc09fc2f93f2ba3a`
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-linux-amd64.tar.gz) | `0506d82a49794089137e0f240deb3bbaf48be6e61b5b242af02a0d2a0f94fb1df55fcb87570cf40f9abec6b2d6bf11d40fa5d66ab0829ea43169448bda2609d1`
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-linux-arm.tar.gz) | `3db12e083be8f7a7555a391311f78064a0ffa51f5cfdb6509ee71f33ee5bf56d986e687c5eb39e01c9def7b2154b0298a41c0b960c1fc76228b99c39546529c9`
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-linux-arm64.tar.gz) | `c907332aa6b38ecd82ed7c16741d80e8f23857b49ebff05bdb89692ab286955e03537cc2aba631e932c56bcbcc2aafee2a9b69c5bb6496b869d31b771dc93759`
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-linux-ppc64le.tar.gz) | `fccda39ca81d171ff8bf30a27596ea6e13a6fca7113c3b46f1c2915d50a367b96b6db2e7d8e27fd76c5b3b00f3d447b1da4d1a70fbaf652a7b2b2c4aae71853f`
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-linux-s390x.tar.gz) | `f177677ba4c7e59901ba5cbc10d23384e69cabbbe7f924d0bf0267534eefca4ed0b855ad7193533f5034b080d9894278393b9012b008dd17056d9684aa36e7d3`
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-windows-386.tar.gz) | `bc3cc07e5101c1d1b4484f28b748b03083ad8b1a74c51d67b7c9d628c65a9db07d20f0695f458508a88a259d1f4396b2008d898476716998a32d74dd84901320`
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-windows-amd64.tar.gz) | `6969e153438cfc3165530562b4bb4cb620588a8b59f1f28bb6a369d7e4ba5f636613d3cab2925d1b00385239ac82bb46bffa00cbbadc3aed1ab54ed620909de2`
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-client-windows-arm64.tar.gz) | `3955501ba210e0af25df1431e8b9f641b6d746d0fb882192a24b1d2b18b55ddb48e0c3ab086a0f6e0bf2156a38e11b979f78d5d7b898e24dc34e10f1d558abb0`
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-server-linux-amd64.tar.gz) | `36476c4c64f23b3a48d6e79885ddc764102eefd2ab25b1d721386bdfc15c954d7940ab905615bec357dc5530610ecf7b8e640790206cb5a8da8312cea46db97a`
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-server-linux-arm64.tar.gz) | `42c17eb2229f4210521bb78ff7712de9ac4a3e483b503e4a4f9a889a4aea3015890360242d88e9f2dbcb3d4d645bbfc37b31a5d2f151023b594392c23d1b0154`
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-server-linux-ppc64le.tar.gz) | `e8929e915e61cb9380f6e48abf21f6dfd00b52d7afc1867f8999f55fdf8c60404397dcec51f6feda7bd281fdb15035516a3187162349320460574b14c2a63f25`
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-server-linux-s390x.tar.gz) | `88bce78171ff9b12796ccc3e6781a09afb05b7412dcf154aa1b9a9da151a01de07e8ef3a35e305822a67fbf0dfd482bc7320d0b7a39c6384f74139153b0644f8`
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-node-linux-amd64.tar.gz) | `fed9ba533e54a4dd6cf26788c27d4f41534ee4f6cf22ee75b183afc45764273e8ac008f06297608342797bc9463c82603947800dce37155424489e20987d3dea`
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-node-linux-arm64.tar.gz) | `a500906699bc25a8c0825fd38e92b1ce5fddbb3bfc09561e21cfd04bab0cefe6430231b9f347835c8e1d06ffb926b72ae272c119eb72d64580b318d7fcacad20`
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-node-linux-ppc64le.tar.gz) | `16bb9a4762fdb4f5140cf518a93d812ddb04c08cc98f0447d1c540d290648a8a050d2d6133e244b40645bb25813d149a96a313de5af178ed30a5dab2919fa845`
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-node-linux-s390x.tar.gz) | `fb2cf3f819f8c3329fbfc13588a8b206bb16e3b4e351680ed03e3a74cc34b42341743f8f913941e25ed3ca2d7779bf331f31f30821787b1f8cb916f58f183ab4`
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0/kubernetes-node-windows-amd64.tar.gz) | `60e964a33f10cf0b361f50199aa19f0d89dd82073b31377e7d61b6fb761ef779cc3577bf407edb448c104333185d25eba8d8953e917f52cb62b504ea85121ac4`
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.27.0
## Urgent Upgrade Notes
### (No, really, you MUST read this before you upgrade)
- Action required for the custom scheduler plugin developers.
Here's the breaking change in `EnqueueExtension` in the scheduling framework.
The `EventsToRegister` in `EnqueueExtension` changed the return value from `ClusterEvent` to `ClusterEventWithHint`. `ClusterEventWithHint` allows each plugin to filter out more useless events via the callback function named `QueueingHintFn`.
When the scheduling queue receives a cluster event, before moving each Pod from unschedulable pod pool to activeQ/backoffQ, it will call QueueingHintFn of plugins that rejected each Pod in the previous scheduling cycle.
Depending on the value returned from QueueingHintFn, the scheduling queue changes how it queues each Pod:
- if more than one QueueingHintFn returns QueueImmediately, it queues Pod to activeQ.
- If no QueueingHintFn returns QueueImmediately and more than one plugin returns QueueAfterBackoff, it queues Pod to backoffQ if Pod is backing off, or to activeQ if Pod's backoff has already finished.
- If all QueueingHintFn return QueueSkip, it puts this pod back to the unschedulable pod pool
Having appropriate QueueingHintFn contributes to reducing useless retries and thus improves the overall scheduler's performance.
**How can I migrate?**
For backward compatibility, nil `QueueingHintFn` is treated as always returning QueueAfterBackoff.
So, if you want to just keep the existing behavior, you can register `ClusterEventWithHint` with no `QueueingHintFn` in it.
But, registering appropriate `QueueingHintFn` is, of course, better from a scheduling performance perspective. ([#118551](https://github.com/kubernetes/kubernetes/pull/118551), [@sanposhiho](https://github.com/sanposhiho)) [SIG Node, Scheduling, Storage and Testing]
- CephFS volume plugin (`kubernetes.io/cephfs`) has been deprecated in this release and will be removed in a subsequent release. The alternative is to use the CephFS CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes cluster. ([#118143](https://github.com/kubernetes/kubernetes/pull/118143), [@humblec](https://github.com/humblec))
- Deprecated support for CSI migration of `Ceph RBD volumes`. Users who were relying on Kubernetes' ability
to migrate to an out-of-tree storage driver should complete that migration before the support for it is removed. ([#118303](https://github.com/kubernetes/kubernetes/pull/118303), [@carlory](https://github.com/carlory))
- RBD volume plugin (`kubernetes.io/rbd`) has been deprecated in this release
and will be removed in a subsequent release. Alternative is to use RBD CSI driver
(https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. ([#118552](https://github.com/kubernetes/kubernetes/pull/118552), [@humblec](https://github.com/humblec))
## Changes by Kind
### Deprecation
- Changed `kubectl version` default output to be identical to what `kubectl version --short` printed,
and removed `--short` flag entirely. ([#116720](https://github.com/kubernetes/kubernetes/pull/116720), [@soltysh](https://github.com/soltysh))
- Kube-controller-manager deprecate `--volume-host-cidr-denylist` and `--volume-host-allow-local-loopback` flags. ([#118128](https://github.com/kubernetes/kubernetes/pull/118128), [@carlory](https://github.com/carlory)) [SIG API Machinery, Apps, Network, Node, Storage and Testing]
- Kubelet: The `--azure-container-registry-config` flag has been deprecated and will be removed in a future release, please use `--image-credential-provider-config` and `--image-credential-provider-bin-dir` to setup acr credential provider instead. ([#118596](https://github.com/kubernetes/kubernetes/pull/118596), [@SataQiu](https://github.com/SataQiu)) [SIG Node]
- Removed tracking annotation from validation and defaulting. ([#117633](https://github.com/kubernetes/kubernetes/pull/117633), [@kannon92](https://github.com/kannon92))
- Removed withdrawn feature `NetworkPolicyStatus`. ([#115843](https://github.com/kubernetes/kubernetes/pull/115843), [@rikatz](https://github.com/rikatz))
- The deprecated flag `--lock-object-namespace` and `--lock-object-name` have been removed from kube-scheduler. Please use `--leader-elect-resource-namespace` and `--leader-elect-resource-name` or ComponentConfig instead to configure those parameters. ([#119130](https://github.com/kubernetes/kubernetes/pull/119130), [@SataQiu](https://github.com/SataQiu)) [SIG Scheduling]
- `KMSv1` is deprecated and will only receive security updates going forward. Use `KMSv2` instead. In a future release, Set `--feature-gates=KMSv1=true` to use the deprecated KMSv1 feature. ([#119007](https://github.com/kubernetes/kubernetes/pull/119007), [@aramase](https://github.com/aramase))
### API Change
- A CDIDevice field is included in the Device Plugin's `ContainerAllocateResponse`. This field maps to the CDIDevice field in the CRI protocol. ([#118254](https://github.com/kubernetes/kubernetes/pull/118254), [@elezar](https://github.com/elezar)) [SIG Node and Testing]
- ACTION_REQUIRED
When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. ([#118420](https://github.com/kubernetes/kubernetes/pull/118420), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps]
- Added `ServedVersions` field to `StorageVersion` API. ([#118386](https://github.com/kubernetes/kubernetes/pull/118386), [@Richabanker](https://github.com/Richabanker))
- Added `IP mode` field to loadbalancer status ingress. ([#118895](https://github.com/kubernetes/kubernetes/pull/118895), [@RyanAoh](https://github.com/RyanAoh))
- Added `podReplacementPolicy` and terminating field to job api. ([#119301](https://github.com/kubernetes/kubernetes/pull/119301), [@kannon92](https://github.com/kannon92))
- Added a new `namespaceParamRef` field to `admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy`. ([#119215](https://github.com/kubernetes/kubernetes/pull/119215), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing]
- Added a warning that TLS 1.3 ciphers are not configurable. ([#115399](https://github.com/kubernetes/kubernetes/pull/115399), [@3u13r](https://github.com/3u13r)) [SIG API Machinery and Node]
- Added error handling for seccomp localhost configurations that do not properly set a `localhostProfile`. ([#117020](https://github.com/kubernetes/kubernetes/pull/117020), [@cji](https://github.com/cji))
- Added fields `reason` and `fieldPath` into CRD validation rules to allow users to specify reason and field path when validation failed. ([#118041](https://github.com/kubernetes/kubernetes/pull/118041), [@cici37](https://github.com/cici37)) [SIG API Machinery]
- Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a `namespaceObject`
variable with expressions. ([#118267](https://github.com/kubernetes/kubernetes/pull/118267), [@cici37](https://github.com/cici37)) [SIG API Machinery and Testing]
- Added new `CRDValidationRatcheting` alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. ([#118990](https://github.com/kubernetes/kubernetes/pull/118990), [@alexzielenski](https://github.com/alexzielenski))
- Added new annotation `batch.kubernetes.io/cronjob-scheduled-timestamp` to Job objects scheduled from CronJobs. ([#118137](https://github.com/kubernetes/kubernetes/pull/118137), [@helayoty](https://github.com/helayoty))
- Added new config option `delayCacheUntilActive` to `KubeSchedulerConfiguration` that can provide a tradeoff between memory efficiency and scheduling speed when their leadership is updated in `kube-scheduler` ([#115754](https://github.com/kubernetes/kubernetes/pull/115754), [@linxiulei](https://github.com/linxiulei)) [SIG API Machinery and Scheduling]
- Changed how KMS v2 encryption at rest can generate data encryption keys.
When you enable the `KMSv2KDF` feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. ([#118828](https://github.com/kubernetes/kubernetes/pull/118828), [@enj](https://github.com/enj))
- Exposed `rest.DefaultServerUrlFor` function. ([#118055](https://github.com/kubernetes/kubernetes/pull/118055), [@timofurrer](https://github.com/timofurrer))
- Extended the Job API for alpha version of `BackoffLimitPerIndex`. ([#119294](https://github.com/kubernetes/kubernetes/pull/119294), [@mimowo](https://github.com/mimowo))
- Graduated `AdmissionWebhookMatchCondition` feature to beta. ([#119380](https://github.com/kubernetes/kubernetes/pull/119380), [@a-hilaly](https://github.com/a-hilaly))
- If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via `memory.oom.group` . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. ([#117793](https://github.com/kubernetes/kubernetes/pull/117793), [@tzneal](https://github.com/tzneal)) [SIG Apps, Node and Testing]
- In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . ([#118782](https://github.com/kubernetes/kubernetes/pull/118782), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery]
- Indexed Job pods now have the pod completion index set as a pod label. ([#118883](https://github.com/kubernetes/kubernetes/pull/118883), [@danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps]
- Kube-proxy: added `--logging-format` flag to support structured logging. ([#117800](https://github.com/kubernetes/kubernetes/pull/117800), [@cyclinder](https://github.com/cyclinder))
- NodeVolumeLimits implement the `PreFilter` extension point for skipping the Filter phase if the Pod doesn't use volumes with limits. ([#115398](https://github.com/kubernetes/kubernetes/pull/115398), [@tangwz](https://github.com/tangwz)) [SIG Scheduling]
- PersistentVolumes have a new `LastPhaseTransitionTime` field which holds a timestamp of when the volume last transitioned its phase. ([#116469](https://github.com/kubernetes/kubernetes/pull/116469), [@RomanBednar](https://github.com/RomanBednar))
- Pods which set `hostNetwork: true` and declare ports, get the `hostPort` field set automatically. Previously this would happen in the PodTemplate of a Deployment, DaemonSet or other workload API. Now `hostPort` will only be set when an actual Pod is being created. If this presents a problem, setting the feature gate "DefaultHostNetworkHostPortsInPodTemplates" to true will revert this behavior. Please file a kubernetes bug if you need to do this. ([#117696](https://github.com/kubernetes/kubernetes/pull/117696), [@thockin](https://github.com/thockin)) [SIG Apps]
- Promoted API groups `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding` to `v1beta1`. ([#118644](https://github.com/kubernetes/kubernetes/pull/118644), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Apps and Testing]
- Promoted the feature gate `ValidtaingAdmissionPolicy` to beta, and it is turned off by default. ([#119409](https://github.com/kubernetes/kubernetes/pull/119409), [@alexzielenski](https://github.com/alexzielenski))
- Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to `BETA` stability. ([#119264](https://github.com/kubernetes/kubernetes/pull/119264), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]
- Removed `resizeStatus` enum from `pvc.Status` and replaced with `AllocatedResourceStatus`. ([#116335](https://github.com/kubernetes/kubernetes/pull/116335), [@gnufied](https://github.com/gnufied)) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
- Removed `WindowsHostProcessContainers` feature-gate. ([#117570](https://github.com/kubernetes/kubernetes/pull/117570), [@marosset](https://github.com/marosset)) [SIG API Machinery, Apps, Auth, Node and Windows]
- Revised the comment about the feature-gate level for `PodFailurePolicy` from alpha to beta. ([#117802](https://github.com/kubernetes/kubernetes/pull/117802), [@kerthcet](https://github.com/kerthcet)) [SIG API Machinery and Apps]
- StatefulSet pods now have the pod index set as a pod label `apps.kubernetes.io/pod-index`. ([#119232](https://github.com/kubernetes/kubernetes/pull/119232), [@danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps]
- Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver ([#117740](https://github.com/kubernetes/kubernetes/pull/117740), [@Richabanker](https://github.com/Richabanker)) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing]
- Supported `BackoffLimitPerIndex` in Jobs. ([#118009](https://github.com/kubernetes/kubernetes/pull/118009), [@mimowo](https://github.com/mimowo))
- The `IPTablesOwnershipCleanup` feature (KEP-3178) is now GA; kubelet no longer
creates the `KUBE-MARK-DROP` chain (which has been unused for several releases)
or the `KUBE-MARK-MASQ` chain (which is now only created by kube-proxy). ([#119374](https://github.com/kubernetes/kubernetes/pull/119374), [@danwinship](https://github.com/danwinship))
- The `SelfSubjectReview` API is promoted to `authentication.k8s.io/v1` and the `kubectl auth whoami` command is GA. ([#117713](https://github.com/kubernetes/kubernetes/pull/117713), [@nabokihms](https://github.com/nabokihms)) [SIG API Machinery, Architecture, Auth, CLI and Testing]
- The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still `<pod>-<claim name>`, but a random suffix will avoid name collisions. ([#117351](https://github.com/kubernetes/kubernetes/pull/117351), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
- The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. ([#116429](https://github.com/kubernetes/kubernetes/pull/116429), [@gjkim42](https://github.com/gjkim42)) [SIG API Machinery, Apps, Node, Scheduling and Testing]
- Updated the comment about the feature-gate level for `PodFailurePolicy` from alpha to beta ([#118278](https://github.com/kubernetes/kubernetes/pull/118278), [@mimowo](https://github.com/mimowo))
- `client-go`: Improved memory use of reflector caches when watching large numbers
of objects which do not change frequently. ([#113362](https://github.com/kubernetes/kubernetes/pull/113362), [@sxllwx](https://github.com/sxllwx))
- `component-base/logs` is now stricter about not applying configurations multiple
times and will return an error when that is attempted. Can be overridden by binaries
which need to do that. ([#117108](https://github.com/kubernetes/kubernetes/pull/117108), [@pohly](https://github.com/pohly))
- `kube-controller-manager`: The `LegacyServiceAccountTokenCleanUp` feature gate
is now available as alpha (off by default). When enabled, the `legacy-service-account-token-cleaner`
controller loop removes service account token secrets that have not been used
in the time specified by `--legacy-service-account-token-clean-up-period` (defaulting
to one year), **and are** referenced from the `.secrets` list of a ServiceAccount
object, **and are not** referenced from pods. ([#115554](https://github.com/kubernetes/kubernetes/pull/115554), [@yt2985](https://github.com/yt2985))
- `kube-scheduler` component config (KubeSchedulerConfiguration) `kubescheduler.config.k8s.io/v1beta2`
is removed in `v1.28`. Migrate `kube-scheduler` configuration files to `kubescheduler.config.k8s.io/v1`. ([#117649](https://github.com/kubernetes/kubernetes/pull/117649), [@SataQiu](https://github.com/SataQiu))
### Feature
- A ValidatingAdmissionPolicy now has its `messageExpression` field checked against resolved types. ([#119209](https://github.com/kubernetes/kubernetes/pull/119209), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery]
- Added '--concurrent-cron-job-syncs' flag for `kube-controller-manager` to set the number of workers for cron job controller. ([#117550](https://github.com/kubernetes/kubernetes/pull/117550), [@borgerli](https://github.com/borgerli))
- Added '--concurrent-job-syncs' flag for `kube-controller-manager` to set the number of job controller workers. ([#117138](https://github.com/kubernetes/kubernetes/pull/117138), [@tosi3k](https://github.com/tosi3k))
- Added `--concurrency` flag to configure the concurrency of `kubectl diff` execution, defaults to 1. ([#118810](https://github.com/kubernetes/kubernetes/pull/118810), [@brancz](https://github.com/brancz))
- Added `ConsistentListFromCache` feature gate that allows apiserver to serve consistent lists from cache. ([#118508](https://github.com/kubernetes/kubernetes/pull/118508), [@serathius](https://github.com/serathius))
- Added `DisruptionTarget` condition to the pod preempted by kubelet to make room for a critical pod. ([#117586](https://github.com/kubernetes/kubernetes/pull/117586), [@mimowo](https://github.com/mimowo))
- Added `apiserver_admission_match_condition_evaluation_seconds` and `apiserver_admission_match_condition_exclusions_total` metrics. ([#119311](https://github.com/kubernetes/kubernetes/pull/119311), [@ivelichkovich](https://github.com/ivelichkovich))
- Added a container image for `kubectl` at `registry.k8s.io/kubectl` across the same architectures as other images (linux/amd64 linux/arm64 linux/s390x linux/ppc64le) ([#116672](https://github.com/kubernetes/kubernetes/pull/116672), [@dims](https://github.com/dims)) [SIG Architecture and Release]
- Added a new command line argument `--interactive` to kubectl. The new command line argument lets a user confirm deletion requests per resource interactively. ([#114530](https://github.com/kubernetes/kubernetes/pull/114530), [@ardaguclu](https://github.com/ardaguclu)) [SIG CLI and Testing]
- Added a new feature gate, `SchedulerQueueingHints` (enabled by default).
The new feature gate activates a framework for fine-grained filtering of events related to scheduler plugins.
In this release, no default scheduling plugins make use of the hinting framework, so you should not expect any behavior changes. ([#119328](https://github.com/kubernetes/kubernetes/pull/119328), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling]
- Added full cgroup v2 swap support for both `Limited` and `Unlimited` swap.
When `LimitedSwap` is enabled the swap limit would be automatically calculated for
Burstable QoS pods. For Best-Effort/Guaranteed QoS pods, swap would be disabled.
Containers with memory requests equal to their memory limits also won't have
swap access, and it is a way to opt-out of swap for a single container.
The formula for the swap limit for Burstable QoS pods is:
`(<memory-request>/<node-memory-capacity>)*<node-swap-capacity>`.
Support for `cgroup v1` is removed. ([#118764](https://github.com/kubernetes/kubernetes/pull/118764), [@iholder101](https://github.com/iholder101))
- Added handling for pods in podgc for `PodReplacementPolicy` or `PodDisruption`. ([#118772](https://github.com/kubernetes/kubernetes/pull/118772), [@kannon92](https://github.com/kannon92))
- Added reason to metric `attachdetach_controller_forced_detaches` in the attach detach controller. ([#119185](https://github.com/kubernetes/kubernetes/pull/119185), [@xing-yang](https://github.com/xing-yang))
- Added support for pod `hostNetwork` field selector ([#110477](https://github.com/kubernetes/kubernetes/pull/110477), [@halfcrazy](https://github.com/halfcrazy)) [SIG Apps and Node]
- Added swap to stats to Summary API and Prometheus endpoints (stats/summary and /metrics/resource). ([#118865](https://github.com/kubernetes/kubernetes/pull/118865), [@iholder101](https://github.com/iholder101))
- Added the implementation for `PodRecreationPolicy` to wait for the creation of pods once the existing ones are fully terminated. ([#117015](https://github.com/kubernetes/kubernetes/pull/117015), [@kannon92](https://github.com/kannon92))
- Allow to monitor client-go DNS resolver latencies via `rest_client_dns_resolution_duration_seconds` Prometheus metric. ([#115357](https://github.com/kubernetes/kubernetes/pull/115357), [@mfojtik](https://github.com/mfojtik))
- Apiserver adds two new metrics `etcd_requests_total` and `etcd_request_errors_total` that allow users to monitor requests to etcd storage, split by operation and resource type. ([#117222](https://github.com/kubernetes/kubernetes/pull/117222), [@iyear](https://github.com/iyear)) [SIG API Machinery]
- Bumped `distroless-iptables` to 0.2.6 based on Go 1.20.6. ([#119365](https://github.com/kubernetes/kubernetes/pull/119365), [@xmudrii](https://github.com/xmudrii))
- Bumped metrics-server to `v0.6.3`. ([#117120](https://github.com/kubernetes/kubernetes/pull/117120), [@dgrisonnet](https://github.com/dgrisonnet))
- CEL authorizer checks no longer raise runtime errors. Calls to "check" will always return a decision object and the authorization error (if any) can be accessed within expressions using the new decision methods "errored" and "error". ([#118804](https://github.com/kubernetes/kubernetes/pull/118804), [@benluddy](https://github.com/benluddy)) [SIG API Machinery]
- CRI: exposed commit memory bytes in container stats specific to Windows ([#119238](https://github.com/kubernetes/kubernetes/pull/119238), [@kiashok](https://github.com/kiashok))
- Client-go now exposes two new metrics to monitor the client-go logic that
generate http.Transports for the clients.
- `rest_client_transport_cache_entries` is a gauge metric
with the number of existing entries in the internal cache
- `rest_client_transport_create_calls_total` is a counter
that increments each time a new transport is created, storing
the result of the operation needed to generate it: hit, miss
or uncacheable. ([#117295](https://github.com/kubernetes/kubernetes/pull/117295), [@aojea](https://github.com/aojea))
- Cloud controller manager's node controller now emits timing metrics for initial `Node` synchronization. These metrics measure the delay between the creation of a new `Node` and the node controller's initial management actions, such as removing the cloud provider taint. These metrics should be consulted when setting cloud controller manager's `--concurrent-node-syncs` flag. ([#119241](https://github.com/kubernetes/kubernetes/pull/119241), [@cartermckinnon](https://github.com/cartermckinnon)) [SIG Cloud Provider and Instrumentation]
- Dynamic resource allocation: when a claim uses "wait for first consumer" allocation (the default), then it will now get deallocated after it was used by a pod. That ensures that the next pod isn't affected by previous scheduling decision and that resources are not kept allocated unless really needed. If keeping a claim allocated is desired, use "immediate allocation." ([#118936](https://github.com/kubernetes/kubernetes/pull/118936), [@pohly](https://github.com/pohly))
- Enabled use of pods with volumes and user namespaces. The feature gate was renamed from `UserNamespacesStatelessPodsSupport` to `UserNamespacesSupport`. ([#118691](https://github.com/kubernetes/kubernetes/pull/118691), [@giuseppe](https://github.com/giuseppe))
- External credential provider plugins will now have their standard error output logged by kubelet upon failures. ([#117448](https://github.com/kubernetes/kubernetes/pull/117448), [@cartermckinnon](https://github.com/cartermckinnon))
- Faster scheduling when `ResourceClaims` are involved. ([#119078](https://github.com/kubernetes/kubernetes/pull/119078), [@pohly](https://github.com/pohly))
- Fixed the alpha `CloudDualStackNodeIPs` feature. ([#118329](https://github.com/kubernetes/kubernetes/pull/118329), [@danwinship](https://github.com/danwinship))
- Graduated the `LegacyServiceAccountTokenTracking` feature gate to GA. The usage of auto-generated secret-based service account token now produces warnings, and relevant Secrets are labeled with a last-used timestamp (label key `kubernetes.io/legacy-token-last-used`). ([#117591](https://github.com/kubernetes/kubernetes/pull/117591), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Auth and Testing]
- Graduated the `ProbeTerminationGracePeriod` feature gate to GA. ([#114307](https://github.com/kubernetes/kubernetes/pull/114307), [@rphillips](https://github.com/rphillips))
- Hashing of KeyID in Logs
This release adds a feature to hash the `KeyID` values in the logs. The `KeyID` values are sensitive information that should not be exposed in plain text in the logs. By hashing the `KeyID` values, we can protect the confidentiality of the data while still being able to log the necessary information. ([#118988](https://github.com/kubernetes/kubernetes/pull/118988), [@nilekhc](https://github.com/nilekhc)) [SIG API Machinery, Auth and Testing]
- Implemented alpha support for a drop-in kubelet configuration directory. ([#119390](https://github.com/kubernetes/kubernetes/pull/119390), [@sohankunkerkar](https://github.com/sohankunkerkar))
- In the course of admitting a single request, the ValidatingAdmissionPolicy plugin will perform no more than one authorization check per unique authorizer expression. All evaluations of identical authorizer expressions will produce the same decision. ([#116443](https://github.com/kubernetes/kubernetes/pull/116443), [@benluddy](https://github.com/benluddy)) [SIG API Machinery and Testing]
- Introduce support for CEL optionals (see [CEL spec proposal 246](https://github.com/google/cel-spec/wiki/proposal-246)).
This feature will not be fully enabled until a future Kubernetes release (likely to be v1.29), but is added in v1.28 to enable
safe rollback on downgrade. ([#118339](https://github.com/kubernetes/kubernetes/pull/118339), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth, Cloud Provider and Testing]
- Kube-controller-manager: the dynamic resource controller steps in when a pod got created such that the scheduler ignores it (i.e. spec.nodeName is set) and then takes care of triggering delayed resource claim allocation and/or reserving a claim for the pod. ([#118209](https://github.com/kubernetes/kubernetes/pull/118209), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node and Testing]
- Kube-proxy handles Terminating EndpointSlices conditions and enables zero downtime deployments for Services with ExternalTrafficPolicy=Local author: @andrewsykim ([#117718](https://github.com/kubernetes/kubernetes/pull/117718), [@aojea](https://github.com/aojea)) [SIG Network, Testing and Windows]
- Kube-proxy service health returns http header `X-Load-Balancing-Endpoint-Weight` with number of local endpoints. The same information is still available in response body JSON `payload.LocalEndpoints`. ([#118999](https://github.com/kubernetes/kubernetes/pull/118999), [@cezarygerard](https://github.com/cezarygerard))
- Kubelet: plugins for dynamic resource allocation may use the `v1alpha3` API instead of v1alpha2 if they want to do prepare/unprepare operations in batches. ([#119012](https://github.com/kubernetes/kubernetes/pull/119012), [@pohly](https://github.com/pohly))
- Kubelet: security of dynamic resource allocation was enhanced by limiting node access to those objects that are needed on the node. ([#116254](https://github.com/kubernetes/kubernetes/pull/116254), [@pohly](https://github.com/pohly)) [SIG Auth and Testing]
- Kubelet: un-deprecated `--provider-id` flag. ([#116530](https://github.com/kubernetes/kubernetes/pull/116530), [@pacoxu](https://github.com/pacoxu))
- Kubernetes is now built with Go `1.20.4`. ([#117744](https://github.com/kubernetes/kubernetes/pull/117744), [@xmudrii](https://github.com/xmudrii)) [SIG Release and Testing]
- Kubernetes is now built with Go `1.20.5`. ([#118507](https://github.com/kubernetes/kubernetes/pull/118507), [@jeremyrickard](https://github.com/jeremyrickard))
- Kubernetes is now built with Go `1.20.6`. ([#119324](https://github.com/kubernetes/kubernetes/pull/119324), [@xmudrii](https://github.com/xmudrii))
- Metric `scheduler_scheduler_goroutines` is removed. Use `scheduler_goroutines` instead. ([#117727](https://github.com/kubernetes/kubernetes/pull/117727), [@kerthcet](https://github.com/kerthcet)) [SIG Scheduling]
- Migrated `pkg/controller/endpoint` to contextual logging. ([#116755](https://github.com/kubernetes/kubernetes/pull/116755), [@my-git9](https://github.com/my-git9))
- Migrated `pkg/scheduler/framework/preemption` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116835](https://github.com/kubernetes/kubernetes/pull/116835), [@mengjiao-liu](https://github.com/mengjiao-liu))
- Migrated `pod-security-admission` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#114471](https://github.com/kubernetes/kubernetes/pull/114471), [@Namanl2001](https://github.com/Namanl2001)) [SIG Apps and Auth]
- Migrated controller functions to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116930](https://github.com/kubernetes/kubernetes/pull/116930), [@fatsheep9146](https://github.com/fatsheep9146)) [SIG API Machinery, Apps, Network, Node, Storage and Testing]
- Migrated the Job controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116910](https://github.com/kubernetes/kubernetes/pull/116910), [@fatsheep9146](https://github.com/fatsheep9146)) [SIG API Machinery, Apps and Testing]
- Migrated the `EndpointSlice` and `EndpointSliceMirroring` controllers (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#115295](https://github.com/kubernetes/kubernetes/pull/115295), [@Namanl2001](https://github.com/Namanl2001)) [SIG API Machinery, Apps, Network and Testing]
- Migrated the certificate controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#113994](https://github.com/kubernetes/kubernetes/pull/113994), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG API Machinery, Apps, Auth, Instrumentation and Testing]
- Migrated the noderesources scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116748](https://github.com/kubernetes/kubernetes/pull/116748), [@mengjiao-liu](https://github.com/mengjiao-liu))
- Migrated the podtopologyspread scheduler plugins to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116797](https://github.com/kubernetes/kubernetes/pull/116797), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling]
- Moved `non-graceful node` shutdown to GA. ([#118228](https://github.com/kubernetes/kubernetes/pull/118228), [@carlory](https://github.com/carlory))
- New CEL Library functions to support Kubernetes Quantities. ([#118803](https://github.com/kubernetes/kubernetes/pull/118803), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery]
- New Metrics Added for Encryption Configuration Controller
This release adds new metrics to the Encryption Configuration Controller to help monitor the automatic reloading of encryption configuration. The new metrics include:
- `apiserver_encryption_config_controller_automatic_reload_failures_total`: Total number of failed automatic reloads of encryption configuration.
- `apiserver_encryption_config_controller_automatic_reload_success_total`: Total number of successful automatic reloads of encryption configuration.
- `apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds`: Timestamp of the last successful or failed automatic reload of encryption configuration.
These metrics can be used to monitor the health of the Encryption Configuration Controller and to troubleshoot any issues that may arise during automatic reloading of encryption configuration. ([#119008](https://github.com/kubernetes/kubernetes/pull/119008), [@nilekhc](https://github.com/nilekhc))
- New staging repo has been created for the `EndpointSlice` reconciler. ([#118953](https://github.com/kubernetes/kubernetes/pull/118953), [@mskrocki](https://github.com/mskrocki))
- Promoted `ServiceNodePortStaticSubrange` feature gate to beta, and it will be enabled by default. ([#117877](https://github.com/kubernetes/kubernetes/pull/117877), [@xuzhenglun](https://github.com/xuzhenglun))
- Promoted the following apiserver flowcontrol metrics to Beta:
- apiserver_flowcontrol_request_wait_duration_seconds
- apiserver_flowcontrol_current_executing_seats
- apiserver_flowcontrol_nominal_limit_seats
- apiserver_flowcontrol_rejected_requests_total
- apiserver_flowcontrol_dispatched_requests_total
- apiserver_flowcontrol_current_inqueue_requests
- apiserver_flowcontrol_current_executing_requests ([#119110](https://github.com/kubernetes/kubernetes/pull/119110), [@andrewsykim](https://github.com/andrewsykim))
- Renamed `PodHasNetwork` to `PodReadyToStartContainers`. ([#117702](https://github.com/kubernetes/kubernetes/pull/117702), [@kannon92](https://github.com/kannon92)) [SIG Node and Testing]
- Replaced `apiserver_storage_db_total_size_in_bytes` with `apiserver_storage_size_bytes` metric. ([#118812](https://github.com/kubernetes/kubernetes/pull/118812), [@serathius](https://github.com/serathius))
- Scheduler now waits for handlers to finish syncing before the scheduling cycles start. ([#116729](https://github.com/kubernetes/kubernetes/pull/116729), [@AxeZhan](https://github.com/AxeZhan))
- Set metrics-server's metric-resolution to 15s. ([#117121](https://github.com/kubernetes/kubernetes/pull/117121), [@dgrisonnet](https://github.com/dgrisonnet)) [SIG Cloud Provider and Instrumentation]
- SubjectAccessReview requests sent to webhook authorizers now default `spec.resourceAttributes.version` to `*` if unset. ([#116937](https://github.com/kubernetes/kubernetes/pull/116937), [@AxeZhan](https://github.com/AxeZhan)) [SIG Apps and Auth]
- Supported specifying a custom retry period for cloud `load-balancer` operations. ([#94021](https://github.com/kubernetes/kubernetes/pull/94021), [@timoreimann](https://github.com/timoreimann))
- The "value" part in the `wait --for=jsonpath='{expression}'[=value]` is now
optional. If the value is not provided i.e., the command looks like `wait --for=jsonpath='{expression}'`
then the wait condition is interpreted as matched when the expression returns
*any* single JSON value like object or a literal. ([#118160](https://github.com/kubernetes/kubernetes/pull/118160), [@minherz](https://github.com/minherz))
- The Kubernetes apiserver now emits a warning message for Pods with a null labelSelector in podAffinity or topologySpreadConstraints. The null labelSelector means "match none". Using it in podAffinity or topologySpreadConstraint could lead to unintended behavior. ([#117025](https://github.com/kubernetes/kubernetes/pull/117025), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling]
- The `AdvancedAuditing` feature gate that graduated to GA in `v1.12` (and was unconditionally
enabled) has been removed. ([#118763](https://github.com/kubernetes/kubernetes/pull/118763), [@Shubham82](https://github.com/Shubham82))
- The `ExpandedDNSConfig` feature has graduated to GA. 'ExpandedDNSConfig' feature was locked to default value and will be removed in v1.30. If you were setting this feature gate explicitly, please remove it now. ([#116741](https://github.com/kubernetes/kubernetes/pull/116741), [@gjkim42](https://github.com/gjkim42)) [SIG Apps, Network and Node]
- The apiserver debug endpoint `/debug/api_priority_and_fairness/dump_requests` has been extended to dump executing requests as well as queued ones. A column for StartTime has been added to the returned table, with the queued requests having a StartTime of "0001-01-01T00:00:00Z". The executing requests have a RequestIndexInQueue of -1, and the QueueIndex is also -1 for priority levels without queues. ([#119009](https://github.com/kubernetes/kubernetes/pull/119009), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery]
- The helping message of commands which have sub-commands is now clearer and more instructive. It will show the full command instead of `kubectl <command> --help ...`
Changed `kubectl create secret --help` description. There will be a short introduction to the three secret types and clearer guidance on how to use the command. ([#117930](https://github.com/kubernetes/kubernetes/pull/117930), [@LronDC](https://github.com/LronDC))
- The scheduler skips the `InterPodAffinity` Score plugin when nothing to do with the Pod.
It will affect some metrics values related to the InterPodAffinity Score plugin. ([#117794](https://github.com/kubernetes/kubernetes/pull/117794), [@utam0k](https://github.com/utam0k)) [SIG Scheduling]
- The scheduler skips the `PodTopologySpread` Filter plugin if no spread constraints.
It will affect some metrics values related to the `PodTopologySpread` Filter plugin. ([#117683](https://github.com/kubernetes/kubernetes/pull/117683), [@utam0k](https://github.com/utam0k))
- The scheduler skips the `PodTopologySpread` Score plugin when nothing to do with the Pod.
It will affect some metrics values related to the PodTopologySpread Score plugin. ([#118608](https://github.com/kubernetes/kubernetes/pull/118608), [@utam0k](https://github.com/utam0k))
- The short names `vwc` and `mwc` were introduced for the resources `validatingwebhookconfigurations` and `mutatingwebhookconfigurations`. ([#117535](https://github.com/kubernetes/kubernetes/pull/117535), [@hysyeah](https://github.com/hysyeah))
- Updated etcd image to `3.5.9-0`. ([#117999](https://github.com/kubernetes/kubernetes/pull/117999), [@kkkkun](https://github.com/kkkkun)) [SIG API Machinery]
- Updated cAdvisor to `v0.47.2` and fixed metrics in `cri-o` when a container restarts. ([#118774](https://github.com/kubernetes/kubernetes/pull/118774), [@harche](https://github.com/harche))
- Updated distroless I-tables to use registry.k8s.io/build-image/distroless-iptables:v0.2.5 ([#118541](https://github.com/kubernetes/kubernetes/pull/118541), [@jeremyrickard](https://github.com/jeremyrickard)) [SIG Testing]
- Updated distroless iptables to use released image `registry.k8s.io/build-image/distroless-iptables:v0.2.4` ([#117746](https://github.com/kubernetes/kubernetes/pull/117746), [@xmudrii](https://github.com/xmudrii)) [SIG Testing]
- Updated the scheduler interface and cache methods to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116849](https://github.com/kubernetes/kubernetes/pull/116849), [@mengjiao-liu](https://github.com/mengjiao-liu))
- ValidatingAdmissionPolicy type checking now correctly handles `authorizer` variable. ([#118540](https://github.com/kubernetes/kubernetes/pull/118540), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery]
- When a pod is done or not going to run, then `ResourceClaims` for it can be reused by other pods or deleted. ([#118817](https://github.com/kubernetes/kubernetes/pull/118817), [@pohly](https://github.com/pohly))
- With the `KubeletCgroupDriverFromCRI` feature gate enabled and sufficiently new version of a container
runtime, kubelet automatically detects the cgroup driver config from the container runtime, eliminating
the need to specify the `cgroupDriver` configuration option (or `--cgroup-driver` flag) of kubelet. ([#118770](https://github.com/kubernetes/kubernetes/pull/118770), [@marquiz](https://github.com/marquiz))
- [Kube-proxy]: Implemented connection draining for terminating nodes. ([#116470](https://github.com/kubernetes/kubernetes/pull/116470), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu))
- `--version=v1.X.Y...` can now be used to set the prerelease and buildID portions of the version reported by components ([#117688](https://github.com/kubernetes/kubernetes/pull/117688), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Architecture and Release]
- `RetroactiveDefaultStorageClass` feature made stable and enabled by default. ([#118102](https://github.com/kubernetes/kubernetes/pull/118102), [@RomanBednar](https://github.com/RomanBednar))
- `TopologyManagerPolicyOptions` feature-flag is promoted to beta and enabled by default. ([#118816](https://github.com/kubernetes/kubernetes/pull/118816), [@PiotrProkop](https://github.com/PiotrProkop))
- `force_delete_pods_total` and `force_delete_pod_errors_total` metrics count all pod deletion behaviors. ([#118480](https://github.com/kubernetes/kubernetes/pull/118480), [@carlory](https://github.com/carlory))
- `klog` text output now uses JSON as encoding for structs, maps and slices. ([#117687](https://github.com/kubernetes/kubernetes/pull/117687), [@pohly](https://github.com/pohly))
- `kube-proxy` in iptables mode will now have separate `sync_full_proxy_rules_duration_seconds`\nand
`sync_partial_proxy_rules_duration_seconds` (in addition to the existing\n`sync_proxy_rules_duration_seconds`),
giving better information about the duration of each \nsync type, rather than
only giving a weighted average of the two sync types together. ([#117787](https://github.com/kubernetes/kubernetes/pull/117787), [@danwinship](https://github.com/danwinship))
- `kubeadm`: added a new "kubeadm config validate" command that can be used to
validate any input config file. Use the `--config` flag to pass a config file
to it. See the command `--help` screen for more information. As a result of adding
this new command, enhance the validation capabilities of the existing "kubeadm
config migrate" command. For both commands unknown APIs or fields will throw errors. ([#118013](https://github.com/kubernetes/kubernetes/pull/118013), [@neolit123](https://github.com/neolit123))
- `kubeadm`: added the `--allow-experimental-api` flag to "kubeadm config migrate/validate" commands. It can be used to migrate or validate WIP/experimental APIs in the future. ([#118866](https://github.com/kubernetes/kubernetes/pull/118866), [@neolit123](https://github.com/neolit123))
- `kubeadm`: generate CA certificates with a start time that is offset 5
minutes in the past relative to the current system time to workaround cases of
clock desync. ([#118922](https://github.com/kubernetes/kubernetes/pull/118922), [@champtar](https://github.com/champtar))
- `plugin_evaluation_total` metric supports prescore/score extension point.
The metric doesn't get incremented when the prescore/score plugin has nothing to do with an incoming pod. ([#118025](https://github.com/kubernetes/kubernetes/pull/118025), [@AxeZhan](https://github.com/AxeZhan))
### Documentation
- Enhanced clarity in error messaging when waiting for volume creation ([#118262](https://github.com/kubernetes/kubernetes/pull/118262), [@torredil](https://github.com/torredil)) [SIG Apps and Storage]
### Failing Test
- Allowed Azure Disk e2es to use newer topology labels if available from nodes. ([#117216](https://github.com/kubernetes/kubernetes/pull/117216), [@gnufied](https://github.com/gnufied))
- Fixed nil pointer in test AfterEach volumeperf.go for sidecar release. ([#117368](https://github.com/kubernetes/kubernetes/pull/117368), [@sunnylovestiramisu](https://github.com/sunnylovestiramisu))
- Switched back to `debian-base` instead of distroless for conformance image. ([#119422](https://github.com/kubernetes/kubernetes/pull/119422), [@saschagrunert](https://github.com/saschagrunert))
### Bug or Regression
- Added a new event `FailedToRetrieveImagePullSecret` which will be generated when a pod references an `ImagePullSecret` that doesn't exist. ([#117927](https://github.com/kubernetes/kubernetes/pull/117927), [@kaisoz](https://github.com/kaisoz)) [SIG Node]
- Added additional validation for endpoint IP configuration while iterating through queried endpoint list. ([#116749](https://github.com/kubernetes/kubernetes/pull/116749), [@princepereira](https://github.com/princepereira))
- Added warning for dup ports update/patching in pod's container ports and service ports. ([#113245](https://github.com/kubernetes/kubernetes/pull/113245), [@pacoxu](https://github.com/pacoxu))
- As in Kubernetes `v1.26` and `v1.27`, resource claims do not get prepared by `kubelet` when no container uses them. This was changed accidentally in [v1.28.0-alpha.1](https://github.com/kubernetes/kubernetes/releases/tag/v1.28.0-alpha.1). ([#118786](https://github.com/kubernetes/kubernetes/pull/118786), [@pohly](https://github.com/pohly))
- Bumped cadvisor version to `v0.47.3`. ([#119225](https://github.com/kubernetes/kubernetes/pull/119225), [@iholder101](https://github.com/iholder101))
- CI job `ci-kubernetes-node-arm64-ubuntu-serial` will test node e2e on arm64, `use-dockerized-build` and `target-build-arch` are required to run this job. ([#118567](https://github.com/kubernetes/kubernetes/pull/118567), [@chendave](https://github.com/chendave))
- CVE-2023-27561 CVE-2023-25809 CVE-2023-28642: Bump fix runc v1.1.4 -> v1.1.5 ([#117095](https://github.com/kubernetes/kubernetes/pull/117095), [@PushkarJ](https://github.com/PushkarJ)) [SIG Architecture, Node and Security]
- Code blocks in `kubectl {$COMMAND}--help` will move right by 3 indentation. ([#118029](https://github.com/kubernetes/kubernetes/pull/118029), [@ardaguclu](https://github.com/ardaguclu))
- Compute the backoff delay more accurately for deleted pods ([#118413](https://github.com/kubernetes/kubernetes/pull/118413), [@mimowo](https://github.com/mimowo)) [SIG Apps]
- Declare Job as finished only after removing all Pod finalizers to avoid orphan Pods. ([#119159](https://github.com/kubernetes/kubernetes/pull/119159), [@alculquicondor](https://github.com/alculquicondor))
- During device plugin allocation, resources requested by the pod can only be allocated if the device plugin has registered itself to kubelet AND healthy devices are present on the node to be allocated. If these conditions are not sattsfied, the pod would fail with `UnexpectedAdmissionError` error. ([#116376](https://github.com/kubernetes/kubernetes/pull/116376), [@swatisehgal](https://github.com/swatisehgal)) [SIG Node and Testing]
- Dynamic Resource Allocation: logged an error and submitted an event when `Kubelet` failed to prepare dynamic resources. ([#118578](https://github.com/kubernetes/kubernetes/pull/118578), [@bart0sh](https://github.com/bart0sh))
- Ensure Job status updates are batched by 1s. This fixes an unlikely scenario when a sequence of immediately
completing pods could trigger a sequence of non-batched Job status updates. ([#118470](https://github.com/kubernetes/kubernetes/pull/118470), [@mimowo](https://github.com/mimowo)) [SIG Apps]
- Faster `StatefulSet` creation when `Parallel` mode is enabled. ([#117865](https://github.com/kubernetes/kubernetes/pull/117865), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska))
- Fixed a data race in TopologyCache when `AddHints` and `SetNodes` are called concurrently. ([#117249](https://github.com/kubernetes/kubernetes/pull/117249), [@tnqn](https://github.com/tnqn)) [SIG Apps and Network]
- Fixed a race condition in `kube-proxy` when using LocalModeNodeCIDR, to avoid dropping Services traffic if the object node is recreated when `kube-proxy` is starting. ([#118499](https://github.com/kubernetes/kubernetes/pull/118499), [@aojea](https://github.com/aojea))
- Fixed bug where `listOfStrings.join()` in CEL expressions resulted in an unexpected internal error. ([#117593](https://github.com/kubernetes/kubernetes/pull/117593), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery]
- Fixed incorrect calculation for ResourceQuota with PriorityClass as its scope. ([#117677](https://github.com/kubernetes/kubernetes/pull/117677), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG API Machinery]
- Fix: After a Node is down and take some time to get back to up again, the mount point of the evicted Pods cannot be cleaned up successfully. (#111933) Meanwhile Kubelet will print the log `Orphaned pod "xxx" found, but error not a directory occurred when trying to remove the volumes dir` every 2 seconds. (#105536) ([#116134](https://github.com/kubernetes/kubernetes/pull/116134), [@cvvz](https://github.com/cvvz)) [SIG Node and Storage]
- Fix: The volume is not detached after the pod and PVC objects are deleted. ([#116138](https://github.com/kubernetes/kubernetes/pull/116138), [@cvvz](https://github.com/cvvz)) [SIG Storage]
- Fixed Cronjob `status.lastSuccessfulTime` not populated by a manually triggered ([#118530](https://github.com/kubernetes/kubernetes/pull/118530), [@carlory](https://github.com/carlory))
- Fixed Topology Aware Hints not working when the `topology.kubernetes.io/zone` label is added after Node creation. ([#117245](https://github.com/kubernetes/kubernetes/pull/117245), [@tnqn](https://github.com/tnqn))
- Fixed `creationTimestamp: null` causing unnecessary writes to etcd. ([#116865](https://github.com/kubernetes/kubernetes/pull/116865), [@alexzielenski](https://github.com/alexzielenski))
- Fixed `vSphere` cloud provider not to skip detach volumes from nodes at `kube-controller-startup`. ([#117243](https://github.com/kubernetes/kubernetes/pull/117243), [@jsafrane](https://github.com/jsafrane))
- Fixed a bug at `kube-apiserver` start where `APIService` objects for custom resources could be deleted and recreated. ([#118104](https://github.com/kubernetes/kubernetes/pull/118104), [@liggitt](https://github.com/liggitt))
- Fixed a bug that unintentionally overrides custom Accept headers in http (live-/readiness)-probes if the header is in lowercase. ([#114606](https://github.com/kubernetes/kubernetes/pull/114606), [@tuunit](https://github.com/tuunit))
- Fixed a bug where `kubectl port-forward`, when used with a Deployment, could connect to a terminating pod even when a running pod is also available. ([#119256](https://github.com/kubernetes/kubernetes/pull/119256), [@brianpursley](https://github.com/brianpursley)) [SIG CLI]
- Fixed a bug where pv recycler failed to scrub volume with too many files in the directory due to hitting ARG_MAX limit with rm command (#117189). ([#117283](https://github.com/kubernetes/kubernetes/pull/117283), [@defo89](https://github.com/defo89)) [SIG Cloud Provider and Storage]
- Fixed a memory leak in the Kubernetes API server that occurs during APIService processing. ([#117258](https://github.com/kubernetes/kubernetes/pull/117258), [@enj](https://github.com/enj)) [SIG API Machinery]
- Fixed a race condition between `Run()` and `SetTransform()` and `SetWatchErrorHandler()` in shared informers. ([#117870](https://github.com/kubernetes/kubernetes/pull/117870), [@howardjohn](https://github.com/howardjohn)) [SIG API Machinery]
- Fixed a race condition serving `OpenAPI` content ([#117705](https://github.com/kubernetes/kubernetes/pull/117705), [@Jefftree](https://github.com/Jefftree))
- Fixed a regression in `1.27.0` that resulted in `missing metadata in converted object` errors when modifying objects for multi-version custom resource definitions with a conversion strategy of `None`. ([#117301](https://github.com/kubernetes/kubernetes/pull/117301), [@ncdc](https://github.com/ncdc))
- Fixed a regression in `kubectl` and `client-go` discovery when configured with a server URL other than the root of a server ([#117495](https://github.com/kubernetes/kubernetes/pull/117495), [@ardaguclu](https://github.com/ardaguclu))
- Fixed an issue where the API server did not send impersonated UID to authentication webhooks. ([#116681](https://github.com/kubernetes/kubernetes/pull/116681), [@stlaz](https://github.com/stlaz)) [SIG API Machinery and Auth]
- Fixed bug that caused a resource to include patch directives when using strategic merge patch against a non-existent field. ([#117568](https://github.com/kubernetes/kubernetes/pull/117568), [@alexzielenski](https://github.com/alexzielenski))
- Fixed bug to correctly report `ErrRegistryUnavailable` on pulling container images for remote CRI runtimes. ([#117612](https://github.com/kubernetes/kubernetes/pull/117612), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node]
- Fixed bug where explain was not properly respecting jsonpaths. ([#115694](https://github.com/kubernetes/kubernetes/pull/115694), [@mpuckett159](https://github.com/mpuckett159))
- Fixed bug where using the $deleteFromPrimitiveList directive in a strategic merge patch of certain fields would remove the other values from the list instead of the values specified. ([#110472](https://github.com/kubernetes/kubernetes/pull/110472), [@brianpursley](https://github.com/brianpursley)) [SIG API Machinery]
- Fixed component status calling etcd health endpoint over http which exposed kubernetes to the risk of complete watch starvation and is inconsistent with other etcd probing done by `kube-apiserver`. ([#118460](https://github.com/kubernetes/kubernetes/pull/118460), [@serathius](https://github.com/serathius))
- Fixed computing backoff delay when using Job pod failure policy, by including in the backoff delay calculation pod failures ignored from the backoffLimit counter. ([#119434](https://github.com/kubernetes/kubernetes/pull/119434), [@mimowo](https://github.com/mimowo))
- Fixed cronjob controller handling of complex schedules, like `30 6-16/4 * * 1-5`, for example. ([#118724](https://github.com/kubernetes/kubernetes/pull/118724), [@soltysh](https://github.com/soltysh))
- Fixed deletion of non-admissible pods that are deleted during Kubelet restart. ([#118497](https://github.com/kubernetes/kubernetes/pull/118497), [@mimowo](https://github.com/mimowo))
- Fixed issue where `kubectl-convert` would fail when encountering resources that could not be converted to the specified api version. New behavior is to warn the user of the failed conversions and continue to convert the remaining resources. ([#117002](https://github.com/kubernetes/kubernetes/pull/117002), [@gxwilkerson33](https://github.com/gxwilkerson33))
- Fixed issue where there was no response or error from kubectl rollout status when there were no resources of specified kind. ([#117884](https://github.com/kubernetes/kubernetes/pull/117884), [@gxwilkerson33](https://github.com/gxwilkerson33)) [SIG CLI]
- Fixed kubelet startup getting stuck with `NewVolumeManagerReconstruction` feature enabled and a CSI volume present in /var/lib/kubelet/pods. ([#117804](https://github.com/kubernetes/kubernetes/pull/117804), [@jsafrane](https://github.com/jsafrane)) [SIG Node and Storage]
- Fixed performance regression in scheduler caused by frequent metric lookup on critical code path. ([#117594](https://github.com/kubernetes/kubernetes/pull/117594), [@tosi3k](https://github.com/tosi3k))
- Fixed restricted debug profile. ([#117543](https://github.com/kubernetes/kubernetes/pull/117543), [@mochizuki875](https://github.com/mochizuki875))
- Fixed the `preStop` hook. This will now block the pod termination grace period. ([#115835](https://github.com/kubernetes/kubernetes/pull/115835), [@HirazawaUi](https://github.com/HirazawaUi))
- Fixed the discoverability of `apiregistration.k8s.io` in `openapi/v3` ([#118879](https://github.com/kubernetes/kubernetes/pull/118879), [@atiratree](https://github.com/atiratree))
- If `kubeadm reset` finds no etcd member ID for the peer it removes during the `remove-etcd-member` phase, it continues immediately to other phases, instead of retrying the phase for up to 3 minutes before continuing. ([#117724](https://github.com/kubernetes/kubernetes/pull/117724), [@dlipovetsky](https://github.com/dlipovetsky)) [SIG Cluster Lifecycle]
- Improved exponential backoff in Reflector, significantly reducing the load on Kubernetes apiserver in case of throttling of requests. ([#118132](https://github.com/kubernetes/kubernetes/pull/118132), [@marseel](https://github.com/marseel)) [SIG API Machinery and Scalability]
- Known issue: fixed that the PreEnqueue plugins aren't executed for Pods proceeding to activeQ through backoffQ. ([#117194](https://github.com/kubernetes/kubernetes/pull/117194), [@sanposhiho](https://github.com/sanposhiho)) [SIG Release and Scheduling]
- Kubeadm: the limitation that the `ignorePreflightErrors` field can not be set to `all` in kubeadm config file has been removed. ([#119351](https://github.com/kubernetes/kubernetes/pull/119351), [@SataQiu](https://github.com/SataQiu))
- Kubelet terminates pods correctly upon restart, fixing an issue where pods may have not been fully terminated if the kubelet was restarted during pod termination. ([#117019](https://github.com/kubernetes/kubernetes/pull/117019), [@bobbypage](https://github.com/bobbypage)) [SIG Node and Testing]
- Kubelet will now skip pod resource checks when the request is zero. ([#116408](https://github.com/kubernetes/kubernetes/pull/116408), [@ChenLingPeng](https://github.com/ChenLingPeng))
- Number of errors reported to the metric `storage_operation_duration_seconds_count` for emptyDir decreased significantly because previously one error was reported for each projected volume created. ([#117022](https://github.com/kubernetes/kubernetes/pull/117022), [@mpatlasov](https://github.com/mpatlasov)) [SIG Storage]
- Pod termination will be faster when the pod has a missing volume reference. ([#117412](https://github.com/kubernetes/kubernetes/pull/117412), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node and Testing]
- Recording timing traces had a race condition. Impact in practice was probably low. ([#117139](https://github.com/kubernetes/kubernetes/pull/117139), [@pohly](https://github.com/pohly)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
- Reduced CPU and memory consumption of `kube-apiserver` if OpenAPI V2 will not be accessed by any client. Also improved performance of the apiserver on installation of many CRDs. ([#118212](https://github.com/kubernetes/kubernetes/pull/118212), [@Jefftree](https://github.com/Jefftree))
- Removed leading zeros from the etcd member ID in kubeadm log messages. ([#117919](https://github.com/kubernetes/kubernetes/pull/117919), [@dlipovetsky](https://github.com/dlipovetsky)) [SIG Cluster Lifecycle]
- Resolves a spurious "Unknown discovery response content-type" error in client-go discovery requests by tolerating extra content-type parameters in API responses ([#117571](https://github.com/kubernetes/kubernetes/pull/117571), [@seans3](https://github.com/seans3)) [SIG API Machinery]
- Reverted NewVolumeManagerReconstruction and SELinuxMountReadWriteOncePod feature gates to disabled by default to resolve a regression of volume reconstruction on kubelet/node restart ([#117751](https://github.com/kubernetes/kubernetes/pull/117751), [@liggitt](https://github.com/liggitt)) [SIG Storage]
- Setting a mirror pod's phase to Succeeded or Failed can prevent the corresponding static pod from restarting due mutation of a Kubelet cache. ([#116482](https://github.com/kubernetes/kubernetes/pull/116482), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node]
- Show a warning when `volume.beta.kubernetes.io/storage-class` annotation is used in pv or pvc ([#117036](https://github.com/kubernetes/kubernetes/pull/117036), [@haoruan](https://github.com/haoruan)) [SIG Storage]
- Static pods were taking extra time to be restarted after being updated. Static pods that are waiting to restart were not correctly counted in `kubelet_working_pods`. ([#116995](https://github.com/kubernetes/kubernetes/pull/116995), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node]
- The Daemonset controller creates replacements for terminal Pods, which can appear during VM preemptions or when using Pod finalizers. ([#118716](https://github.com/kubernetes/kubernetes/pull/118716), [@alculquicondor](https://github.com/alculquicondor))
- The `pod_scheduling_duration_seconds` metrics won't consider the time when a pod fails `PreEnqueue` (like being gated). ([#118049](https://github.com/kubernetes/kubernetes/pull/118049), [@helayoty](https://github.com/helayoty))
- The kube-proxy `sync_proxy_rules_iptables_total` metric has now reverted back
to its pre-1.27 behavior of tracking the total number of iptables rules that
kube-proxy is responsible for, rather than only counting the number of rules
that it re-synced on the last sync. The new `sync_proxy_rules_iptables_last`
metric now gives the latter number. ([#119140](https://github.com/kubernetes/kubernetes/pull/119140), [@danwinship](https://github.com/danwinship)) [SIG Network]
- The metric `apiserver_flowcontrol_request_concurrency_limit` has been deprecated and will be removed in a future release. It is a duplicate of `apiserver_flowcontrol_nominal_limit_seats` (introduced in release 1.26) but has an outdated name and had an outdated HELP string. ([#118959](https://github.com/kubernetes/kubernetes/pull/118959), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery]
- Updated `etcd` version to `3.5.8`. ([#117335](https://github.com/kubernetes/kubernetes/pull/117335), [@kkkkun](https://github.com/kkkkun))
- Updated apiserver metric `request_filter_duration_seconds` to include a 10s, 15s and 30s bucket.
Updated apiserver metric `request_wait_duration_seconds` to include a 15s bucket. ([#118945](https://github.com/kubernetes/kubernetes/pull/118945), [@andrewsykim](https://github.com/andrewsykim))
- Updated kube-apiserver's priority & fairness work estimator such that 'max seats' is MIN(0.15 x nominalCL, nominalCL / handSize)
This fixes a bug where clients with requests using hand size x max seats greater than the nominal concurrency limit can starve other requests in the same priority level. ([#118601](https://github.com/kubernetes/kubernetes/pull/118601), [@andrewsykim](https://github.com/andrewsykim))
- Updated static pods are restarted 2s faster by correcting a safe but non-optimal ordering bug. ([#116690](https://github.com/kubernetes/kubernetes/pull/116690), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node]
- Users will no longer see an error for failed events caused due to terminating namespace. ([#114849](https://github.com/kubernetes/kubernetes/pull/114849), [@padlar](https://github.com/padlar)) [SIG API Machinery]
- [Dual-stack] Fixed `generateAPIPodStatus()` of kubelet handling Secondary IP. hostIPs order may not be consistent. If secondary IP is before primary one, current logic adds primary IP twice into `PodIPs`, which leads to error: "may specify no more than one IP for each IP family". ([#116879](https://github.com/kubernetes/kubernetes/pull/116879), [@lzhecheng](https://github.com/lzhecheng))
- [KCCM] service controller: change the cloud controller manager to make `providerID` a predicate when synchronizing nodes. This change allows load balancer integrations to ensure that the `providerID` is set when configuring
load balancers and targets. ([#117388](https://github.com/kubernetes/kubernetes/pull/117388), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider and Network]
- `kube-apiserver` will now always remove its endpoint from Kubernetes service during
graceful shutdown (even if it's the only/last one). ([#116685](https://github.com/kubernetes/kubernetes/pull/116685), [@nayihz](https://github.com/nayihz))
- `kubeadm:` fixed a bug where the static pod changes detection logic is inconsistent
with kubelet. ([#118069](https://github.com/kubernetes/kubernetes/pull/118069), [@SataQiu](https://github.com/SataQiu))
- `kubeadm`: `crictl pull` should use `-i` to set the image service endpoint. ([#117835](https://github.com/kubernetes/kubernetes/pull/117835), [@pacoxu](https://github.com/pacoxu))
- `kubeadm`: fixed a bug where file copy(backup) could not be executed correctly
on Windows platform during upgrade. ([#117861](https://github.com/kubernetes/kubernetes/pull/117861), [@SataQiu](https://github.com/SataQiu))
- `kubeadm`: speedup init by 0s or 20s. kubelet-start phase is now after etcd
and control-plane phases, removing a race condition between kubelet looking for
static pod manifests and kubeadm writing them. ([#117984](https://github.com/kubernetes/kubernetes/pull/117984), [@champtar](https://github.com/champtar))
- `kubeadm`: will now throw warnings instead of errors for deprecated feature gates. ([#118270](https://github.com/kubernetes/kubernetes/pull/118270), [@pacoxu](https://github.com/pacoxu))
- `kubectl events --for` will also support fully qualified names such as replicasets.apps,
etc. ([#117034](https://github.com/kubernetes/kubernetes/pull/117034), [@ardaguclu](https://github.com/ardaguclu))
- `kubectl explain` should correctly work for all resources. ([#118876](https://github.com/kubernetes/kubernetes/pull/118876), [@atiratree](https://github.com/atiratree))
- `kubectl expose` supports the creation of different protocol services on the same port. ([#114909](https://github.com/kubernetes/kubernetes/pull/114909), [@aimuz](https://github.com/aimuz))
- `kubelet` will ensure /etc/hosts file is mode 0644 regardless of umask. ([#113209](https://github.com/kubernetes/kubernetes/pull/113209), [@luozhiwenn](https://github.com/luozhiwenn))
- `kubelet`: print sorted volumes message in events. ([#117079](https://github.com/kubernetes/kubernetes/pull/117079), [@qingwave](https://github.com/qingwave))
- `wait.PollUntilContextTimeout` function, if immediate is true, the condition
will be invoked before waiting and guarantees that the condition is invoked at
least once, regardless of whether the context has been cancelled. ([#118686](https://github.com/kubernetes/kubernetes/pull/118686), [@aojea](https://github.com/aojea))
### Other (Cleanup or Flake)
- A `v2-level` info log will be added, which will output the details of the pod being preempted, including victim and preemptor. ([#117214](https://github.com/kubernetes/kubernetes/pull/117214), [@HirazawaUi](https://github.com/HirazawaUi))
- Allowed container runtimes to use `ErrSignatureValidationFailed` as possible image pull failure. ([#117717](https://github.com/kubernetes/kubernetes/pull/117717), [@saschagrunert](https://github.com/saschagrunert))
- Deprecated `genericclioptions.IOStreams` and used `genericiooptions.IOStreams`. ([#117102](https://github.com/kubernetes/kubernetes/pull/117102), [@ardaguclu](https://github.com/ardaguclu))
- E2e framework: the `node-role.kubernetes.io/master` taint has been removed from the default value of `--non-blocking-taints` flag. You may need to set `--non-blocking-taints` explicitly if the cluster to be tested has nodes with the deprecated `node-role.kubernetes.io/master` taint. ([#118510](https://github.com/kubernetes/kubernetes/pull/118510), [@SataQiu](https://github.com/SataQiu)) [SIG Testing]
- Enabled the `node-local` kubelet podresources API endpoint on windows, alongside unix. ([#115133](https://github.com/kubernetes/kubernetes/pull/115133), [@ffromani](https://github.com/ffromani))
- Fixed dra e2e image build on non-amd64 architectures. ([#117912](https://github.com/kubernetes/kubernetes/pull/117912), [@bart0sh](https://github.com/bart0sh)) [SIG Node and Testing]
- Kube-apiserver adds two new alpha metrics `conversion_webhook_request_total` and `conversion_webhook_duration_seconds` that allow users to monitor requests to CRD conversion webhooks, split by result, and failure_type (In case of failure). ([#118292](https://github.com/kubernetes/kubernetes/pull/118292), [@cchapla](https://github.com/cchapla)) [SIG API Machinery, Architecture and Instrumentation]
- Kube-proxy will now warn at startup if the configuration seems inconsistent
with respect to IP families. (For example, if you have an IPv4 node IP, but
`--cluster-cidr` is IPv6.) ([#119003](https://github.com/kubernetes/kubernetes/pull/119003), [@danwinship](https://github.com/danwinship)) [SIG Network]
- Kube-proxy: removed log warning about not using config file. ([#118115](https://github.com/kubernetes/kubernetes/pull/118115), [@TommyStarK](https://github.com/TommyStarK)) [SIG Network]
- Made Job controller batching of syncJob invocations enabled unconditionally (it was conditional on JobReadyPods feature before).
Also, Job controller's constants for default backoff and maximal backoff are lowered down to 1s (from 10s) and 1min (from 6min), respectively. These constants are used to determine the backoff delay for the next Job controller sync in case of a request failure. ([#118615](https://github.com/kubernetes/kubernetes/pull/118615), [@mimowo](https://github.com/mimowo)) [SIG Apps and Testing]
- Marked the feature gate `ExperimentalHostUserNamespaceDefaulting` as deprecated.
Enabling the feature gate already had no effect; the deprecation allows for removing the feature gate in a future release. ([#116723](https://github.com/kubernetes/kubernetes/pull/116723), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Node]
- Migrated `pkg/scheduler/framework/runtime` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116842](https://github.com/kubernetes/kubernetes/pull/116842), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling]
- Migrated the disruption controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#119147](https://github.com/kubernetes/kubernetes/pull/119147), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG API Machinery, Apps, Instrumentation and Testing]
- Migrated the interpodaffinity scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116635](https://github.com/kubernetes/kubernetes/pull/116635), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling]
- Migrated the podgc controller and some other remaining log calls within `kube-controller-manager` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). `kube-controller-manager` is now converted completely. ([#119250](https://github.com/kubernetes/kubernetes/pull/119250), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Cloud Provider, Instrumentation, Network, Storage and Testing]
- Migrated the volumezone scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116829](https://github.com/kubernetes/kubernetes/pull/116829), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling]
- Moved `k8s.io/kubernetes/pkg/kubelet/cri/streaming` package to `k8s.io/kubelet/pkg/cri/streaming`. ([#118253](https://github.com/kubernetes/kubernetes/pull/118253), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node, Release and Security]
- OpenAPI proto deserializations should use `gnostic-models` instead of the gnostic library. ([#118384](https://github.com/kubernetes/kubernetes/pull/118384), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Instrumentation, Node, Storage and Testing]
- Projects which use k8s.io/code-generator and invoke `generate-groups` or `generate-internal-groups.sh` have a new, simpler script (`kube_codegen.sh`) they can use. The old scripts are deprecated but remain intact. ([#117262](https://github.com/kubernetes/kubernetes/pull/117262), [@thockin](https://github.com/thockin)) [SIG API Machinery and Instrumentation]
- Promoted `kubernetes_healthcheck` and `kubernetes_healthchecks_total` to `BETA` stability level. ([#118986](https://github.com/kubernetes/kubernetes/pull/118986), [@logicalhan](https://github.com/logicalhan))
- Reduced delay when processing jobs after a transient API error. ([#118759](https://github.com/kubernetes/kubernetes/pull/118759), [@mimowo](https://github.com/mimowo))
- Removed GA'ed feature gate `DelegateFSGroupToCSIDriver`. ([#117655](https://github.com/kubernetes/kubernetes/pull/117655), [@carlory](https://github.com/carlory))
- Removed GA'ed feature gate `DevicePlugins`. ([#117656](https://github.com/kubernetes/kubernetes/pull/117656), [@carlory](https://github.com/carlory))
- Removed GA'ed feature gate `KubeletCredentialProviders`. ([#116901](https://github.com/kubernetes/kubernetes/pull/116901), [@pacoxu](https://github.com/pacoxu))
- Removed GA'ed feature gates: `MixedProtocolLBService`, `ServiceInternalTrafficPolicy`,
`ServiceIPStaticSubrange`, and `EndpointSliceTerminatingCondition`. ([#117237](https://github.com/kubernetes/kubernetes/pull/117237), [@yulng](https://github.com/yulng))
- Removed `KUBECTL_EXPLAIN_OPENAPIV3` which is already redundant. ([#119286](https://github.com/kubernetes/kubernetes/pull/119286), [@ardaguclu](https://github.com/ardaguclu))
- Removed the deprecated `azureFile` in-tree storage plugin. ([#118236](https://github.com/kubernetes/kubernetes/pull/118236), [@andyzhangx](https://github.com/andyzhangx))
- Revised `OpenAPI v2` fetching for CustomResourceDefinitions. CRDs are now aggregated lazily,
which improves resource usage during installation of many CRDs. As a result, the first request
to fetch the OpenAPI may be slower. ([#118808](https://github.com/kubernetes/kubernetes/pull/118808), [@Jefftree](https://github.com/Jefftree))
- Shrank the `OpenAPI v2` spec by more than 50%, especially for less CPU resource consumption. ([#118204](https://github.com/kubernetes/kubernetes/pull/118204), [@sttts](https://github.com/sttts))
- Structured logging of `NamespacedName` was inconsistent with `klog.KObj`. Now both will use lower case field names and namespace is optional. ([#117238](https://github.com/kubernetes/kubernetes/pull/117238), [@pohly](https://github.com/pohly))
- The `GetAllocatableResources` podresources API endpoint is now GA. ([#118973](https://github.com/kubernetes/kubernetes/pull/118973), [@ffromani](https://github.com/ffromani))
- The `NetworkPolicyLegacy` test suite (deprecated in `v1.21`) has now officially been removed in favor of the new table driven e2e tests. ([#118915](https://github.com/kubernetes/kubernetes/pull/118915), [@astoycos](https://github.com/astoycos))
- The `generate_groups.sh` and `generate_internal_groups.sh` scripts from the `k8s.io/code-generator` repo are deprecated (but still work) in favor of `kube_codegen.sh` in that same repo. Projects which use the old scripts are encouraged to look at adopting the new one. ([#117897](https://github.com/kubernetes/kubernetes/pull/117897), [@thockin](https://github.com/thockin)) [SIG API Machinery]
- The feature gate `CSIStorageCapacity` have been removed and must no longer be referenced in `--feature-gates` flags. ([#118018](https://github.com/kubernetes/kubernetes/pull/118018), [@humblec](https://github.com/humblec))
- The feature gates `CSIMigrationGCE` is graduated to GA and were unconditionally enabled have been removed in `v1.25`, and the entire `gcepd` package has been removed. ([#117055](https://github.com/kubernetes/kubernetes/pull/117055), [@cyclinder](https://github.com/cyclinder))
- The feature gates `DisableAcceleratorUsageMetrics` and `PodSecurity` that graduated to GA and were unconditionally enabled have been removed in v1.28. ([#114068](https://github.com/kubernetes/kubernetes/pull/114068), [@cyclinder](https://github.com/cyclinder)) [SIG API Machinery, Node, Scheduling and Storage]
- The kubelet podresources endpoint is GA and always enabled. ([#116525](https://github.com/kubernetes/kubernetes/pull/116525), [@ffromani](https://github.com/ffromani)) [SIG Node]
- The metric `apiserver_flowcontrol_current_executing_seats` has been introduced as a duplicate of `apiserver_flowcontrol_request_concurrency_in_use` because the latter has a confusing name and will be removed in a later release. ([#118960](https://github.com/kubernetes/kubernetes/pull/118960), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery]
- Updated `Cluster Autosaler` to version 1.26.1. ([#116526](https://github.com/kubernetes/kubernetes/pull/116526), [@pacoxu](https://github.com/pacoxu)) [SIG Autoscaling and Cloud Provider]
- Updated `cri-tools` to `v1.27.0`. ([#117545](https://github.com/kubernetes/kubernetes/pull/117545), [@saschagrunert](https://github.com/saschagrunert))
- Updated `setcap` image to debian bookworm v1.0.0. ([#119247](https://github.com/kubernetes/kubernetes/pull/119247), [@saschagrunert](https://github.com/saschagrunert))
- Updated `cri-tools` to `v1.26.1`. ([#116649](https://github.com/kubernetes/kubernetes/pull/116649), [@saschagrunert](https://github.com/saschagrunert)) [SIG Architecture and Release]
- Updated debian-base image to `bookworm-v1.0.0`. ([#119095](https://github.com/kubernetes/kubernetes/pull/119095), [@saschagrunert](https://github.com/saschagrunert))
- Use table-driven test for `TestPerPodSchedulingMetrics`. ([#118842](https://github.com/kubernetes/kubernetes/pull/118842), [@helayoty](https://github.com/helayoty))
- When retrieving event resources, the `reportingController` and `reportingInstance` fields in the event will contain values. ([#116506](https://github.com/kubernetes/kubernetes/pull/116506), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG API Machinery and Instrumentation]
- [KCCM] drop filtering nodes for the providerID when syncing load balancers, but have changes to the field trigger a re-sync of load balancers. This should ensure that cloud providers which don't specify providerID, can still use the service controller implementation to provision load balancers. ([#117602](https://github.com/kubernetes/kubernetes/pull/117602), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider and Network]
- `kube-apiserver` added two new metrics `authorization_attempts_total` and `authorization_duration_seconds`
that allow users to monitor requests to authorization webhooks, split by result. ([#117211](https://github.com/kubernetes/kubernetes/pull/117211), [@HirazawaUi](https://github.com/HirazawaUi))
- `kube-apiserver`: Improved memory use when performing `GetList` on the cache. ([#116327](https://github.com/kubernetes/kubernetes/pull/116327), [@sxllwx](https://github.com/sxllwx))
- `kube-controller-manager` and `cloud-controller-manager` have changed the
name of controllers that can be turned on/off that are passed to the `--controllers`
flag (e.g., `pod-garbage-collector-controller`). The old names (eg `podgc`) are
also accepted and aliased to the new names. ([#115813](https://github.com/kubernetes/kubernetes/pull/115813), [@atiratree](https://github.com/atiratree))
- `kubeadm`: Introduced a new feature gate `UpgradeAddonsBeforeControlPlane` to
fix a kube-proxy skew policy misalignment. Its default value is `false`. Upgrade
of the CoreDNS and kube-proxy addons will now trigger after all the control plane
instances have been upgraded, unless the fearure gate is set to true. This feature
gate will be removed in a future release. ([#117660](https://github.com/kubernetes/kubernetes/pull/117660), [@pacoxu](https://github.com/pacoxu))
## Dependencies
### Added
- cloud.google.com/go/accessapproval: v1.6.0
- cloud.google.com/go/accesscontextmanager: v1.7.0
- cloud.google.com/go/aiplatform: v1.37.0
- cloud.google.com/go/analytics: v0.19.0
- cloud.google.com/go/apigateway: v1.5.0
- cloud.google.com/go/apigeeconnect: v1.5.0
- cloud.google.com/go/apigeeregistry: v0.6.0
- cloud.google.com/go/appengine: v1.7.1
- cloud.google.com/go/area120: v0.7.1
- cloud.google.com/go/artifactregistry: v1.13.0
- cloud.google.com/go/asset: v1.13.0
- cloud.google.com/go/assuredworkloads: v1.10.0
- cloud.google.com/go/automl: v1.12.0
- cloud.google.com/go/baremetalsolution: v0.5.0
- cloud.google.com/go/batch: v0.7.0
- cloud.google.com/go/beyondcorp: v0.5.0
- cloud.google.com/go/billing: v1.13.0
- cloud.google.com/go/binaryauthorization: v1.5.0
- cloud.google.com/go/certificatemanager: v1.6.0
- cloud.google.com/go/channel: v1.12.0
- cloud.google.com/go/cloudbuild: v1.9.0
- cloud.google.com/go/clouddms: v1.5.0
- cloud.google.com/go/cloudtasks: v1.10.0
- cloud.google.com/go/compute/metadata: v0.2.3
- cloud.google.com/go/compute: v1.19.0
- cloud.google.com/go/contactcenterinsights: v1.6.0
- cloud.google.com/go/container: v1.15.0
- cloud.google.com/go/containeranalysis: v0.9.0
- cloud.google.com/go/datacatalog: v1.13.0
- cloud.google.com/go/dataflow: v0.8.0
- cloud.google.com/go/dataform: v0.7.0
- cloud.google.com/go/datafusion: v1.6.0
- cloud.google.com/go/datalabeling: v0.7.0
- cloud.google.com/go/dataplex: v1.6.0
- cloud.google.com/go/dataproc: v1.12.0
- cloud.google.com/go/dataqna: v0.7.0
- cloud.google.com/go/datastream: v1.7.0
- cloud.google.com/go/deploy: v1.8.0
- cloud.google.com/go/dialogflow: v1.32.0
- cloud.google.com/go/dlp: v1.9.0
- cloud.google.com/go/documentai: v1.18.0
- cloud.google.com/go/domains: v0.8.0
- cloud.google.com/go/edgecontainer: v1.0.0
- cloud.google.com/go/errorreporting: v0.3.0
- cloud.google.com/go/essentialcontacts: v1.5.0
- cloud.google.com/go/eventarc: v1.11.0
- cloud.google.com/go/filestore: v1.6.0
- cloud.google.com/go/functions: v1.13.0
- cloud.google.com/go/gaming: v1.9.0
- cloud.google.com/go/gkebackup: v0.4.0
- cloud.google.com/go/gkeconnect: v0.7.0
- cloud.google.com/go/gkehub: v0.12.0
- cloud.google.com/go/gkemulticloud: v0.5.0
- cloud.google.com/go/gsuiteaddons: v1.5.0
- cloud.google.com/go/iam: v0.13.0
- cloud.google.com/go/iap: v1.7.1
- cloud.google.com/go/ids: v1.3.0
- cloud.google.com/go/iot: v1.6.0
- cloud.google.com/go/kms: v1.10.1
- cloud.google.com/go/language: v1.9.0
- cloud.google.com/go/lifesciences: v0.8.0
- cloud.google.com/go/logging: v1.7.0
- cloud.google.com/go/longrunning: v0.4.1
- cloud.google.com/go/managedidentities: v1.5.0
- cloud.google.com/go/maps: v0.7.0
- cloud.google.com/go/mediatranslation: v0.7.0
- cloud.google.com/go/memcache: v1.9.0
- cloud.google.com/go/metastore: v1.10.0
- cloud.google.com/go/monitoring: v1.13.0
- cloud.google.com/go/networkconnectivity: v1.11.0
- cloud.google.com/go/networkmanagement: v1.6.0
- cloud.google.com/go/networksecurity: v0.8.0
- cloud.google.com/go/notebooks: v1.8.0
- cloud.google.com/go/optimization: v1.3.1
- cloud.google.com/go/orchestration: v1.6.0
- cloud.google.com/go/orgpolicy: v1.10.0
- cloud.google.com/go/osconfig: v1.11.0
- cloud.google.com/go/oslogin: v1.9.0
- cloud.google.com/go/phishingprotection: v0.7.0
- cloud.google.com/go/policytroubleshooter: v1.6.0
- cloud.google.com/go/privatecatalog: v0.8.0
- cloud.google.com/go/pubsublite: v1.7.0
- cloud.google.com/go/recaptchaenterprise/v2: v2.7.0
- cloud.google.com/go/recommendationengine: v0.7.0
- cloud.google.com/go/recommender: v1.9.0
- cloud.google.com/go/redis: v1.11.0
- cloud.google.com/go/resourcemanager: v1.7.0
- cloud.google.com/go/resourcesettings: v1.5.0
- cloud.google.com/go/retail: v1.12.0
- cloud.google.com/go/run: v0.9.0
- cloud.google.com/go/scheduler: v1.9.0
- cloud.google.com/go/secretmanager: v1.10.0
- cloud.google.com/go/security: v1.13.0
- cloud.google.com/go/securitycenter: v1.19.0
- cloud.google.com/go/servicedirectory: v1.9.0
- cloud.google.com/go/shell: v1.6.0
- cloud.google.com/go/spanner: v1.45.0
- cloud.google.com/go/speech: v1.15.0
- cloud.google.com/go/storagetransfer: v1.8.0
- cloud.google.com/go/talent: v1.5.0
- cloud.google.com/go/texttospeech: v1.6.0
- cloud.google.com/go/tpu: v1.5.0
- cloud.google.com/go/trace: v1.9.0
- cloud.google.com/go/translate: v1.7.0
- cloud.google.com/go/video: v1.15.0
- cloud.google.com/go/videointelligence: v1.10.0
- cloud.google.com/go/vision/v2: v2.7.0
- cloud.google.com/go/vmmigration: v1.6.0
- cloud.google.com/go/vmwareengine: v0.3.0
- cloud.google.com/go/vpcaccess: v1.6.0
- cloud.google.com/go/webrisk: v1.8.0
- cloud.google.com/go/websecurityscanner: v1.5.0
- cloud.google.com/go/workflows: v1.10.0
- github.com/alecthomas/kingpin/v2: [v2.3.2](https://github.com/alecthomas/kingpin/v2/tree/v2.3.2)
- github.com/antlr/antlr4/runtime/Go/antlr/v4: [8188dc5](https://github.com/antlr/antlr4/runtime/Go/antlr/v4/tree/8188dc5)
- github.com/google/gnostic-models: [v0.6.8](https://github.com/google/gnostic-models/tree/v0.6.8)
- github.com/googleapis/enterprise-certificate-proxy: [v0.2.3](https://github.com/googleapis/enterprise-certificate-proxy/tree/v0.2.3)
- github.com/xhit/go-str2duration/v2: [v2.1.0](https://github.com/xhit/go-str2duration/v2/tree/v2.1.0)
- go.etcd.io/gofail: v0.1.0
- google.golang.org/genproto/googleapis/api: dd9d682
- google.golang.org/genproto/googleapis/rpc: 28d5490
### Changed
- cloud.google.com/go/bigquery: v1.8.0 → v1.50.0
- cloud.google.com/go/datastore: v1.1.0 → v1.11.0
- cloud.google.com/go/firestore: v1.1.0 → v1.9.0
- cloud.google.com/go/pubsub: v1.3.1 → v1.30.0
- cloud.google.com/go: v0.97.0 → v0.110.0
- github.com/Azure/azure-sdk-for-go: [v55.0.0+incompatible → v68.0.0+incompatible](https://github.com/Azure/azure-sdk-for-go/compare/v55.0.0...v68.0.0)
- github.com/Azure/go-autorest/autorest/adal: [v0.9.20 → v0.9.23](https://github.com/Azure/go-autorest/autorest/adal/compare/v0.9.20...v0.9.23)
- github.com/Azure/go-autorest/autorest/validation: [v0.1.0 → v0.3.1](https://github.com/Azure/go-autorest/autorest/validation/compare/v0.1.0...v0.3.1)
- github.com/Azure/go-autorest/autorest: [v0.11.27 → v0.11.29](https://github.com/Azure/go-autorest/autorest/compare/v0.11.27...v0.11.29)
- github.com/Microsoft/go-winio: [v0.4.17 → v0.6.0](https://github.com/Microsoft/go-winio/compare/v0.4.17...v0.6.0)
- github.com/alecthomas/units: [f65c72e → b94a6e3](https://github.com/alecthomas/units/compare/f65c72e...b94a6e3)
- github.com/cenkalti/backoff/v4: [v4.1.3 → v4.2.1](https://github.com/cenkalti/backoff/v4/compare/v4.1.3...v4.2.1)
- github.com/census-instrumentation/opencensus-proto: [v0.2.1 → v0.4.1](https://github.com/census-instrumentation/opencensus-proto/compare/v0.2.1...v0.4.1)
- github.com/cespare/xxhash/v2: [v2.1.2 → v2.2.0](https://github.com/cespare/xxhash/v2/compare/v2.1.2...v2.2.0)
- github.com/cilium/ebpf: [v0.7.0 → v0.9.1](https://github.com/cilium/ebpf/compare/v0.7.0...v0.9.1)
- github.com/cncf/udpa/go: [04548b0 → c52dc94](https://github.com/cncf/udpa/go/compare/04548b0...c52dc94)
- github.com/cncf/xds/go: [cb28da3 → 06c439d](https://github.com/cncf/xds/go/compare/cb28da3...06c439d)
- github.com/cockroachdb/datadriven: [bf6692d → v1.0.2](https://github.com/cockroachdb/datadriven/compare/bf6692d...v1.0.2)
- github.com/container-storage-interface/spec: [v1.7.0 → v1.8.0](https://github.com/container-storage-interface/spec/compare/v1.7.0...v1.8.0)
- github.com/containerd/cgroups: [v1.0.1 → v1.1.0](https://github.com/containerd/cgroups/compare/v1.0.1...v1.1.0)
- github.com/containerd/ttrpc: [v1.1.0 → v1.2.2](https://github.com/containerd/ttrpc/compare/v1.1.0...v1.2.2)
- github.com/coredns/caddy: [v1.1.0 → v1.1.1](https://github.com/coredns/caddy/compare/v1.1.0...v1.1.1)
- github.com/coreos/go-oidc: [v2.1.0+incompatible → v2.2.1+incompatible](https://github.com/coreos/go-oidc/compare/v2.1.0...v2.2.1)
- github.com/coreos/go-semver: [v0.3.0 → v0.3.1](https://github.com/coreos/go-semver/compare/v0.3.0...v0.3.1)
- github.com/coreos/go-systemd/v22: [v22.4.0 → v22.5.0](https://github.com/coreos/go-systemd/v22/compare/v22.4.0...v22.5.0)
- github.com/docker/distribution: [v2.8.1+incompatible → v2.8.2+incompatible](https://github.com/docker/distribution/compare/v2.8.1...v2.8.2)
- github.com/dustin/go-humanize: [v1.0.0 → v1.0.1](https://github.com/dustin/go-humanize/compare/v1.0.0...v1.0.1)
- github.com/envoyproxy/go-control-plane: [49ff273 → v0.10.3](https://github.com/envoyproxy/go-control-plane/compare/49ff273...v0.10.3)
- github.com/envoyproxy/protoc-gen-validate: [v0.1.0 → v0.9.1](https://github.com/envoyproxy/protoc-gen-validate/compare/v0.1.0...v0.9.1)
- github.com/evanphx/json-patch: [v4.12.0+incompatible → v5.6.0+incompatible](https://github.com/evanphx/json-patch/compare/v4.12.0...v5.6.0)
- github.com/frankban/quicktest: [v1.11.3 → v1.14.0](https://github.com/frankban/quicktest/compare/v1.11.3...v1.14.0)
- github.com/fvbommel/sortorder: [v1.0.1 → v1.1.0](https://github.com/fvbommel/sortorder/compare/v1.0.1...v1.1.0)
- github.com/go-kit/log: [v0.2.0 → v0.2.1](https://github.com/go-kit/log/compare/v0.2.0...v0.2.1)
- github.com/go-logr/logr: [v1.2.3 → v1.2.4](https://github.com/go-logr/logr/compare/v1.2.3...v1.2.4)
- github.com/go-openapi/jsonreference: [v0.20.1 → v0.20.2](https://github.com/go-openapi/jsonreference/compare/v0.20.1...v0.20.2)
- github.com/go-task/slim-sprig: [348f09d → 52ccab3](https://github.com/go-task/slim-sprig/compare/348f09d...52ccab3)
- github.com/gofrs/uuid: [v4.0.0+incompatible → v4.4.0+incompatible](https://github.com/gofrs/uuid/compare/v4.0.0...v4.4.0)
- github.com/golang-jwt/jwt/v4: [v4.4.2 → v4.5.0](https://github.com/golang-jwt/jwt/v4/compare/v4.4.2...v4.5.0)
- github.com/google/cadvisor: [v0.47.1 → v0.47.3](https://github.com/google/cadvisor/compare/v0.47.1...v0.47.3)
- github.com/google/cel-go: [v0.12.6 → v0.16.0](https://github.com/google/cel-go/compare/v0.12.6...v0.16.0)
- github.com/google/gofuzz: [v1.1.0 → v1.2.0](https://github.com/google/gofuzz/compare/v1.1.0...v1.2.0)
- github.com/googleapis/gax-go/v2: [v2.1.1 → v2.7.1](https://github.com/googleapis/gax-go/v2/compare/v2.1.1...v2.7.1)
- github.com/inconshreveable/mousetrap: [v1.0.1 → v1.1.0](https://github.com/inconshreveable/mousetrap/compare/v1.0.1...v1.1.0)
- github.com/kr/pretty: [v0.3.0 → v0.3.1](https://github.com/kr/pretty/compare/v0.3.0...v0.3.1)
- github.com/matttproud/golang_protobuf_extensions: [v1.0.2 → v1.0.4](https://github.com/matttproud/golang_protobuf_extensions/compare/v1.0.2...v1.0.4)
- github.com/mitchellh/go-wordwrap: [v1.0.0 → v1.0.1](https://github.com/mitchellh/go-wordwrap/compare/v1.0.0...v1.0.1)
- github.com/mitchellh/mapstructure: [v1.4.1 → v1.1.2](https://github.com/mitchellh/mapstructure/compare/v1.4.1...v1.1.2)
- github.com/onsi/ginkgo/v2: [v2.9.1 → v2.9.4](https://github.com/onsi/ginkgo/v2/compare/v2.9.1...v2.9.4)
- github.com/onsi/gomega: [v1.27.4 → v1.27.6](https://github.com/onsi/gomega/compare/v1.27.4...v1.27.6)
- github.com/opencontainers/runc: [v1.1.4 → v1.1.7](https://github.com/opencontainers/runc/compare/v1.1.4...v1.1.7)
- github.com/prometheus/client_golang: [v1.14.0 → v1.16.0](https://github.com/prometheus/client_golang/compare/v1.14.0...v1.16.0)
- github.com/prometheus/client_model: [v0.3.0 → v0.4.0](https://github.com/prometheus/client_model/compare/v0.3.0...v0.4.0)
- github.com/prometheus/common: [v0.37.0 → v0.44.0](https://github.com/prometheus/common/compare/v0.37.0...v0.44.0)
- github.com/prometheus/procfs: [v0.8.0 → v0.10.1](https://github.com/prometheus/procfs/compare/v0.8.0...v0.10.1)
- github.com/seccomp/libseccomp-golang: [f33da4d → v0.10.0](https://github.com/seccomp/libseccomp-golang/compare/f33da4d...v0.10.0)
- github.com/spf13/cobra: [v1.6.0 → v1.7.0](https://github.com/spf13/cobra/compare/v1.6.0...v1.7.0)
- github.com/stretchr/testify: [v1.8.1 → v1.8.2](https://github.com/stretchr/testify/compare/v1.8.1...v1.8.2)
- github.com/vishvananda/netns: [v0.0.2 → v0.0.4](https://github.com/vishvananda/netns/compare/v0.0.2...v0.0.4)
- github.com/xlab/treeprint: [v1.1.0 → v1.2.0](https://github.com/xlab/treeprint/compare/v1.1.0...v1.2.0)
- go.etcd.io/bbolt: v1.3.6 → v1.3.7
- go.etcd.io/etcd/api/v3: v3.5.7 → v3.5.9
- go.etcd.io/etcd/client/pkg/v3: v3.5.7 → v3.5.9
- go.etcd.io/etcd/client/v2: v2.305.7 → v2.305.9
- go.etcd.io/etcd/client/v3: v3.5.7 → v3.5.9
- go.etcd.io/etcd/pkg/v3: v3.5.7 → v3.5.9
- go.etcd.io/etcd/raft/v3: v3.5.7 → v3.5.9
- go.etcd.io/etcd/server/v3: v3.5.7 → v3.5.9
- go.opencensus.io: v0.23.0 → v0.24.0
- go.starlark.net: 8dd3e2e → a134d8f
- go.uber.org/atomic: v1.7.0 → v1.10.0
- go.uber.org/multierr: v1.6.0 → v1.11.0
- golang.org/x/crypto: v0.1.0 → v0.11.0
- golang.org/x/exp: 6cc2880 → a9213ee
- golang.org/x/mod: v0.9.0 → v0.10.0
- golang.org/x/net: v0.8.0 → v0.13.0
- golang.org/x/oauth2: ee48083 → v0.8.0
- golang.org/x/sync: v0.1.0 → v0.2.0
- golang.org/x/sys: v0.6.0 → v0.10.0
- golang.org/x/term: v0.6.0 → v0.10.0
- golang.org/x/text: v0.8.0 → v0.11.0
- golang.org/x/time: 90d013b → v0.3.0
- golang.org/x/tools: v0.7.0 → v0.8.0
- google.golang.org/api: v0.60.0 → v0.114.0
- google.golang.org/genproto: c8bf987 → 0005af6
- google.golang.org/grpc: v1.51.0 → v1.54.0
- google.golang.org/protobuf: v1.28.1 → v1.30.0
- gopkg.in/gcfg.v1: v1.2.0 → v1.2.3
- gopkg.in/natefinch/lumberjack.v2: v2.0.0 → v2.2.1
- gopkg.in/warnings.v0: v0.1.1 → v0.1.2
- k8s.io/klog/v2: v2.90.1 → v2.100.1
- k8s.io/kube-openapi: 15aac26 → 2695361
- k8s.io/utils: a36077c → d93618c
- sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.1.1 → v0.1.2
- sigs.k8s.io/kustomize/api: v0.13.2 → 6ce0bf3
- sigs.k8s.io/kustomize/cmd/config: v0.11.1 → v0.11.2
- sigs.k8s.io/kustomize/kustomize/v5: v5.0.1 → 6ce0bf3
- sigs.k8s.io/kustomize/kyaml: v0.14.1 → 6ce0bf3
### Removed
- github.com/antlr/antlr4/runtime/Go/antlr: [v1.4.10](https://github.com/antlr/antlr4/runtime/Go/antlr/tree/v1.4.10)
- github.com/certifi/gocertifi: [2c3bb06](https://github.com/certifi/gocertifi/tree/2c3bb06)
- github.com/cockroachdb/errors: [v1.2.4](https://github.com/cockroachdb/errors/tree/v1.2.4)
- github.com/cockroachdb/logtags: [eb05cc2](https://github.com/cockroachdb/logtags/tree/eb05cc2)
- github.com/docopt/docopt-go: [ee0de3b](https://github.com/docopt/docopt-go/tree/ee0de3b)
- github.com/getsentry/raven-go: [v0.2.0](https://github.com/getsentry/raven-go/tree/v0.2.0)
- github.com/google/gnostic: [v0.5.7-v3refs](https://github.com/google/gnostic/tree/v0.5.7-v3refs)
# v1.28.0-rc.1
## Downloads for v1.28.0-rc.1
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes.tar.gz) | efc4ec914eb4e1147cdbadb9a5dccc4608a983ba6308c85d4c2e8e1c984f35c12e04b027d0a0f6e07c2371fae9aa4879b4831158e7cfe77887da7e20778e717b
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-src.tar.gz) | cfdd470979b447dab1678c03bd9bc7745ef7d6907a043d5180e96494d4a5b91b4d8a08b09726e15cda4668437cc296528df646f5f58c870af8134312cf8851ba
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-darwin-amd64.tar.gz) | ca12f940ef00fc38152dae75d8f817b03be4d59c7a8d64a80a515fbbf78d526a2b98311efbc9a8d34361b70ba0b07156cff7fbb6c19dc503c7c16e0dfc8e3ec0
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-darwin-arm64.tar.gz) | 9d1d0aaedd698a4c5ad5ed2903e8492e52e6f16f858b8d186129edb5c9b199d4352e443c8aba67f58db7fcf950162ffbb4f7211570a0c4be684656ad5ada42bf
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-linux-386.tar.gz) | 59e1c0692821eb64b59147baf43985226ae2223fff66981a2a13d5d60d5e102faf7c5cecfa5b8ec1c5a48c9a9fed35223f8ebbc93ac972ea949f2a3096f64672
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-linux-amd64.tar.gz) | a348ad33f936e075083e2ffec4405a726984df8c522e10d34ad65b64eee53902fb6483903581c30b047872fd130cf24f6cdc193458fcf7d5774364bf78c1c982
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-linux-arm.tar.gz) | 3069b3000445218a98b7d11bd196cbfb84aed16dcbd16adff88935bb01a87f8da29cc4824de4c8af7ddc2050134e2c3467408218fc7209700c0e1c0aec2d3ced
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-linux-arm64.tar.gz) | d59ebea7b7d78ee1ef59317073a5a4b9e513b9f43026ffc1b7745211e8cd9de738a05d1fa2c29501d3cee24252732c3348f109f2b7c1e7425a4eff46cf1b4654
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-linux-ppc64le.tar.gz) | 2490266ee44469c56bd85f86774668bc9a6d6f2d4f21ddeb95da8eb638f743501e9ed3646c817db0c746730f93b381b6c5ae46d25d9dbadc78d3ca8f89eccfd6
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-linux-s390x.tar.gz) | a843bc8df85594ab5fca9e1e17997e14cdf8e45ef2e74165222b6dc8d26a8f85d85a972a5c3e1740f3eb6d3647b81e3dfb66787cdac6dcd42a59c2f5507f6031
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-windows-386.tar.gz) | 1ce7d2802cb9ebbc61f68ae3a8380056fc039e9959ea999f3d7da1254b33225809932a9679b2a0f96ca4adad73aa4aedba3ec9f20182899f62ff59133e48f4d1
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-windows-amd64.tar.gz) | 85986399b437aa48d276301a7a06b17e4004d5423dec2faf699a3a377dd28f3e734b0655848168407fb25b6898389daf45f0ad695519e1f3f31586e9a8586531
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-client-windows-arm64.tar.gz) | ac18cf32f0273470841024a762d7ded78993128bada439340d0f4c604af6d7001971f3075437e65471ab7ae89d15fd82f2689b6d47da681dcc8779c277a9cea5
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-server-linux-amd64.tar.gz) | 6b5ebfe42050e48f108da43275cdfb0b5ec652867d12a632bf5ed4b00482efd2470184028bf94b36a1f05c5a70ad1057f334483461f9212bdb48dcac6b169600
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-server-linux-arm64.tar.gz) | 09b1a44182ab30a4740b703cae5f46ce4cb4af9716ed1f2cb2a4dc59c9baa2eaa402b01167b04c6801b550035ba9f939d4d1209689363daffe870dd2f44e4528
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-server-linux-ppc64le.tar.gz) | 09ba87bfb42f7f710b446a588d2047fe6aea26df171aacc3157c3fa4c9e718856ad3efc45b0050d35a9153e94d5da81c632ddaec71663d30c5d43284292b305f
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-server-linux-s390x.tar.gz) | 42ae7f05cb279e57ef89570b1596759cc771663ceb72f358a9119c91b981b99335b46a887f59f8a8727303366c3111bd4696817343ddbee3ee02811bd6022e4c
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-node-linux-amd64.tar.gz) | 4a0c7ae5ce52698087eaca1ec108ca5c1333ddc607a5fcb63d5e65cde17e3c8e64037905da02656e4a663037be1b00441754af4563c5eed1ec8ab57bf692c4ed
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-node-linux-arm64.tar.gz) | aba8ddbc9cd9cf0e2fd5eaac8bee2490135c31fcd43751dd5e4438eca813372104e99f34517acfdd2abddf3a28cffef4cc42eb9bfecd76b50d89adff5675f32b
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-node-linux-ppc64le.tar.gz) | d16d79c2f5680d31aa6fcc659de210632d0ed761b6c4c067ab5976fe41360a9cc7a75cad545e04831bf9d2b8669523dd9cf4c756337c5328ff10a8d61a5301bc
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-node-linux-s390x.tar.gz) | f7bb37de732b35db011f1d2f52ac461c7f912de39bb16c13a638b2dcd160876c6d1e278d36fdee07d8598b69b30e33f7c9bc980b6b25651e4b74cf3517514371
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.1/kubernetes-node-windows-amd64.tar.gz) | 86114fbfa8980e678bddb3d01290b5b3158d92ffdc92b970d1a224fff1f7914415c9adb3b663f5f036d5abb0aad95c96c1a819b309e157aba9392a6c77e65ff5
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.0-rc.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.28.0-rc.0
## Changes by Kind
### API Change
- Aggregated discovery now returns `responseKind: {}` for resources which are missing group/version/kind information, to ensure compatibility with v0.26.0-v0.26.3 clients. ([#119835](https://github.com/kubernetes/kubernetes/pull/119835), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Testing]
- Fix CustomResourceDefinition status.storedVersions validation error messages. ([#119653](https://github.com/kubernetes/kubernetes/pull/119653), [@sttts](https://github.com/sttts)) [SIG API Machinery]
- Kube-proxy in Kubernetes >= 1.28 up until v1.28.0-beta.0 ignored the `-v` command line flag when combined with `--config`. ([#119867](https://github.com/kubernetes/kubernetes/pull/119867), [@pohly](https://github.com/pohly)) [SIG Network]
### Feature
- Bump distroless-iptables to 0.2.7 based on Go 1.20.7 ([#119818](https://github.com/kubernetes/kubernetes/pull/119818), [@jeremyrickard](https://github.com/jeremyrickard)) [SIG Testing]
- Kubernetes is now built with Go 1.20.7 ([#119804](https://github.com/kubernetes/kubernetes/pull/119804), [@jeremyrickard](https://github.com/jeremyrickard)) [SIG Release and Testing]
### Bug or Regression
- Fixes issue https://github.com/kubernetes-sigs/cloud-provider-azure/issues/4230 and removes the additional filtering on `NotReady` nodes by the azure cloud provider code ([#119128](https://github.com/kubernetes/kubernetes/pull/119128), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider]
- Kube-scheduler: Fine-grained tracking of events (introduced in 1.28) suffered from a data race when binding fails. ([#119729](https://github.com/kubernetes/kubernetes/pull/119729), [@pohly](https://github.com/pohly)) [SIG Scheduling]
- Revert kubelet prober metrics `pod` tag to include actual pod name ([#118549](https://github.com/kubernetes/kubernetes/pull/118549), [@a7i](https://github.com/a7i)) [SIG Node]
- When the cluster size is small and the scheduler doesn't get unscheduled Pods frequently, the scheduler doesn't try to reschedule Pods in some cases. ([#119784](https://github.com/kubernetes/kubernetes/pull/119784), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling and Testing]
## Dependencies
### Added
_Nothing has changed._
### Changed
- golang.org/x/net: v0.12.0 → v0.13.0
### Removed
_Nothing has changed._
# v1.28.0-rc.0
## Downloads for v1.28.0-rc.0
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes.tar.gz) | bd3feaf924371be8fe3e2130c6cac5fd5fd3c90d42be383e076c16160c95ec48668b5b330e0742d562a3b0eae18eda71bab76dff5e2aebad61513c2be6b251b6
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-src.tar.gz) | 9f3fbb4c624124bf82473028e9c21a123f525e9dc8a224ede4f00fbf0630ae812ba58d56be69aa45a2b039da1deb4ce9052061b40699945f9fb88bb59fcb3977
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-darwin-amd64.tar.gz) | fabc769eef82c242d905cee81d7e876280adeffa95208ddd4d1c0de69e82d775afef984cc9b3b375ee2072e8dceb846da105c76c94e6d323f9778c9e9c0b49ba
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-darwin-arm64.tar.gz) | 0a321eaa366d73feab93bab77e56dbd8ee2fde298766a8f7e37b98f6053c4e86ae77f64561bd083cd5f3b4f61d727de013d6ef74a0bd0c35b7afb8cb110a063c
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-linux-386.tar.gz) | a8720f87b0eafefca413547e3f603660d147b9198eabac03ac59a57a70fc011c48daaef4cf1d63a974578e8b20f98e6f3dca7997f6feaa009944e16ec47ea8c9
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-linux-amd64.tar.gz) | 918dc97380ebe56c16de8131d35f9ddc21b2196f8b6b0b24361fa4a23a2cf56c75edf4555eccbad3453663c007bd51d147e0a589f933e0759410879e2aeadfd8
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-linux-arm.tar.gz) | c1f87fff085884632cb1b60c72f4cf168079bff4150270e67855b1211a1240e3252e6791dfb61672a2ffbe4314b360b917657608b3c65d661871852ba84e8ca3
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-linux-arm64.tar.gz) | 2f33cb523472e162dc206edee4e33903a4550d3f73adbe327c7c34f8084b44dd0ed6b6b28c9d85eca4e6e2a1d1124c9dbc0d8a4a14abc7b810a7e4085f5e97bc
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-linux-ppc64le.tar.gz) | 9edc3618d12e4480ccb448e928017bd9c3f1e8ad16bc83cddea4a73c81ab2d7a5085bbedf2a0324a9377d17faa7168e2d5c27a7de5bed8e07809b1227c4b9079
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-linux-s390x.tar.gz) | 53a31035114aaa7d837dcdc7663a9ee91e6d49d549eea6c7337a6f3a4325b34a6931e65ce471a758bed152a92adb434581084e2810b16c8143582501b48e4363
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-windows-386.tar.gz) | 8158c3947c5838fef84b6427fb27e6cf2375344f6d27bd1a2d0563d3a242bc445278968bcf36b7657a4db7322b2a9d5aad028480c6bb5fbf2faf3a2dddad931e
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-windows-amd64.tar.gz) | aa809425d557fc9323748a0484f9b59f9f6e089ab6256da5690014efbda9a9c1a96110a3511f930e4e2714315005bd803ea059b1a5221a825f109a69d6c60967
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-client-windows-arm64.tar.gz) | f72acfa3ac61cbf7fc2b612a9c8c9d19de42488752120e6b4d69fafa35beda00bfbe9dc839430b3987ff5e9737930d0a9fae867ab35824f0a0eef47b6496404d
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-server-linux-amd64.tar.gz) | a3ca2b91e204f6bcf84e1d54412ca6c908a1e02a48b8022da821bfd4fe136b565a980da3fef2270a34ca637ff4ff306cd3b09760556602db15a7b8a3dfead0d1
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-server-linux-arm64.tar.gz) | 40e6aea2c1fa0f9f6c135363991216d80b927c6aabcf30fc3288512c38fd75b8fff868770209cde8016a2a33dee7ad862709840c7a040f6203240b06bc2e5c5d
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-server-linux-ppc64le.tar.gz) | da41491b39fb8a1a2f7619f14ff1165ff0d467bef82348e71bfd31d481678dff27b03a01d3fa2deccdf04227a0fe7c9593d8d7b9e7745070afcb53dfd70b2bc6
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-server-linux-s390x.tar.gz) | 6deb9a3625f510cdd5dea4ced9648fe563595cddfd2f9223e4acbf3fe74cd2b4283011984105f79794731e81b7b3725a16c964b70f25a4bad76b60b75f54372d
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-node-linux-amd64.tar.gz) | d4178a311495afe6e9d4a5c4fa9b1d0d17c7a0918305b782406ea5f5fbfcd8c4f60073e02b9aff3c37eb9f4fd331844177e48650b4e489b0a5430da5e00a33a2
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-node-linux-arm64.tar.gz) | 8ee5e87248caca033fafea7ea1470fcc282cc402f6591d9120fd87c4e533bade19e125c1a840d1ebf503fb0eda21096a047571ffde79a6b0263494799bdf042f
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-node-linux-ppc64le.tar.gz) | 022f4804d408788d426176f5dda8046005084261c426766476b429e2d62f60e22509d66e9199350ba501b17f6e6cb72e7e4bd2581b950c85e5c5a05efcb0139c
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-node-linux-s390x.tar.gz) | d9e95ce6e0f886d8a89df8d53f2533c9be2ef733be25a7455cfc89028ee5ed5b7fbdfb81f7c3e452b405dc8b3d4c252f74039d300a0fd01c7b75e7b5f6c0c551
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-rc.0/kubernetes-node-windows-amd64.tar.gz) | 6bd4809f6b8d1bea8da4565ddda294d39eb3ab424dd47e01a5919e974b33316c64fc21bc8b2c62a05f14db71621951ecc82c5988da11b8190a6e1ee2eec79cd5
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.0-rc.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.28.0-beta.0
## Changes by Kind
### API Change
- PersistentVolumes have a new LastPhaseTransitionTime field which holds a timestamp of when the volume last transitioned its phase. ([#116469](https://github.com/kubernetes/kubernetes/pull/116469), [@RomanBednar](https://github.com/RomanBednar)) [SIG API Machinery, Apps, Auth, Node, Release, Storage and Testing]
- Promoted API groups `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding` to `v1beta1`. ([#118644](https://github.com/kubernetes/kubernetes/pull/118644), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Apps and Testing]
- Promoted the feature gate `ValidtaingAdmissionPolicy` to beta and it is turned off by default. ([#119409](https://github.com/kubernetes/kubernetes/pull/119409), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Apps, Auth, Instrumentation, Node, Release, Storage and Testing]
- Changed how KMS v2 encryption at rest can generate data encryption keys. When you enable the `KMSv2KDF` feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. ([#118828](https://github.com/kubernetes/kubernetes/pull/118828), [@enj](https://github.com/enj)) [SIG API Machinery, Auth and Testing]
### Feature
- Add implementation for PodRecreationPolicy to wait for creation of pods once the existing ones are fully terminated. ([#117015](https://github.com/kubernetes/kubernetes/pull/117015), [@kannon92](https://github.com/kannon92)) [SIG API Machinery, Apps and Testing]
## Dependencies
### Added
_Nothing has changed._
### Changed
_Nothing has changed._
### Removed
_Nothing has changed._
# v1.28.0-beta.0
## Downloads for v1.28.0-beta.0
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes.tar.gz) | 5a4575157380ff5dd66fd87e7045f2f57ed0db59513bfef57ee768a7a98f855faa06503a7480e77cdf5128fe66461a6c91f0705f8148347f903342f45b65f8da
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-src.tar.gz) | 8efebf779daf168929dd2798d2e52750d09339a17256091b4315b1de82167b26388725a60bc3525468b1a23656932faf1c45ba6957df0bd2b3f48bafc8b62138
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-darwin-amd64.tar.gz) | 956bd780cb36815ae8969e345d4f6004740167f3e5e3a1d1b1deda254ac2b167371b7c9e79497bd01e3f11d2e2f8e1c35f8fdc3114f08324470635cff1efab20
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-darwin-arm64.tar.gz) | 68ebd65edb40e23c5e70b6666fea34a774ccd66313dc884e28a8032cfd8c166c1a2dc66c635a61b0d568fe825b208bfd32a040e3eded536f71617acac625f3ea
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-linux-386.tar.gz) | 0291c01019e47af1631f3f90f46aa92e4572301fe2411586fb2d69291fdf113a8ff78531f51530d05c6113e28e0e69e23f1d2e2143f832f6b8f77a133b09e493
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-linux-amd64.tar.gz) | bd9a98bd35ed9bb113399e50456da7008629e060381a7f6b9071fd7bcd498cb252da2824376662a4d4d4cae637feb90f3768414751e7ec0339c6c9711f6aceda
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-linux-arm.tar.gz) | 311b3c1a42d68fa337f6e8b2290817196c97024535ee94f2180971e09ca78c9037b41fca48c1eda1fa75079631ea8805f0c6a173e35fe4a9762d13bf3bdf1c58
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-linux-arm64.tar.gz) | 039491fddc63e279821ff5287cada111b5c94dc0389d697adc2503f77905a74cda1a67dd52d4b42be0141896d04cfb18f3dfc0c387620d9a9fb3a7f2aa5b399e
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-linux-ppc64le.tar.gz) | 70e4011dce6777c511a69872a1e015b3a2137cea100cd6b988946ea227bd05827f4223a44c9d24433043cc414d3746465603dc4de2e84128e7689712dd29b00d
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-linux-s390x.tar.gz) | 51ea51fbcdcf7f2c9c4c97b6f28ceb42753e52ccf69318c05befa77b94d9c17e871061b4d03dbc12632663efed161424b3dba97ac6df46cf27d6deb1a0c011fb
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-windows-386.tar.gz) | e069121289664d53a6570d90ac2d66911d18cf8c47461f5e8e2cd52ddd651a0e119327d23455b9c273e979378379d27e5cf102c7f8b8c98871c9ef9c7e790e52
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-windows-amd64.tar.gz) | f19a06ad641a282b9895673a0628cc937222b53d9b852fd7543de01294d403100d0bcd4659fbf73bf8cdc55a8e3c7f494991db2b9f4d8bc63446b7810232d3aa
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-client-windows-arm64.tar.gz) | 7bd6a6d6ae7b487aa35c162532e9b59e184aac0fb6b65c61b2af06217007f78aeb5d0bed039983c534358152155739a6e30f533488c49a13d3c81d54eea6b8c6
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-server-linux-amd64.tar.gz) | 7cd83dc6e7013ef8638965fb3e30f8ba122f045d987029da345c185662bea824fc6a3fae34ff549c457638daf703833312893180538552d194fcc7f4fb0642fd
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-server-linux-arm64.tar.gz) | 07e9304f6864f4334b9710ad1a130044c9b1349bf9e47e5c9857b688322abe5babf45a95cdbad3f2650b5447c11864edc3d50fa86de5d485e84730260efbdc8e
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-server-linux-ppc64le.tar.gz) | 84f7c2b7021bd136d0c9013b349d63102433f7485552a0f19cb61f4630f256aef1f99a54f9da14b5d6c242778488c5539e0c358b9e421aafaf746ce783773e9b
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-server-linux-s390x.tar.gz) | 80263d622d219ae687ea24d9b552c5e6fcb920edc6adc73fe5c742cbb34db08a045f52babb0e6c5acfab98616b9f9f2c87150db9a04f5799836b8c8fd0709f31
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-node-linux-amd64.tar.gz) | bb24b87e1971fde06882d4a2b91b2663ee9daad1e1f8f31f457eaac235b26466ec2413d947d06803b8fc9c356e56e77f7ed31b1f021eae0fdb3df426bc610717
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-node-linux-arm64.tar.gz) | 91aa0771cf8f6615655aeda2484d967f53dbe10cee7ac724be23570d5ca60e3fe11e354cd8b715b882ba20534dab67fee505cb2cd6df1c90d124f778eadff67b
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-node-linux-ppc64le.tar.gz) | e30e0efdd8ee02b3ee9c4da1e106a41ffdff7606401f44647e6bd03ebf4ef1900bf7c7d5a2382412a0e2b4bd7013e04e3baa08637d4dbe2b2993fcdab7e2378d
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-node-linux-s390x.tar.gz) | e011016917d19bc5b84a76899a3eb6d7e2a6bf270e2d799ba77a9c90daabaf2055655ed0b6a62f2b9e5edfbd4b902e6e4a2408d2dc5c63a19a706d3f838a3864
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-beta.0/kubernetes-node-windows-amd64.tar.gz) | b9f365607de7112b2e62458462f1261905b1731d40c125db228bbc5aa48cc108872552db3c873702df0607bc0eb1996a1dca16cd27526f002bf1842e2eefc4ef
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.0-beta.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.0-beta.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.0-beta.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.0-beta.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.0-beta.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.0-beta.0](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.28.0-alpha.4
## Changes by Kind
### Deprecation
- Changed `kubectl version` default output to be identical to what `kubectl version --short` printed,
and remove `--short` flag entirely. ([#116720](https://github.com/kubernetes/kubernetes/pull/116720), [@soltysh](https://github.com/soltysh)) [SIG CLI and Testing]
- Deprecated support for CSI migration of Ceph RBD volumes.
Users who were relying on Kubernetes' ability to migrate to an out-of-tree storage driver should complete
that migration before the support for it is removed. ([#118303](https://github.com/kubernetes/kubernetes/pull/118303), [@carlory](https://github.com/carlory)) [SIG Storage]
- KMSv1 is deprecated and will only receive security updates going forward. Use KMSv2 instead. In the future, set --feature-gates=KMSv1=true to use the deprecated KMSv1 feature. ([#119007](https://github.com/kubernetes/kubernetes/pull/119007), [@aramase](https://github.com/aramase)) [SIG API Machinery and Auth]
- The deprecated flag `--lock-object-namespace` and `--lock-object-name` have been removed from kube-scheduler. Please use `--leader-elect-resource-namespace` and `--leader-elect-resource-name` or ComponentConfig instead to configure those parameters. ([#119130](https://github.com/kubernetes/kubernetes/pull/119130), [@SataQiu](https://github.com/SataQiu)) [SIG Scheduling]
### API Change
- A CDIDevice field is includes in the Device Plugin's `ContainerAllocateResponse`. This field maps to the CDIDevice field in the CRI protocol. ([#118254](https://github.com/kubernetes/kubernetes/pull/118254), [@elezar](https://github.com/elezar)) [SIG Node and Testing]
- Add new annotation `batch.kubernetes.io/cronjob-scheduled-timestamp` to Job objects scheduled from CronJobs. ([#118137](https://github.com/kubernetes/kubernetes/pull/118137), [@helayoty](https://github.com/helayoty)) [SIG Apps]
- Add podReplacementPolicy and terminating field to job api ([#119301](https://github.com/kubernetes/kubernetes/pull/119301), [@kannon92](https://github.com/kannon92)) [SIG API Machinery and Apps]
- Added fields `reason` and `fieldPath` into CRD validation rules to allow users to specify reason and field path when validation failed. ([#118041](https://github.com/kubernetes/kubernetes/pull/118041), [@cici37](https://github.com/cici37)) [SIG API Machinery]
- Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a `namespaceObject`
variable with expressions. ([#118267](https://github.com/kubernetes/kubernetes/pull/118267), [@cici37](https://github.com/cici37)) [SIG API Machinery and Testing]
- Adds new CRDValidationRatcheting alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. ([#118990](https://github.com/kubernetes/kubernetes/pull/118990), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
- Adds new namespaceParamRef to admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy ([#119215](https://github.com/kubernetes/kubernetes/pull/119215), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing]
- Extend the Job API for alpha version of BackoffLimitPerIndex ([#119294](https://github.com/kubernetes/kubernetes/pull/119294), [@mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]
- Graduate `AdmissionWebhookMatchCondition` feature to beta ([#119380](https://github.com/kubernetes/kubernetes/pull/119380), [@a-hilaly](https://github.com/a-hilaly)) [SIG API Machinery]
- In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . ([#118782](https://github.com/kubernetes/kubernetes/pull/118782), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery]
- Indexed Job pods now have the pod completion index set as a pod label. ([#118883](https://github.com/kubernetes/kubernetes/pull/118883), [@danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps]
- Kube-proxy: add '--logging-format' flag to support structured logging ([#117800](https://github.com/kubernetes/kubernetes/pull/117800), [@cyclinder](https://github.com/cyclinder)) [SIG API Machinery, Architecture, Instrumentation and Network]
- Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to `BETA` stability. ([#119264](https://github.com/kubernetes/kubernetes/pull/119264), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]
- Removed `resizeStatus` enum from `pvc.Status` and replaced with `AllocatedResourceStatus` ([#116335](https://github.com/kubernetes/kubernetes/pull/116335), [@gnufied](https://github.com/gnufied)) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
- StatefulSet pods now have the pod index set as a pod label `statefulset.kubernetes.io/pod-index`. ([#119232](https://github.com/kubernetes/kubernetes/pull/119232), [@danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps]
- Support BackoffLimitPerIndex in Jobs ([#118009](https://github.com/kubernetes/kubernetes/pull/118009), [@mimowo](https://github.com/mimowo)) [SIG API Machinery, Apps and Testing]
- Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver ([#117740](https://github.com/kubernetes/kubernetes/pull/117740), [@Richabanker](https://github.com/Richabanker)) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing]
- The IPTablesOwnershipCleanup feature (KEP-3178) is now GA; kubelet no longer
creates the KUBE-MARK-DROP chain (which has been unused for several releases)
or the KUBE-MARK-MASQ chain (which is now only created by kube-proxy). ([#119374](https://github.com/kubernetes/kubernetes/pull/119374), [@danwinship](https://github.com/danwinship)) [SIG API Machinery, Network and Node]
- The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still `<pod>-<claim name>`, but a random suffix will avoid name collisions. ([#117351](https://github.com/kubernetes/kubernetes/pull/117351), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
- The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. ([#116429](https://github.com/kubernetes/kubernetes/pull/116429), [@gjkim42](https://github.com/gjkim42)) [SIG API Machinery, Apps, Node, Scheduling and Testing]
### Feature
- A ValidatingAdmissionPolicy now has its `messageExpression` field checked against resolved types. ([#119209](https://github.com/kubernetes/kubernetes/pull/119209), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery]
- Add ConsistentListFromCache feature gate that allows apiserver to serve consistent lists from cache ([#118508](https://github.com/kubernetes/kubernetes/pull/118508), [@serathius](https://github.com/serathius)) [SIG API Machinery, Instrumentation and Testing]
- Add full cgroup v2 swap support for both Limited and Unlimited swap.
When LimitedSwap is enabled the swap limit would be automatically calculated for
Burstable QoS pods. For Best-Effort / Guaranteed QoS pods, swap would be disabled.
Containers with memory requests equal to their memory limits also won't have
swap access, and it is a way to opt-out of swap for a single container.
The formula for the swap limit for Burstable QoS pods is:
`(<memory-request>/<node-memory-capacity>)*<node-swap-capacity>`.
Support for cgroup v1 is removed. ([#118764](https://github.com/kubernetes/kubernetes/pull/118764), [@iholder101](https://github.com/iholder101)) [SIG Node and Testing]
- Add handling for pods in podgc for PodReplacementPolicy or PodDisruption ([#118772](https://github.com/kubernetes/kubernetes/pull/118772), [@kannon92](https://github.com/kannon92)) [SIG Apps and Testing]
- Add reason to metric `attachdetach_controller_forced_detaches` in the attach detach controller. ([#119185](https://github.com/kubernetes/kubernetes/pull/119185), [@xing-yang](https://github.com/xing-yang)) [SIG Apps and Storage]
- Add swap to stats to Summary API and Prometheus endpoints (stats/summary and /metrics/resource). ([#118865](https://github.com/kubernetes/kubernetes/pull/118865), [@iholder101](https://github.com/iholder101)) [SIG Node and Testing]
- Added a new command line argument `--interactive` to kubectl. The new command line argument lets a user confirm deletion requests per resource interactively. ([#114530](https://github.com/kubernetes/kubernetes/pull/114530), [@ardaguclu](https://github.com/ardaguclu)) [SIG CLI and Testing]
- Added a new feature gate, `SchedulerQueueingHints` (enabled by default).
The new feature gate activates a framework for fine-grained filtering of events related to scheduler plugins.
In this release, no default scheduling plugins make use of the hinting framework, so you should not expect any behavior changes. ([#119328](https://github.com/kubernetes/kubernetes/pull/119328), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling]
- Adds apiserver_admission_match_condition_evaluation_seconds and apiserver_admission_match_condition_exclusions_total metrics ([#119311](https://github.com/kubernetes/kubernetes/pull/119311), [@ivelichkovich](https://github.com/ivelichkovich)) [SIG API Machinery]
- Bump distroless-iptables to 0.2.6 based on Go 1.20.6 ([#119365](https://github.com/kubernetes/kubernetes/pull/119365), [@xmudrii](https://github.com/xmudrii)) [SIG Testing]
- CEL authorizer checks no longer raise runtime errors. Calls to "check" will always return a decision object and the authorization error (if any) can be accessed within expressions using the new decision methods "errored" and "error". ([#118804](https://github.com/kubernetes/kubernetes/pull/118804), [@benluddy](https://github.com/benluddy)) [SIG API Machinery]
- CRI: expose commit memory bytes in container stats specific to Windows ([#119238](https://github.com/kubernetes/kubernetes/pull/119238), [@kiashok](https://github.com/kiashok)) [SIG Node and Windows]
- Cloud controller manager's node controller now emits timing metrics for initial `Node` synchronization. These metrics measure the delay between the creation of a new `Node` and the node controller's initial management actions, such as removing the cloud provider taint. These metrics should be consulted when setting cloud controller manager's `--concurrent-node-syncs` flag. ([#119241](https://github.com/kubernetes/kubernetes/pull/119241), [@cartermckinnon](https://github.com/cartermckinnon)) [SIG Cloud Provider and Instrumentation]
- Faster scheduling when ResourceClaims are involved ([#119078](https://github.com/kubernetes/kubernetes/pull/119078), [@pohly](https://github.com/pohly)) [SIG Node and Scheduling]
- Graduate the `ProbeTerminationGracePeriod` feature gate to GA ([#114307](https://github.com/kubernetes/kubernetes/pull/114307), [@rphillips](https://github.com/rphillips)) [SIG Apps and Node]
- Hashing of KeyID in Logs
This release adds a feature to hash the `KeyID` values in the logs. The `KeyID` values are sensitive information that should not be exposed in plain text in the logs. By hashing the `KeyID` values, we can protect the confidentiality of the data while still being able to log the necessary information. ([#118988](https://github.com/kubernetes/kubernetes/pull/118988), [@nilekhc](https://github.com/nilekhc)) [SIG API Machinery, Auth and Testing]
- Implement alpha support for a drop-in kubelet configuration directory ([#119390](https://github.com/kubernetes/kubernetes/pull/119390), [@sohankunkerkar](https://github.com/sohankunkerkar)) [SIG Node]
- In the course of admitting a single request, the ValidatingAdmissionPolicy plugin will perform no more than one authorization check per unique authorizer expression. All evaluations of identical authorizer expressions will produce the same decision. ([#116443](https://github.com/kubernetes/kubernetes/pull/116443), [@benluddy](https://github.com/benluddy)) [SIG API Machinery and Testing]
- Kube-controller-manager: the dynamic resource controller steps in when a pod got created such that the scheduler ignores it (i.e. spec.nodeName is set) and then takes care of triggering delayed resource claim allocation and/or reserving a claim for the pod. ([#118209](https://github.com/kubernetes/kubernetes/pull/118209), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node and Testing]
- Kube-proxy service health returns http header "X-Load-Balancing-Endpoint-Weight" with number of local endpoints. The same information is still available in response body JSON payload.LocalEndpoints. ([#118999](https://github.com/kubernetes/kubernetes/pull/118999), [@cezarygerard](https://github.com/cezarygerard)) [SIG Network]
- Kubelet: plugins for dynamic resource allocation may use the v1alpha3 API instead of v1alpha2 if they want to do prepare/unprepare operations in batches. ([#119012](https://github.com/kubernetes/kubernetes/pull/119012), [@pohly](https://github.com/pohly)) [SIG Node and Testing]
- Kubelet: security of dynamic resource allocation was enhanced by limiting node access to those objects that are needed on the node. ([#116254](https://github.com/kubernetes/kubernetes/pull/116254), [@pohly](https://github.com/pohly)) [SIG Auth and Testing]
- Kubernetes is now built with Go 1.20.6 ([#119324](https://github.com/kubernetes/kubernetes/pull/119324), [@xmudrii](https://github.com/xmudrii)) [SIG API Machinery, Auth, Cloud Provider, Release and Testing]
- Migrate `pkg/controller/endpoint` to contextual logging ([#116755](https://github.com/kubernetes/kubernetes/pull/116755), [@my-git9](https://github.com/my-git9)) [SIG Apps, Instrumentation and Network]
- Migrated the `EndpointSlice` and `EndpointSliceMirroring` controllers (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#115295](https://github.com/kubernetes/kubernetes/pull/115295), [@Namanl2001](https://github.com/Namanl2001)) [SIG API Machinery, Apps, Network and Testing]
- Move non-graceful node shutdown to GA. ([#118228](https://github.com/kubernetes/kubernetes/pull/118228), [@carlory](https://github.com/carlory)) [SIG Apps, Storage and Testing]
- New CEL Library functions to support Kubernetes Quantities. ([#118803](https://github.com/kubernetes/kubernetes/pull/118803), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery]
- New Metrics Added for Encryption Configuration Controller
This release adds new metrics to the Encryption Configuration Controller to help monitor the automatic reloading of encryption configuration. The new metrics include:
- `apiserver_encryption_config_controller_automatic_reload_failures_total`: Total number of failed automatic reloads of encryption configuration.
- `apiserver_encryption_config_controller_automatic_reload_success_total`: Total number of successful automatic reloads of encryption configuration.
- `apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds`: Timestamp of the last successful or failed automatic reload of encryption configuration.
These metrics can be used to monitor the health of the Encryption Configuration Controller and to troubleshoot any issues that may arise during automatic reloading of encryption configuration. ([#119008](https://github.com/kubernetes/kubernetes/pull/119008), [@nilekhc](https://github.com/nilekhc)) [SIG API Machinery, Auth and Instrumentation]
- New staging repo has been created for the EndpointSlice reconciler. ([#118953](https://github.com/kubernetes/kubernetes/pull/118953), [@mskrocki](https://github.com/mskrocki)) [SIG Apps, Network and Release]
- Promote the following apiserver flowcontrol metrics to Beta:
apiserver_flowcontrol_request_wait_duration_seconds
apiserver_flowcontrol_current_executing_seats
apiserver_flowcontrol_nominal_limit_seats
apiserver_flowcontrol_rejected_requests_total
apiserver_flowcontrol_dispatched_requests_total
apiserver_flowcontrol_current_inqueue_requests
apiserver_flowcontrol_current_executing_requests ([#119110](https://github.com/kubernetes/kubernetes/pull/119110), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery and Instrumentation]
- Replace `apiserver_storage_db_total_size_in_bytes` with `apiserver_storage_size_bytes` metric ([#118812](https://github.com/kubernetes/kubernetes/pull/118812), [@serathius](https://github.com/serathius)) [SIG API Machinery, Instrumentation and Testing]
- The apiserver debug endpoint `/debug/api_priority_and_fairness/dump_requests` has been extended to dump executing requests as well as queued ones. A column for StartTime has been added to the returned table, with the queued requests having a StartTime of "0001-01-01T00:00:00Z". The executing requests have a RequestIndexInQueue of -1, and the QueueIndex is also -1 for priority levels without queues. ([#119009](https://github.com/kubernetes/kubernetes/pull/119009), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery]
- The scheduler skips the PodTopologySpread Score plugin when nothing to do with the Pod.
It will affect some metrics values related to the PodTopologySpread Score plugin. ([#118608](https://github.com/kubernetes/kubernetes/pull/118608), [@utam0k](https://github.com/utam0k)) [SIG Scheduling]
- TopologyManagerPolicyOptions feature-flag is promoted to beta and enabled by default. ([#118816](https://github.com/kubernetes/kubernetes/pull/118816), [@PiotrProkop](https://github.com/PiotrProkop)) [SIG Node]
- Update kube-apiserver's priority & fairness work estimator such that 'max seats' is MIN(0.15 x nominalCL, nomincalCL / handSize) ([#118601](https://github.com/kubernetes/kubernetes/pull/118601), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery]
- ValidatingAdmissionPolicy type checking now correctly handles `authorizer` variable. ([#118540](https://github.com/kubernetes/kubernetes/pull/118540), [@jiahuif](https://github.com/jiahuif)) [SIG API Machinery]
- With the KubeletCgroupDriverFromCRI feature gate enabled and sufficiently new version of a container
runtime, kubelet automatically detects the cgroup driver config from the container runtime, eliminating
the need to specify the `cgroupDriver` configuration option (or --cgroup-driver` flag) of kubelet. ([#118770](https://github.com/kubernetes/kubernetes/pull/118770), [@marquiz](https://github.com/marquiz)) [SIG Node]
- [Kube-proxy]: implement connection draining for terminating nodes, KEP-3836 ([#116470](https://github.com/kubernetes/kubernetes/pull/116470), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Network]
- `force_delete_pods_total ` and `force_delete_pod_errors_total ` metrics count all pod deletion behaviors. ([#118480](https://github.com/kubernetes/kubernetes/pull/118480), [@carlory](https://github.com/carlory)) [SIG Apps]
### Failing Test
- Switched back to debian-base instead of distroless for conformance image. ([#119422](https://github.com/kubernetes/kubernetes/pull/119422), [@saschagrunert](https://github.com/saschagrunert)) [SIG Architecture, Release and Testing]
### Bug or Regression
- Add warning for dup ports update/patching in pod's container ports and service ports ([#113245](https://github.com/kubernetes/kubernetes/pull/113245), [@pacoxu](https://github.com/pacoxu)) [SIG Network]
- Bump cadvisor version to v0.47.3 ([#119225](https://github.com/kubernetes/kubernetes/pull/119225), [@iholder101](https://github.com/iholder101)) [SIG Node and Testing]
- Dynamic Resource Allocation: log a error and submit an event when Kubelet fails to prepare dynamic resources ([#118578](https://github.com/kubernetes/kubernetes/pull/118578), [@bart0sh](https://github.com/bart0sh)) [SIG Node]
- Fix computing backoff delay when using Job pod failure policy, by including in the backoff delay calculation pod failures ignored from the backoffLimit counter ([#119434](https://github.com/kubernetes/kubernetes/pull/119434), [@mimowo](https://github.com/mimowo)) [SIG Apps]
- Fix discoverability of apiregistration.k8s.io in openapi/v3 ([#118879](https://github.com/kubernetes/kubernetes/pull/118879), [@atiratree](https://github.com/atiratree)) [SIG API Machinery]
- Fixed a bug where `kubectl port-forward`, when used with a Deployment, could connect to a terminating pod even when a running pod is also available. ([#119256](https://github.com/kubernetes/kubernetes/pull/119256), [@brianpursley](https://github.com/brianpursley)) [SIG CLI]
- Fixed kubelet startup getting stuck with `NewVolumeManagerReconstruction` feature enabled and a CSI volume present in /var/lib/kubelet/pods. ([#117804](https://github.com/kubernetes/kubernetes/pull/117804), [@jsafrane](https://github.com/jsafrane)) [SIG Node and Storage]
- Kubeadm: the limitation that the 'ignorePreflightErrors' field can not be set to 'all' in kubeadm config file has been removed ([#119351](https://github.com/kubernetes/kubernetes/pull/119351), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle]
- Only declare Job as finished after removing all Pod finalizers to avoid orphan Pods ([#119159](https://github.com/kubernetes/kubernetes/pull/119159), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps and Testing]
- Reduces CPU and memory consumption of kube-apiserver if OpenAPI V2 is not accessed by any client. Also improves performance of the apiserver on installation of many CRDs. ([#118212](https://github.com/kubernetes/kubernetes/pull/118212), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
- The kube-proxy `sync_proxy_rules_iptables_total` metric has now reverted back
to its pre-1.27 behavior of tracking the total number of iptables rules that
kube-proxy is responsible for, rather than only counting the number of rules
that it re-synced on the last sync. The new `sync_proxy_rules_iptables_last`
metric now gives the latter number. ([#119140](https://github.com/kubernetes/kubernetes/pull/119140), [@danwinship](https://github.com/danwinship)) [SIG Network]
- The metric `apiserver_flowcontrol_request_concurrency_limit` has been deprecated and will be removed in a future release. It is a duplicate of `apiserver_flowcontrol_nominal_limit_seats` (introduced in release 1.26) but has an outdated name and had an outdated HELP string. ([#118959](https://github.com/kubernetes/kubernetes/pull/118959), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery]
- [Dual-stack] Fix generateAPIPodStatus() of kubelet handling Secondary IP. hostIPs order may not be be consistent. If secondary IP is before primary one, current logic adds primary IP twice into PodIPs, which leads to error: "may specify no more than one IP for each IP family". ([#116879](https://github.com/kubernetes/kubernetes/pull/116879), [@lzhecheng](https://github.com/lzhecheng)) [SIG Node]
### Other (Cleanup or Flake)
- Migrated the disruption controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#119147](https://github.com/kubernetes/kubernetes/pull/119147), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG API Machinery, Apps, Instrumentation and Testing]
- Migrated the podgc controller and some other remaining log calls within `kube-controller-manager` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). `kube-controller-manager` is now converted completely. ([#119250](https://github.com/kubernetes/kubernetes/pull/119250), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Cloud Provider, Instrumentation, Network, Storage and Testing]
- Remove KUBECTL_EXPLAIN_OPENAPIV3 which is already redundant ([#119286](https://github.com/kubernetes/kubernetes/pull/119286), [@ardaguclu](https://github.com/ardaguclu)) [SIG CLI]
- Revised OpenAPI v2 fetching for CustomResourceDefinitions. CRDs are now aggregated lazily,
which improves resource usage during installation of many CRDs. As a result, the first request
to fetch the OpenAPI may be slower. ([#118808](https://github.com/kubernetes/kubernetes/pull/118808), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery and Testing]
- Shrink the OpenAPI v2 spec by more than 50%, especially for less CPU resource consumption. ([#118204](https://github.com/kubernetes/kubernetes/pull/118204), [@sttts](https://github.com/sttts)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
- The GetAllocatableResources podresources API endpoint is now GA ([#118973](https://github.com/kubernetes/kubernetes/pull/118973), [@ffromani](https://github.com/ffromani)) [SIG Node and Testing]
- Updated debian-base image to bookworm-v1.0.0. ([#119095](https://github.com/kubernetes/kubernetes/pull/119095), [@saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Architecture, Release and Testing]
- Updated setcap image to debian bookworm v1.0.0. ([#119247](https://github.com/kubernetes/kubernetes/pull/119247), [@saschagrunert](https://github.com/saschagrunert)) [SIG Release]
## Dependencies
### Added
- github.com/xhit/go-str2duration/v2: [v2.1.0](https://github.com/xhit/go-str2duration/v2/tree/v2.1.0)
### Changed
- github.com/alecthomas/kingpin/v2: [v2.3.1 → v2.3.2](https://github.com/alecthomas/kingpin/v2/compare/v2.3.1...v2.3.2)
- github.com/google/cadvisor: [v0.47.2 → v0.47.3](https://github.com/google/cadvisor/compare/v0.47.2...v0.47.3)
- github.com/prometheus/client_model: [v0.3.0 → v0.4.0](https://github.com/prometheus/client_model/compare/v0.3.0...v0.4.0)
- github.com/prometheus/common: [v0.42.0 → v0.44.0](https://github.com/prometheus/common/compare/v0.42.0...v0.44.0)
- github.com/rogpeppe/go-internal: [v1.6.1 → v1.10.0](https://github.com/rogpeppe/go-internal/compare/v1.6.1...v1.10.0)
- golang.org/x/crypto: v0.6.0 → v0.11.0
- golang.org/x/net: v0.9.0 → v0.12.0
- golang.org/x/oauth2: v0.6.0 → v0.8.0
- golang.org/x/sys: v0.8.0 → v0.10.0
- golang.org/x/term: v0.7.0 → v0.10.0
- golang.org/x/text: v0.9.0 → v0.11.0
- k8s.io/kube-openapi: 7562a10 → 2695361
### Removed
- github.com/xhit/go-str2duration: [v1.2.0](https://github.com/xhit/go-str2duration/tree/v1.2.0)
# v1.28.0-alpha.4
## Downloads for v1.28.0-alpha.4
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes.tar.gz) | 05a404e2a5a526cb4713a9cab1cfcadb03cbeb065663a8ccec9c7eaf60277e1c69bea422716fc3b805ca569effb036b2d88adc752409b4f6103f10111f620736
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-src.tar.gz) | 0707c72499098c2eb8ca3cffd1baf0cb4da553ded8acbf6bc1b725461484a75ba5baf277ccbe318cdb5df0c970cb31bf8afe3df0130acdd23c35b8a2fbc8a15f
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-darwin-amd64.tar.gz) | 5fc07afae149003d53d8bc72f9d3bbf578efd7ec7c36fda46a436774f4471cd457317dada967fc3596d369783219bdca1974d62f47c09fb8b2d158a78d48aebe
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-darwin-arm64.tar.gz) | a025567c86d8af69e34d589f36f079eeff85105e342047f5d74b1ba749b9b857d19ffadd280910fd58926dfae54eb7eb8203009ac96362877a05e3cb88c49e4b
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-linux-386.tar.gz) | b17a16d8f3ce78e92b2988e726a9c818f0f0f36b8ae22809f4db2568c1746b585888820cb3cb276d00b76781e75bb10d1e9a19887d438ffee58c435775f114f0
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-linux-amd64.tar.gz) | 8133aaf1c3a964d32666fa7616917235ca62ed57b879915cae3acc4867db46e35b127cc302d1a3fa7fff143ade6f73c0c1667b45eb1debef052b2b69f5c407f1
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-linux-arm.tar.gz) | 84f30f2d113f9003207e547dcb4c3467e17b93b05dade5c6b8cd847bd040e7b21574c1d75d923d8f1d3906a4793ab8a78ab477cb16ccd72a98221c0edd394ce9
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-linux-arm64.tar.gz) | e7cad9b40187afa63168e40ec96128d4f2469115049cd0642e3a255d25b325c662fd99c1866dd6798a634d2de179493be9c05de11372f86f6d31329b24b8c283
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-linux-ppc64le.tar.gz) | 3597df23599a6f9da7886601c0ff7e49fc23b0817b6463aa426e7191f23955f772986494d9b8926b9b9dcb1f7f2f75054739b3d25a893f4d65e3f58d567eeb2b
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-linux-s390x.tar.gz) | a5630f732ef831722c778484742368a3768276bd4e443bcedcbd2c02b1164265e7a70fb55b6e7560558b0a3d4eaca3cbc7c7ded436c19024e6826224d73b4ef6
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-windows-386.tar.gz) | 0be37194c9d1fc75877346eb1ab1f612286068732558d59f862c1901c3217b91a7e758f41aaba2308142aab4170bb4e5f4e7291fde7717062a2fb4ca91b159bf
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-windows-amd64.tar.gz) | f53a4216ac8e959b40d10257bc01044b1c1e430c5da02b61b6ed5184f0acb7317d19f36946e60de0d168d45d71d404bd615a2beb9b70b6495b2db0182fde375c
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-client-windows-arm64.tar.gz) | 0bce42d19642c6dd794ce7944274b0dd997df5984fa8a22f72f07551f00533f8f018bd209994c69615ca33d4f8a41a873b364e5c3d996b2edccf849f94621236
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-server-linux-amd64.tar.gz) | 593409f6e3accafbd448da6d6b775cc1f85cc4b787acb7b7580e94221a528c6b805e73d5fe16fc36a9c2838da6bad3928b18c3771ec95c5bbd0efd19d404d8e2
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-server-linux-arm64.tar.gz) | 3e8f26b51f85b61ad249f54255077a8f4ffaf80c55935cfa2f490f56eea112eb2df569882a7f486cd19371e41c1f65c43aa2bffeece3e35269b67c19ff9e7ae4
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-server-linux-ppc64le.tar.gz) | 3e4d8f4ebed1632878a78936e62331973fa57e8b394c79c262f98316a81f460b6bd8ee4cf9dd74d77df289ea2cf3716e58a431a5f52c610c916a7f45cee80bdd
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-server-linux-s390x.tar.gz) | 6aa22b0fa568b70dd9c34408dfb70b60d09b2fd65429671e23d0becfc83aac75d082818c21737105e4e2485752fd9e5d5f1d92e8dbbc46b257d269237155a85a
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-node-linux-amd64.tar.gz) | 7f0a902031857e8ec8189dd37c52e788c36e02c5e19439ec24602822e95466b008064419961f728f3c02b661dfc23d89fcc0bca15b085f9ed3d001b4cd94adb8
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-node-linux-arm64.tar.gz) | fa564b5fe5d69f2c31b453da9024ab505adfb62373722fe3d6d3852eb13e938d271f5a90542982ea9ea4fd2182f67a720be0b8c77f2e7353a6bbda3ff16e34a5
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-node-linux-ppc64le.tar.gz) | 14ea08bf2ad7f177de8f59b4ac44bd7ba451a9b6493c1b6a3fbc00337e2c7865824b790e30d3a4dbb6cbce9ecbfa62a4e12fb7da04049e2bdf718d273131fff2
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-node-linux-s390x.tar.gz) | a1119b19f271a5ddb90b71f4b3e844b3d46889340349fcbf297c2ffaae253538e303535f7b180a1107d189a7fc66fdf3f029da90db761f1ca5faf52aedfd0c64
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.4/kubernetes-node-windows-amd64.tar.gz) | a3b03111ff946bb21722d1f140cfa483b9692381aaadc3cf7726d2e53b717b573887f3a541ab42cb3244a95f023817d2fefd5c4314d1b5fe30ecd68643709295
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.0-alpha.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.0-alpha.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.0-alpha.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.0-alpha.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.0-alpha.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.0-alpha.4](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.28.0-alpha.3
## Urgent Upgrade Notes
### (No, really, you MUST read this before you upgrade)
- Action required for the custom scheduler plugin developers.
Here's the breaking change in `EnqueueExtension` in the scheduling framework.
The `EventsToRegister` in `EnqueueExtension` changed the return value from `ClusterEvent` to `ClusterEventWithHint`. `ClusterEventWithHint` allows each plugin to filter out more useless events via the callback function named `QueueingHintFn`.
When the scheduling queue receives a cluster event, before moving each Pod from unschedulable pod pool to activeQ/backoffQ, it will call QueueingHintFn of plugins that rejected each Pod in the previous scheduling cycle.
Depending on the value returned from QueueingHintFn, the scheduling queue changes how it queues each Pod:
- if more than one QueueingHintFn returns QueueImmediately, it queues Pod to activeQ.
- If no QueueingHintFn returns QueueImmediately and more than one plugin returns QueueAfterBackoff, it queues Pod to backoffQ if Pod is backing off, or to activeQ if Pod's backoff has already finished.
- If all QueueingHintFn return QueueSkip, it puts this pod back to the unschedulable pod pool
Having appropriate QueueingHintFn contributes to reducing useless retries and thus improves the overall scheduler's performance.
**How can I migrate?**
For backward compatibility, nil `QueueingHintFn` is treated as always returning QueueAfterBackoff.
So, if you want to just keep the existing behavior, you can register `ClusterEventWithHint` with no `QueueingHintFn` in it.
But, registering appropriate `QueueingHintFn` is, of course, better from a scheduling performance perspective. ([#118551](https://github.com/kubernetes/kubernetes/pull/118551), [@sanposhiho](https://github.com/sanposhiho)) [SIG Node, Scheduling, Storage and Testing]
- RBD volume plugin ( `kubernetes.io/rbd`) has been deprecated in this release and will be removed in a subsequent release. Alternative is to use RBD CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. ([#118552](https://github.com/kubernetes/kubernetes/pull/118552), [@humblec](https://github.com/humblec)) [SIG Storage]
## Changes by Kind
### Deprecation
- KMSv1 is deprecated and will only receive security updates going forward. Use KMSv2 instead. Set --feature-gates=KMSv1=true to use the deprecated KMSv1 feature. ([#119007](https://github.com/kubernetes/kubernetes/pull/119007), [@aramase](https://github.com/aramase)) [SIG API Machinery and Auth]
### API Change
- Add ServedVersions field to StorageVersion API ([#118386](https://github.com/kubernetes/kubernetes/pull/118386), [@Richabanker](https://github.com/Richabanker)) [SIG API Machinery and Testing]
- Component-base/logs is now more strict about not applying configurations multiple times and will return an error when that is attempted. Can be overridden by binaries which need to do that. ([#117108](https://github.com/kubernetes/kubernetes/pull/117108), [@pohly](https://github.com/pohly)) [SIG API Machinery, Architecture, Cloud Provider, Instrumentation, Scheduling and Testing]
### Feature
- "plugin_evaluation_total" metric supports prescore/score extension point.
The metric doesn't get incremented when the prescore/score plugin has nothing to do with an incoming Pod. ([#118025](https://github.com/kubernetes/kubernetes/pull/118025), [@AxeZhan](https://github.com/AxeZhan)) [SIG Scheduling]
- Add `--concurrency` flag to configure the concurrency of `kubectl diff` execution, defaults to 1 ([#118810](https://github.com/kubernetes/kubernetes/pull/118810), [@brancz](https://github.com/brancz)) [SIG CLI]
- AdvancedAuditing feature gate that graduated to GA in 1.12 and was unconditionally enabled has been removed in v1.28. ([#118763](https://github.com/kubernetes/kubernetes/pull/118763), [@Shubham82](https://github.com/Shubham82)) [SIG API Machinery and Auth]
- Allow to monitor client-go DNS resolver latencies via `rest_client_dns_resolution_duration_seconds` Prometheus metric ([#115357](https://github.com/kubernetes/kubernetes/pull/115357), [@mfojtik](https://github.com/mfojtik)) [SIG API Machinery, Architecture and Instrumentation]
- Dynamic resource allocation: when a claim uses "wait for first consumer" allocation (the default), then it will now get deallocated after it was used by a pod. That ensures that the next pod isn't affected by previous scheduling decision and that resources are not kept allocated unless really needed. If keeping a claim allocated is desired, use "immediate allocation". ([#118936](https://github.com/kubernetes/kubernetes/pull/118936), [@pohly](https://github.com/pohly)) [SIG Apps, Node and Testing]
- Kubeadm: add the --allow-experimental-api flag to "kubeadm config migrate/validate" commands. It can be used to migrate or validate WIP / experimental APIs in the future. ([#118866](https://github.com/kubernetes/kubernetes/pull/118866), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle]
- Kubeadm: generate CA certificates with a start time that is offset 5 minutes in the past relative to the current system time to workaround cases of clock desync.
client-go: allow to set NotBefore in NewSelfSignedCACert() ([#118922](https://github.com/kubernetes/kubernetes/pull/118922), [@champtar](https://github.com/champtar)) [SIG API Machinery, Auth and Cluster Lifecycle]
- Migrated controller functions to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116930](https://github.com/kubernetes/kubernetes/pull/116930), [@fatsheep9146](https://github.com/fatsheep9146)) [SIG API Machinery, Apps, Network, Node, Storage and Testing]
- Migrated the certificate controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#113994](https://github.com/kubernetes/kubernetes/pull/113994), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG API Machinery, Apps, Auth, Instrumentation and Testing]
- Now it is possible to use pods with volumes and user namespaces. The feature gate was renamed from UserNamespacesStatelessPodsSupport to UserNamespacesSupport ([#118691](https://github.com/kubernetes/kubernetes/pull/118691), [@giuseppe](https://github.com/giuseppe)) [SIG Apps, Node and Testing]
- RetroactiveDefaultStorageClass feature is stable and enabled by default. ([#118102](https://github.com/kubernetes/kubernetes/pull/118102), [@RomanBednar](https://github.com/RomanBednar)) [SIG Apps, Storage and Testing]
- Schedular now waits for handlers to finish syncing before the scheduling cycles start. ([#116729](https://github.com/kubernetes/kubernetes/pull/116729), [@AxeZhan](https://github.com/AxeZhan)) [SIG Apps, Scheduling and Testing]
- The "value" part in the `wait --for=jsonpath='{expression}'[=value]` is now optional. If the value is not provided i.e. the command looks like `wait --for=jsonpath='{expression}'` then the wait condition is interpreted as matched when the expression returns *any* single JSON value like object or a literal. ([#118160](https://github.com/kubernetes/kubernetes/pull/118160), [@minherz](https://github.com/minherz)) [SIG CLI and Testing]
- Updated cAdvisor to v0.47.2 - Fix metrics in cri-o when a container restarts ([#118774](https://github.com/kubernetes/kubernetes/pull/118774), [@harche](https://github.com/harche)) [SIG Node]
- When a pod is done or not going to run, then ResourceClaims for it can be reused by other pods or deleted. ([#118817](https://github.com/kubernetes/kubernetes/pull/118817), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node and Testing]
### Bug or Regression
- Added a new event `FailedToRetrieveImagePullSecret` which will be generated when a pod references an `ImagePullSecret` that doesn't exist. ([#117927](https://github.com/kubernetes/kubernetes/pull/117927), [@kaisoz](https://github.com/kaisoz)) [SIG Node]
- As in Kubernetes 1.26 and 1.27, resource claims do not get prepared by kubelet when no container uses them. This was changed accidentally in [v1.28.0-alpha.1](https://github.com/kubernetes/kubernetes/releases/tag/v1.28.0-alpha.1). ([#118786](https://github.com/kubernetes/kubernetes/pull/118786), [@pohly](https://github.com/pohly)) [SIG Node and Testing]
- Faster StatefulSet creation when `Parallel` mode is enabled. ([#117865](https://github.com/kubernetes/kubernetes/pull/117865), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) [SIG Apps]
- Fix cronjob controller handling of complex schedules, like "30 6-16/4 * * 1-5", for example ([#118724](https://github.com/kubernetes/kubernetes/pull/118724), [@soltysh](https://github.com/soltysh)) [SIG Apps]
- Fix deletion of non-admissible pods that are deleted during Kubelet restart ([#118497](https://github.com/kubernetes/kubernetes/pull/118497), [@mimowo](https://github.com/mimowo)) [SIG Node and Testing]
- Fix discoverability of apiregistration.k8s.io in openapi/v3 ([#118879](https://github.com/kubernetes/kubernetes/pull/118879), [@atiratree](https://github.com/atiratree)) [SIG API Machinery]
- Kubectl explain should correctly work for all resources ([#118876](https://github.com/kubernetes/kubernetes/pull/118876), [@atiratree](https://github.com/atiratree)) [SIG CLI]
- Kubectl expose supports the creation of different protocol service on the same port ([#114909](https://github.com/kubernetes/kubernetes/pull/114909), [@aimuz](https://github.com/aimuz)) [SIG CLI]
- The Daemonset controller creates replacements for terminal Pods, which can appear during VM preemptions or when using Pod finalizers ([#118716](https://github.com/kubernetes/kubernetes/pull/118716), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps, Node and Testing]
- The `pod_scheduling_duration_seconds` metrics won't consider the time when a Pod fails PreEnqueue (like being gated). ([#118049](https://github.com/kubernetes/kubernetes/pull/118049), [@helayoty](https://github.com/helayoty)) [SIG Scheduling]
- Update apiserver metric request_filter_duration_seconds to include a 10s, 15s and 30s bucket.
- Update apiserver metric request_wait_duration_seconds to include a 15s bucket. ([#118945](https://github.com/kubernetes/kubernetes/pull/118945), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery, Instrumentation and Testing]
- Users will no longer see an error for failed events caused due to terminating namespace. ([#114849](https://github.com/kubernetes/kubernetes/pull/114849), [@padlar](https://github.com/padlar)) [SIG API Machinery]
- Wait.PollUntilContextTimeout function, if immediate is true, the condition will be invoked before waiting and guarantees that the condition is invoked at least once, regardless of whether the context has been cancelled. ([#118686](https://github.com/kubernetes/kubernetes/pull/118686), [@aojea](https://github.com/aojea)) [SIG API Machinery]
### Other (Cleanup or Flake)
- Kube-controller-manager and cloud-controller-manager have changed the name of controllers that can be turned off/on that are passed to the `--controllers` flag (eg `pod-garbage-collector-controller` ). The old names (eg `podgc`) are also accepted and aliased to the new names ([#115813](https://github.com/kubernetes/kubernetes/pull/115813), [@atiratree](https://github.com/atiratree)) [SIG API Machinery and Cloud Provider]
- Kube-proxy will now warn at startup if the configuration seems inconsistent
with respect to IP families. (For example, if you have an IPv4 node IP, but
`--cluster-cidr` is IPv6.) ([#119003](https://github.com/kubernetes/kubernetes/pull/119003), [@danwinship](https://github.com/danwinship)) [SIG Network]
- Promote `kubernetes_healthcheck` and `kubernetes_healthchecks_total` to `BETA` stability level. ([#118986](https://github.com/kubernetes/kubernetes/pull/118986), [@logicalhan](https://github.com/logicalhan)) [SIG Architecture, Instrumentation and Testing]
- Reduce delay when processing jobs after a transient API error ([#118759](https://github.com/kubernetes/kubernetes/pull/118759), [@mimowo](https://github.com/mimowo)) [SIG Apps]
- The NetworkPolicyLegacy test suite (deprecated in v1.21) has now officially been removed in favor of the new table driven e2e tests. ([#118915](https://github.com/kubernetes/kubernetes/pull/118915), [@astoycos](https://github.com/astoycos)) [SIG Network and Testing]
- The feature gates `CSIMigrationGCE` is graduated to GA and were unconditionally enabled have been removed in v1.25, and the entire gcepd package has been removed. ([#117055](https://github.com/kubernetes/kubernetes/pull/117055), [@cyclinder](https://github.com/cyclinder)) [SIG API Machinery, Node, Scheduling and Storage]
- The metric `apiserver_flowcontrol_current_executing_seats` has been introduced as a duplicate of `apiserver_flowcontrol_request_concurrency_in_use` because the latter has a confusing name and will be removed in a later release. ([#118960](https://github.com/kubernetes/kubernetes/pull/118960), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery]
- Use table-driven test for TestPerPodSchedulingMetrics ([#118842](https://github.com/kubernetes/kubernetes/pull/118842), [@helayoty](https://github.com/helayoty)) [SIG Scheduling]
## Dependencies
### Added
_Nothing has changed._
### Changed
- github.com/google/cadvisor: [v0.47.1 → v0.47.2](https://github.com/google/cadvisor/compare/v0.47.1...v0.47.2)
### Removed
_Nothing has changed._
# v1.28.0-alpha.3
## Downloads for v1.28.0-alpha.3
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes.tar.gz) | 1479e19873837e09f08f4c14d4b7587b1bc40d0b0d3214637311b63068301d34a63663f5d13b8ec62c81095a30eef1e8589633c630fe613eb825eb4afa0ddeb9
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-src.tar.gz) | 1f5d3486f15574220d3c5a5d7c7a2b7412347418650deeda326eb513ba2bb43c197e089999756eee09ac4b55dcafc18eef96c6e15e20fcc9b91183a35f224cc1
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-darwin-amd64.tar.gz) | 4e5139d5a5a98343ded5b506451c3b9b052803a2e2ebb4e2328e17edbbde56dde749407ef8fc816283a1a1b1f80939a76b7c64a09a9496d4448ec47fe34cd95b
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-darwin-arm64.tar.gz) | c58fdf32d3f1b411e10a7ede56115020c7a4c50912a899f06cf94d2f06a5e24a21df1deda6eb0e87f70d88afec46186f64dd18bb1b26b94b24b01059aef88b2c
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-linux-386.tar.gz) | 26d3d2663de49063d02c682557dce616f5630b69a1ca219024ccd3405081193732bee42d24d9e5f5cf9fcd214da50defd64c141775a6aea372dbe3e9793e4547
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-linux-amd64.tar.gz) | f9c29ea075c4b842c39587e911b83e2c798d6d69ed5f4601ce17ce3c33bb9db8f511b3d2149dfe3d86576763e92e0b6eb6d3aa1b12bc04868fb6c8a013a2fdd1
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-linux-arm.tar.gz) | be984144ebe8c2f3156b3d13aaebd362543399f4c91f8f18b1936999327468ccb5af7d95723280000b627e2f81fb119d6e175cc305391ffd83b3b632eb3c30df
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-linux-arm64.tar.gz) | bfc44ed472351dee434984e49be2cc9959df2f98048beeb1ad85a50bfe833a5396bb3f5c7f81ced2048de269bd60f6d4cf55944c0d44d288fb858b5552354389
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-linux-ppc64le.tar.gz) | d7ee2633c172442a04d317d4d935b55aa892131f716308b78d0f7b3cb5d2bfa9069f7b55ec837e4ff9a260717313e7fc4c7ca4a931f9f2b36768a9c54593ba45
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-linux-s390x.tar.gz) | 4621ed7a77990430cbb03a14f85780f059a408a567c26c14eb7c7127300f4e1ac1b2013206f00a92e35da9f1142909745262ad43a8026a9ebe31cf423dbb3b90
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-windows-386.tar.gz) | 6526de9f4ccd6f036152d73ebbf3a2b626ca7291c85d5b46977e8a8643ff80616665f4bd8b009310cc887f7d5e1827045a0b2489dd19621338e727e18cca4097
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-windows-amd64.tar.gz) | 784e8d07b1db0331f454faac34b279bffba16c11f98ba25aa54f64eb6a0d9d55f0e8221828aa4e6e4d0d6bfb66a3c22a0d8a66d6287b37a33890a960d25fc54e
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-client-windows-arm64.tar.gz) | 523d2809d4ff4616ffd94e0868ac15ee10f87bee65760741b09ba2a04626e3ce226a5fc44a1bddb32835403ee374a6c12afa74ec0d2b9441304a11d923f43cee
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-server-linux-amd64.tar.gz) | 5c2716137752b514fe7a8b23291cc7a661984deaf642d111e65690e928ecfbb6b5168b1f043847f3d4689cafbbdf5bb6fb5c97d2fd6f5f83c5b6cc701a4f388f
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-server-linux-arm64.tar.gz) | 814e4ec6b46960e90ccce6ef1c234666f372b2b41583ead530b4163b829c4bb9001df822daf55493e54ad749076ab5391ebde7261d73f4e524f6cace402d49c3
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-server-linux-ppc64le.tar.gz) | 8a7749282ba4d3df877097dc07843112e815879ee911379d02ed33d5aab59cb60ffd27127ebd94879bef45d9534470d6aa7e48d71ecb455e3055c2dbb169fcdb
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-server-linux-s390x.tar.gz) | 031a55294a06e5773d3c277dcab6b1e32c5e6409448015282275cb62e5bafb6a10845f0de3d37d501067f3f4176f48d04cafdd4eb96f6f7e30c7274ce3adaec6
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-node-linux-amd64.tar.gz) | 488d426c09f92a6f30d77352af7154b9fda8758a946566b31d855a4c30f82ccdb071d40708f96e78f9bf5ca961be27fa67ecc833101818e43da19bc967c9eb28
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-node-linux-arm64.tar.gz) | e5f6914bd6e393915ea814401c09f4b0accc3454087091e60b01ea7c87aa64edd1a0af954cd1a90a3dc83c5559772d1ba5a8ff04134638e911d972947e3e94a0
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-node-linux-ppc64le.tar.gz) | 459453ad83c9cdd2568e43ed988ec2635b5413b435e587e8a6394feb7d49c550622c491f337c48f90ad32cc02b7ef47e24069f09737817a1c37f1a8930d6c5c2
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-node-linux-s390x.tar.gz) | e5a6ff5bb58275720c18b6181ffb9d135dcfaf05a229f87787538775e0280543db43e7218168f0a2de8a74acca6899dccd1cce88780b086ce6a96fb3e0368870
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.3/kubernetes-node-windows-amd64.tar.gz) | 7cc51923e34cfeb00681e7c2d26dff5c442b39d0000f11ad30ec5c843d80847a0353b3b4e6e5a228ad5cae7f33dfc0d4c38bf4b843b98b8115d28c3dc683dd9e
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.0-alpha.3](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.28.0-alpha.2
## Changes by Kind
### Deprecation
- Kube-controller-manager deprecate `--volume-host-cidr-denylist` and `--volume-host-allow-local-loopback` flags. ([#118128](https://github.com/kubernetes/kubernetes/pull/118128), [@carlory](https://github.com/carlory)) [SIG API Machinery, Apps, Network, Node, Storage and Testing]
- Kubelet: The `--azure-container-registry-config` flag has been deprecated and will be removed in a future release, please use `--image-credential-provider-config` and `--image-credential-provider-bin-dir` to setup acr credential provider instead. ([#118596](https://github.com/kubernetes/kubernetes/pull/118596), [@SataQiu](https://github.com/SataQiu)) [SIG Node]
### API Change
- ACTION_REQUIRED
When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. ([#118420](https://github.com/kubernetes/kubernetes/pull/118420), [@alculquicondor](https://github.com/alculquicondor)) [SIG Apps]
- Expose rest.DefaultServerUrlFor function ([#118055](https://github.com/kubernetes/kubernetes/pull/118055), [@timofurrer](https://github.com/timofurrer)) [SIG API Machinery]
- If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via `memory.oom.group` . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. ([#117793](https://github.com/kubernetes/kubernetes/pull/117793), [@tzneal](https://github.com/tzneal)) [SIG Apps, Node and Testing]
- Update the comment about the feature-gate level for PodFailurePolicy from alpha to beta ([#118278](https://github.com/kubernetes/kubernetes/pull/118278), [@mimowo](https://github.com/mimowo)) [SIG Apps]
### Feature
- Add '--concurrent-cron-job-syncs' flag for kube-controller-manager to set the number of workers for cron job controller ([#117550](https://github.com/kubernetes/kubernetes/pull/117550), [@borgerli](https://github.com/borgerli)) [SIG Apps]
- Client-go: make generated CA certificates valid 1 hour in the past (NewSelfSignedCACert). Applies to CA certificates and other certificates generated by kubeadm. ([#118631](https://github.com/kubernetes/kubernetes/pull/118631), [@champtar](https://github.com/champtar)) [SIG Auth]
- Fixes the alpha `CloudDualStackNodeIPs` feature. ([#118329](https://github.com/kubernetes/kubernetes/pull/118329), [@danwinship](https://github.com/danwinship)) [SIG Network and Node]
- Kubelet: un-deprecate --provider-id flag ([#116530](https://github.com/kubernetes/kubernetes/pull/116530), [@pacoxu](https://github.com/pacoxu)) [SIG Node]
- Migrated the Job controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116910](https://github.com/kubernetes/kubernetes/pull/116910), [@fatsheep9146](https://github.com/fatsheep9146)) [SIG API Machinery, Apps and Testing]
- Rename PodHasNetwork to PodReadyToStartContainers ([#117702](https://github.com/kubernetes/kubernetes/pull/117702), [@kannon92](https://github.com/kannon92)) [SIG Node and Testing]
### Bug or Regression
- CI job `ci-kubernetes-node-arm64-ubuntu-serial` will test node e2e on arm64, `use-dockerized-build` and `target-build-arch` are required to run this job. ([#118567](https://github.com/kubernetes/kubernetes/pull/118567), [@chendave](https://github.com/chendave)) [SIG Node and Testing]
- Fix Cronjob status.lastSuccessfulTime not populated by a manually triggered job ([#118530](https://github.com/kubernetes/kubernetes/pull/118530), [@carlory](https://github.com/carlory)) [SIG CLI]
- Fix component status calling etcd health endpoint over http which exposed kubernetes to the risk of complete watch starvation and is inconsistent with other etcd probing done by kube-apiserver. ([#118460](https://github.com/kubernetes/kubernetes/pull/118460), [@serathius](https://github.com/serathius)) [SIG API Machinery]
- Fixed the preStop hook will block the pod termination grace period ([#115835](https://github.com/kubernetes/kubernetes/pull/115835), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG Node and Testing]
- Users will no longer see an error for failed events caused due to terminating namespace. ([#114849](https://github.com/kubernetes/kubernetes/pull/114849), [@padlar](https://github.com/padlar)) [SIG API Machinery]
### Other (Cleanup or Flake)
- Kube-proxy: remove log warning about not using config file ([#118115](https://github.com/kubernetes/kubernetes/pull/118115), [@TommyStarK](https://github.com/TommyStarK)) [SIG Network]
- Make Job controller batching of syncJob invocations enabled unconditionally (it was conditional on JobReadyPods feature before).
Also, Job controller's constants for default backoff and maximal backoff are lowered down to 1s (from 10s) and 1min (from 6min), respectively. These constants are used to determine the backoff delay for the next Job controller sync in case of a request failure. ([#118615](https://github.com/kubernetes/kubernetes/pull/118615), [@mimowo](https://github.com/mimowo)) [SIG Apps and Testing]
- Migrated the interpodaffinity scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116635](https://github.com/kubernetes/kubernetes/pull/116635), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling]
## Dependencies
### Added
- github.com/alecthomas/kingpin/v2: [v2.3.1](https://github.com/alecthomas/kingpin/v2/tree/v2.3.1)
- github.com/xhit/go-str2duration: [v1.2.0](https://github.com/xhit/go-str2duration/tree/v1.2.0)
### Changed
- github.com/alecthomas/units: [f65c72e → b94a6e3](https://github.com/alecthomas/units/compare/f65c72e...b94a6e3)
- github.com/go-kit/log: [v0.2.0 → v0.2.1](https://github.com/go-kit/log/compare/v0.2.0...v0.2.1)
- github.com/kr/pretty: [v0.3.0 → v0.3.1](https://github.com/kr/pretty/compare/v0.3.0...v0.3.1)
- github.com/matttproud/golang_protobuf_extensions: [v1.0.2 → v1.0.4](https://github.com/matttproud/golang_protobuf_extensions/compare/v1.0.2...v1.0.4)
- github.com/prometheus/client_golang: [v1.14.0 → v1.16.0](https://github.com/prometheus/client_golang/compare/v1.14.0...v1.16.0)
- github.com/prometheus/common: [v0.37.0 → v0.42.0](https://github.com/prometheus/common/compare/v0.37.0...v0.42.0)
- github.com/prometheus/procfs: [v0.8.0 → v0.10.1](https://github.com/prometheus/procfs/compare/v0.8.0...v0.10.1)
- golang.org/x/sync: v0.1.0 → v0.2.0
### Removed
_Nothing has changed._
# v1.28.0-alpha.2
## Downloads for v1.28.0-alpha.2
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes.tar.gz) | 3ef803c3df0a722e4fdaa045fa7aca8c39892916b0788e0c3216747397fe4e6e8fc02483c7e1a7280ed9f2716324c067cd806037dbf9b635e268f8b62db43841
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-src.tar.gz) | c7594dfdefc92fabd6ffeef08fb98e6686c6bbf8dc952197e6629838e645c83d4b7d375e078dc3f61f6cdbd425ca68eb88f0ffdbb1e3d2e034af9efeb8f7f34b
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-darwin-amd64.tar.gz) | 66e80a10b94d81ca08bd3eb94168afa429c3bc2b036e80bd55e4d0c93b4eb9e645d575ff520e6961368d2559e9bec786fbf115d0d902af7a3b0eae43213fb9e2
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-darwin-arm64.tar.gz) | 0654d9ec7234a042d08db30cf01b69f66889451219a57a9e8a1dda2e11e7767e0ea1a171c2f16957bc6fedf1777bea455a03f453b9b06046d1588824ddb72627
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-linux-386.tar.gz) | cdb3fdaacad12e8910f7b3dc26cd4bb684fcfbf8e9d0e594a1dda3da8ca804a9f69d0d030893b15cdd4c0daf613b3efdf8c2be54aaa8406764aaa17550df2fb8
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-linux-amd64.tar.gz) | ceb64baa175b6444f719311f76b5de7cc1cfcad13650d3a3dc50fa5fbb6f5c724aa7fc63aa4df27fc1e3b58c419a3d1b3aaa75fb415c3709bd4f4c7253b4d99a
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-linux-arm.tar.gz) | c9a9f307fe69ce45a1f31ff8f68a158c08de202676952fc243cba4bca4f66f020fc68c7f36191d731ac536f9ff96a0e8c75a7ecc2884eac59468b92af7f72c5c
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-linux-arm64.tar.gz) | b5ee31bfb316559a9bec2bf7c5c2e555d1cfc07a307e1bfd2f4022b3a9988f2b6ae2e1189079a81dac0d3cc46d8982e25eafb2aaad5500be1c872313427e3544
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-linux-ppc64le.tar.gz) | d91e713944e21cd7729933fcb13118a3aae1a59e9809f7c8c23c047a55ceb9428459758b27d1496be3909d5097cdf5bcdc9089f6dbbfca6154a763ede0fc38d1
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-linux-s390x.tar.gz) | b34d9111b967d295092a57c9838f97400f526c9ad8086c562e090b3e92d171159dcece2576dfd0345c59c3adab06f5b01ff51a840e5997764b5bdc15df44c00f
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-windows-386.tar.gz) | 82dc590e72c866e561617aed045b4377b90e4eb42574e998c71fe287202acf37ff0a3f1d55545d3465b2410108e0b4a4ab921a0fd749a99efc36723fe945c1f6
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-windows-amd64.tar.gz) | a7aba1527252e942bb3a88df21a0c7a6c9f9fdbfbb1e5b2e6e28c431a413fbb51d1e50682f05a353fc6f997ed7b5e7b229f08aea0d12b92631c1b8152b029fce
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-client-windows-arm64.tar.gz) | 728574089c01b253a1e81da19680ffaf58a88780653583d1af35ad68ee50c05cbbc21a7ca1b33c293d6db0eaa1de04988b8540467f3b4dfa22e48d5ca7a93604
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-server-linux-amd64.tar.gz) | fe373323bd6940fc1801d04f3fd1f574a85d103ce6526f58bc894cf0bbbbe387a26f4a4f248249aec36f88bc15c1eceb67b85af9dc876b139bad245b0551f219
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-server-linux-arm64.tar.gz) | 4e76b003da0dedc9b457cb47fbcb4e8f719144aced497b6ebaaf9c515c8125a72e72c3a059757de3cdfc177dd4e3b18368ba47fd2fefb9cb2e4bc6a23c73f802
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-server-linux-ppc64le.tar.gz) | c17f57f13d1393d4cf32a4ac0c122b6ebe400d88a26b3bd2eae9c6b4b4ed9f26a23ab99032847fb758bc4cbfa09a4dc7b843603d81659d0b79d42953efeca15b
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-server-linux-s390x.tar.gz) | 36ce4946d3879c1e4b5e6e928be8da0eb9f146c96633d1bcb7d6541fa158f5a176d8174fba113d17bacff0db2debcedad73aed2d4054e3cc65bbf29256e62942
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-node-linux-amd64.tar.gz) | 34274b82a809fea6bdbfc1602dc66929efa71da0b3e510880a7b498253aa379a5ddc1fb1875666d2e6b5bbdd6473a7471a6b80e46526d57358ae9f97ec0e1904
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-node-linux-arm64.tar.gz) | 610dc57e34222170c57b8c31ae15392cb960c2bff750f57acf0925d1597b54326c2f78b610c0f79f9762ba67a3af077f985b0ed8ef275e0d81794775b6448d58
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-node-linux-ppc64le.tar.gz) | 2b65b45735680dab34e462f45c34c64c809bd8e80b2a45297446b1489dc974a54f8cd11364192e090e98edd987b4d7fb81b81d439987b1131d64a943d94ebb8f
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-node-linux-s390x.tar.gz) | b3cba42fa9532bb9c7d80e31b1be8cb904bad1d32d6f3c651a690be1435d7f8d610bb203c89c420e96997a11a51b3e46f05fe626d2818fcc1bd3cef7d5a15b92
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.2/kubernetes-node-windows-amd64.tar.gz) | 0b00eac88f2220dbb7f6243f9d4de41789bf3f38c14eec1c44bd48b9a067a43ef4cc9c468dc5869598d9f5ba06828b8bc5b7ea5eaa42faabaf584a8bdcb9ea29
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.0-alpha.2](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.28.0-alpha.1
## Urgent Upgrade Notes
### (No, really, you MUST read this before you upgrade)
- CephFS volume plugin ( `kubernetes.io/cephfs`) has been deprecated in this release and will be removed in a subsequent release. Alternative is to use CephFS CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. ([#118143](https://github.com/kubernetes/kubernetes/pull/118143), [@humblec](https://github.com/humblec)) [SIG Storage]
## Changes by Kind
### Feature
- Introduce support for CEL optionals (see [CEL spec proposal 246](https://github.com/google/cel-spec/wiki/proposal-246)).
This feature will not be fully enabled until a future Kubernetes release (likely to be v1.29), but is added in v1.28 to enable
safe rollback on downgrade. ([#118339](https://github.com/kubernetes/kubernetes/pull/118339), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth, Cloud Provider and Testing]
- Kubernetes is now built with Go 1.20.5 ([#118507](https://github.com/kubernetes/kubernetes/pull/118507), [@jeremyrickard](https://github.com/jeremyrickard)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Storage and Testing]
- Promote ServiceNodePortStaticSubrange to beta and it will be enabled by default ([#117877](https://github.com/kubernetes/kubernetes/pull/117877), [@xuzhenglun](https://github.com/xuzhenglun)) [SIG Network]
- The `ExpandedDNSConfig` feature has graduated to GA. 'ExpandedDNSConfig' feature was locked to default value and will be removed in v1.30. If you were setting this feature gate explicitly, please remove it now. ([#116741](https://github.com/kubernetes/kubernetes/pull/116741), [@gjkim42](https://github.com/gjkim42)) [SIG Apps, Network and Node]
- The helping message of commands which have sub-commands is now clearer and more instructive. It will show the full command instead of 'kubectl <command> --help ...'
Changed 'kubectl create secret --help' description. There will be a short introduction to the three secret types and clearer guidance on how to use the command. ([#117930](https://github.com/kubernetes/kubernetes/pull/117930), [@LronDC](https://github.com/LronDC)) [SIG CLI and Testing]
- Updated distroless I-tables to use registry.k8s.io/build-image/distroless-iptables:v0.2.5 ([#118541](https://github.com/kubernetes/kubernetes/pull/118541), [@jeremyrickard](https://github.com/jeremyrickard)) [SIG Testing]
### Bug or Regression
- Compute the backoff delay more accurately for deleted pods ([#118413](https://github.com/kubernetes/kubernetes/pull/118413), [@mimowo](https://github.com/mimowo)) [SIG Apps]
- Ensure Job status updates are batched by 1s. This fixes an unlikely scenario when a sequence of immediately
completing pods could trigger a sequence of non-batched Job status updates. ([#118470](https://github.com/kubernetes/kubernetes/pull/118470), [@mimowo](https://github.com/mimowo)) [SIG Apps]
- Fix a race condition in kube-proxy when using LocalModeNodeCIDR to avoid dropping Services traffic if the object node is recreated when kube-proxy is starting ([#118499](https://github.com/kubernetes/kubernetes/pull/118499), [@aojea](https://github.com/aojea)) [SIG Network]
- Fixed a race condition between `Run()` and `SetTransform()` and `SetWatchErrorHandler()` in shared informers. ([#117870](https://github.com/kubernetes/kubernetes/pull/117870), [@howardjohn](https://github.com/howardjohn)) [SIG API Machinery]
- Fixes bug where explain was not properly respecting jsonpaths ([#115694](https://github.com/kubernetes/kubernetes/pull/115694), [@mpuckett159](https://github.com/mpuckett159)) [SIG CLI]
- Kubelet: print sorted volumes message in events ([#117079](https://github.com/kubernetes/kubernetes/pull/117079), [@qingwave](https://github.com/qingwave)) [SIG Node]
### Other (Cleanup or Flake)
- E2e framework: the `node-role.kubernetes.io/master` taint has been removed from the default value of `--non-blocking-taints` flag. You may need to set `--non-blocking-taints` explicitly if the cluster to be tested has nodes with the deprecated `node-role.kubernetes.io/master` taint. ([#118510](https://github.com/kubernetes/kubernetes/pull/118510), [@SataQiu](https://github.com/SataQiu)) [SIG Testing]
- Kube-apiserver adds two new alpha metrics `conversion_webhook_request_total` and `conversion_webhook_duration_seconds` that allow users to monitor requests to CRD conversion webhooks, split by result, and failure_type (In case of failure). ([#118292](https://github.com/kubernetes/kubernetes/pull/118292), [@cchapla](https://github.com/cchapla)) [SIG API Machinery, Architecture and Instrumentation]
- Moved `k8s.io/kubernetes/pkg/kubelet/cri/streaming` package to `k8s.io/kubelet/pkg/cri/streaming`. ([#118253](https://github.com/kubernetes/kubernetes/pull/118253), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node, Release and Security]
- OpenAPI proto deserializations should use gnostic-models instead of the gnostic library ([#118384](https://github.com/kubernetes/kubernetes/pull/118384), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Instrumentation, Node, Storage and Testing]
- [KCCM] drop filtering nodes for the providerID when syncing load balancers, but have changes to the field trigger a re-sync of load balancers. This should ensure that cloud providers which don't specify providerID, can still use the service controller implementation to provision load balancers. ([#117602](https://github.com/kubernetes/kubernetes/pull/117602), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider and Network]
## Dependencies
### Added
- github.com/antlr/antlr4/runtime/Go/antlr/v4: [8188dc5](https://github.com/antlr/antlr4/runtime/Go/antlr/v4/tree/8188dc5)
- github.com/google/gnostic-models: [v0.6.8](https://github.com/google/gnostic-models/tree/v0.6.8)
### Changed
- github.com/dustin/go-humanize: [v1.0.0 → v1.0.1](https://github.com/dustin/go-humanize/compare/v1.0.0...v1.0.1)
- github.com/evanphx/json-patch: [v4.12.0+incompatible → v5.6.0+incompatible](https://github.com/evanphx/json-patch/compare/v4.12.0...v5.6.0)
- github.com/go-openapi/jsonreference: [v0.20.1 → v0.20.2](https://github.com/go-openapi/jsonreference/compare/v0.20.1...v0.20.2)
- github.com/google/cel-go: [v0.12.6 → v0.16.0](https://github.com/google/cel-go/compare/v0.12.6...v0.16.0)
- github.com/mitchellh/mapstructure: [v1.4.1 → v1.1.2](https://github.com/mitchellh/mapstructure/compare/v1.4.1...v1.1.2)
- go.starlark.net: 8dd3e2e → a134d8f
- golang.org/x/exp: 6cc2880 → a9213ee
- golang.org/x/sys: v0.7.0 → v0.8.0
- k8s.io/kube-openapi: 7828149 → 7562a10
- sigs.k8s.io/kustomize/api: v0.13.2 → 6ce0bf3
- sigs.k8s.io/kustomize/cmd/config: v0.11.1 → v0.11.2
- sigs.k8s.io/kustomize/kustomize/v5: v5.0.1 → 6ce0bf3
- sigs.k8s.io/kustomize/kyaml: v0.14.1 → 6ce0bf3
### Removed
- github.com/antlr/antlr4/runtime/Go/antlr: [v1.4.10](https://github.com/antlr/antlr4/runtime/Go/antlr/tree/v1.4.10)
- github.com/docopt/docopt-go: [ee0de3b](https://github.com/docopt/docopt-go/tree/ee0de3b)
- github.com/google/gnostic: [v0.5.7-v3refs](https://github.com/google/gnostic/tree/v0.5.7-v3refs)
# v1.28.0-alpha.1
## Downloads for v1.28.0-alpha.1
### Source Code
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes.tar.gz) | 65d841f778b00a04a13f3e722753704d4164f8590c2b0aca9cbb9bf85822be5343205ead8c71f9502d8b22fc84d80804fed5edc665662b0405bb0efa65fec808
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-src.tar.gz) | 82fbe3f389b922cc635a896fa6c3e8cc342e4ca70003ca5491c7b3eb2e38065349e270da9c0deb0e541271978ade247ff3a420806a51d035a5a850262e41baa9
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-darwin-amd64.tar.gz) | c5be770467a8617221021255a22a970a72ccee3672b1973fb31c65b1de02767d014a8e9058f710f0d9b402f2b056fd17ed216cb1d6126f9738efb16f88e184c0
[kubernetes-client-darwin-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-darwin-arm64.tar.gz) | a194b07e23b8cee142080361394e0db7f3fb0488c16eeef3059dfb178f4cef6e124ad31c511a516058b8f82a6ab0f0194183714016ebd88e3060368528405e2c
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-linux-386.tar.gz) | ada349bd3f76b5572467a8fad504c26a223eeb50ad7677287b39db434adb5a59d2ceadd1922712f99878153f20fa8b0cd2b30a16e8e178a41c6ac747b55ee79c
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-linux-amd64.tar.gz) | c729d419e53a006996f5e583e0fa9a541ea7d2df7dc875dae729c63cd8222f10121908750c48ff34942fcbdf6456ed977bef86c4b979202fab120de0a7a42fc9
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-linux-arm.tar.gz) | 6bf4a115b4f4b7b21d193fe44f99c5b019e9f2097e831bd44958de6e63bd8068a70a9cfa535dc18dca23c0c4461195e8a62c8f1cd9faff7f5bb3c7b1b13ad604
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-linux-arm64.tar.gz) | 536101d9f50bf71e66e35781e0ca729156227405225986198276a43d2cf32aa2cbae32f0743bcb967701309ea3bd19e9e9f6150e532a2d251440f18ca8afbd16
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-linux-ppc64le.tar.gz) | a8dd8c0aaa7dce825f982edbff1ecd57671643e2725390c60b43450118abf2dd3594f306af6cbbd2df1aa146a0b21d0576c1b6e8e1dd2b50190702d1e879ad3a
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-linux-s390x.tar.gz) | b6ab35eb6c55536f91c4c0ae32b8db3462426fea11a4cce3e06581129995b42c4acdd16674e357d92280dae5ab9f50bcb6b8d5052d65c0a06b9c21fbb646e830
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-windows-386.tar.gz) | 362c2f7f7327775a75b0c6cc2e3e372475d7d9291ad5f7c224632e037fe181b149d6def98dbd034d8ba73d3bac335a7788fbaa08df924e05c9ed9844fa75135f
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-windows-amd64.tar.gz) | 75297a5c9f7d8f39f640d97bf4ece9a78b2226103d6b66865dcf6752375bf76b9d3e3d4b13efb291275621e7b1e4858eaa36f469ac73495bba43dfca2b900085
[kubernetes-client-windows-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-client-windows-arm64.tar.gz) | e79cddab0abb31ef7f17855d9b14799fc7a66247c3aa71eed01231d40cb5caa7dad08082904fd18cc126cef1d3a7c2f42b8a8994e7ab40271eb0d8baa1a42f74
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-server-linux-amd64.tar.gz) | 64b5c5e1502fbe6a21ff6cde999408ff83f1d3b1088fbe05d720f90e5f0a9193b5ba1b1aaaee65e6ec1354e63e60d29c55a90535f79624f4526dea96295ad48d
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-server-linux-arm64.tar.gz) | 26519f8406e2900b00a22d4e03260701ded84ddba0730f25a794f5b4bfcba452ab1c321f32fe30a7e2bf748fc93cf05fe81b2fdec7fa86af1e9f882428179f85
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-server-linux-ppc64le.tar.gz) | c66df63d33607d8a3f2ae57ca80e4134b423bd8448ee3ecd72936f0c5973d027ab27f92481fc83e41b4b929cdae4be3865477e59f316dc102e19aa79e52afe6b
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-server-linux-s390x.tar.gz) | 5a6c30cdf7f24b2ab906cf1a27f07bb7e5fafef100942b33320c2e8445b7934c2663ae7b7cc47f8aec173c1788ace9576144df357bef83e3d7a42e827f1a7c94
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-node-linux-amd64.tar.gz) | 95ce88f26c3809f268e8b83122dc4d0685e7b31f44dedad3b1360edd76c921e2a6e0c9077c136fea078299f4451280fbf49c9f956fc30339db752e5aa0e73367
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-node-linux-arm64.tar.gz) | bf36de0876bab1b08e1268dd5602d5af46e99a9939e8befcb9d6fea91d04fc67438d136ae28503c3342dcff63e9849b2ca81b00c29627a9a477fcaed5e4f3443
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-node-linux-ppc64le.tar.gz) | 649b49fe2319a9fd149d08665bdbe3c825f21bb96d4695dbb4fadad367e027f000272326217194f8319cb074ee6f15dc9b6bf4c0ff4dfcda08003680b39faebf
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-node-linux-s390x.tar.gz) | 2b0c9466e9d42576d1bae61b2141e41521cfb0ae2c13ff3b59ea8abec124a44601c76a3e9e0a6283b6c74e9fee27d420b131238811f4dd4bdee789247b44941c
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.28.0-alpha.1/kubernetes-node-windows-amd64.tar.gz) | a26243c3e7bab5180b5ff44139dfcecb6975326fdc6dec9b71f5dfccd89889710bcfadcde5c5a0c9ef03378396729e9b2763b38d6b67840239cb144981b98317
### Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
[registry.k8s.io/conformance:v1.28.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/conformance-s390x)
[registry.k8s.io/kube-apiserver:v1.28.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-apiserver-s390x)
[registry.k8s.io/kube-controller-manager:v1.28.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-controller-manager-s390x)
[registry.k8s.io/kube-proxy:v1.28.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-proxy-s390x)
[registry.k8s.io/kube-scheduler:v1.28.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kube-scheduler-s390x)
[registry.k8s.io/kubectl:v1.28.0-alpha.1](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl) | [amd64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-amd64), [arm64](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-arm64), [ppc64le](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-ppc64le), [s390x](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/kubectl-s390x)
## Changelog since v1.27.0
## Changes by Kind
### Deprecation
- Remove tracking annotation from validation and defaulting ([#117633](https://github.com/kubernetes/kubernetes/pull/117633), [@kannon92](https://github.com/kannon92)) [SIG Apps]
- Remove withdrawn feature NetworkPolicyStatus ([#115843](https://github.com/kubernetes/kubernetes/pull/115843), [@rikatz](https://github.com/rikatz)) [SIG API Machinery, Apps, Architecture, Network and Testing]
### API Change
- Added a warning that TLS 1.3 ciphers are not configurable. ([#115399](https://github.com/kubernetes/kubernetes/pull/115399), [@3u13r](https://github.com/3u13r)) [SIG API Machinery and Node]
- Added error handling for seccomp localhost configurations that do not properly set a localhostProfile ([#117020](https://github.com/kubernetes/kubernetes/pull/117020), [@cji](https://github.com/cji)) [SIG API Machinery and Node]
- Added new config option `delayCacheUntilActive` to `KubeSchedulerConfiguration` that can provide a tradeoff between memory efficiency and scheduling speed when their leadership is updated in `kube-scheduler` ([#115754](https://github.com/kubernetes/kubernetes/pull/115754), [@linxiulei](https://github.com/linxiulei)) [SIG API Machinery and Scheduling]
- Client-go: Improved memory use of reflector caches when watching large numbers of objects which do not change frequently ([#113362](https://github.com/kubernetes/kubernetes/pull/113362), [@sxllwx](https://github.com/sxllwx)) [SIG API Machinery]
- Kube-controller-manager: The `LegacyServiceAccountTokenCleanUp` feature gate is now available as alpha (off by default). When enabled, the `legacy-service-account-token-cleaner` controller loop removes service account token secrets that have not been used in the time specified by `--legacy-service-account-token-clean-up-period` (defaulting to one year), **and are** referenced from the `.secrets` list of a ServiceAccount object, **and are not** referenced from pods. ([#115554](https://github.com/kubernetes/kubernetes/pull/115554), [@yt2985](https://github.com/yt2985)) [SIG API Machinery, Apps, Auth, Release and Testing]
- Kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta2 is removed in v1.28. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. ([#117649](https://github.com/kubernetes/kubernetes/pull/117649), [@SataQiu](https://github.com/SataQiu)) [SIG API Machinery, Scheduling and Testing]
- NodeVolumeLimits implement the PreFilter extension point for skipping the Filter phase if the Pod doesn't use volumes with limits. ([#115398](https://github.com/kubernetes/kubernetes/pull/115398), [@tangwz](https://github.com/tangwz)) [SIG Scheduling]
- Pods which set `hostNetwork: true` and declare ports get the `hostPort` field set automatically. Previously this would happen in the PodTemplate of a Deployment, DaemonSet or other workload API. Now `hostPort` will only be set when an actual Pod is being created. If this presents a problem, setting the feature gate "DefaultHostNetworkHostPortsInPodTemplates" to true will revert this behavior. Please file a kubernetes bug if you need to do this. ([#117696](https://github.com/kubernetes/kubernetes/pull/117696), [@thockin](https://github.com/thockin)) [SIG Apps]
- Removing WindowsHostProcessContainers feature-gate ([#117570](https://github.com/kubernetes/kubernetes/pull/117570), [@marosset](https://github.com/marosset)) [SIG API Machinery, Apps, Auth, Node and Windows]
- Revised the comment about the feature-gate level for PodFailurePolicy from alpha to beta ([#117802](https://github.com/kubernetes/kubernetes/pull/117802), [@kerthcet](https://github.com/kerthcet)) [SIG API Machinery and Apps]
- The `SelfSubjectReview` API is promoted to `authentication.k8s.io/v1` and the `kubectl auth whoami` command is GA. ([#117713](https://github.com/kubernetes/kubernetes/pull/117713), [@nabokihms](https://github.com/nabokihms)) [SIG API Machinery, Architecture, Auth, CLI and Testing]
### Feature
- Add '--concurrent-job-syncs' flag for kube-controller-manager to set the number of job controller workers ([#117138](https://github.com/kubernetes/kubernetes/pull/117138), [@tosi3k](https://github.com/tosi3k)) [SIG API Machinery and CLI]
- Add DisruptionTarget condition to the pod preempted by Kubelet to make room for a critical pod ([#117586](https://github.com/kubernetes/kubernetes/pull/117586), [@mimowo](https://github.com/mimowo)) [SIG Node and Testing]
- Added a container image for `kubectl` at `registry.k8s.io/kubectl` across the same architectures as other images (linux/amd64 linux/arm64 linux/s390x linux/ppc64le) ([#116672](https://github.com/kubernetes/kubernetes/pull/116672), [@dims](https://github.com/dims)) [SIG Architecture and Release]
- Added support for pod `hostNetwork` field selector ([#110477](https://github.com/kubernetes/kubernetes/pull/110477), [@halfcrazy](https://github.com/halfcrazy)) [SIG Apps and Node]
- Apiserver adds two new metrics `etcd_requests_total` and `etcd_request_errors_total` that allow users to monitor requests to etcd storage, split by operation and resource type. ([#117222](https://github.com/kubernetes/kubernetes/pull/117222), [@iyear](https://github.com/iyear)) [SIG API Machinery]
- Bump metrics-server to v0.6.3. ([#117120](https://github.com/kubernetes/kubernetes/pull/117120), [@dgrisonnet](https://github.com/dgrisonnet)) [SIG Cloud Provider and Instrumentation]
- Client-go exposes two new metrics to monitor the client-go logic that
generate http.Transports for the clients.
- rest_client_transport_cache_entries is a gauge metric
with the number of existin entries in the internal cache
- rest_client_transport_create_calls_total is a counter
that increments each time a new transport is created, storing
the result of the operation needed to generate it: hit, miss
or uncacheable ([#117295](https://github.com/kubernetes/kubernetes/pull/117295), [@aojea](https://github.com/aojea)) [SIG API Machinery, Architecture, Instrumentation, Network, Node and Testing]
- External credential provider plugins now have their standard error output logged by kubelet upon failures. ([#117448](https://github.com/kubernetes/kubernetes/pull/117448), [@cartermckinnon](https://github.com/cartermckinnon)) [SIG Node]
- Graduated the `LegacyServiceAccountTokenTracking` feature gate to GA. The usage of auto-generated secret-based service account token now produces warnings, and relevant Secrets are labeled with a last-used timestamp (label key `kubernetes.io/legacy-token-last-used`). ([#117591](https://github.com/kubernetes/kubernetes/pull/117591), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Auth and Testing]
- Klog text output now uses JSON as encoding for structs, maps and slices. ([#117687](https://github.com/kubernetes/kubernetes/pull/117687), [@pohly](https://github.com/pohly)) [SIG Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
- Kube-proxy handles Terminating EndpointSlices conditions and enables zero downtime deployments for Services with ExternalTrafficPolicy=Local author: @andrewsykim ([#117718](https://github.com/kubernetes/kubernetes/pull/117718), [@aojea](https://github.com/aojea)) [SIG Network, Testing and Windows]
- Kube-proxy in iptables mode now has separate `sync_full_proxy_rules_duration_seconds`
and `sync_partial_proxy_rules_duration_seconds` (in addition to the existing
`sync_proxy_rules_duration_seconds`), to give better information about how long
each sync type is taking, rather than only giving a weighted average of the two
sync types together. ([#117787](https://github.com/kubernetes/kubernetes/pull/117787), [@danwinship](https://github.com/danwinship)) [SIG Network]
- Kubeadm: add `--feature-gates` flag for `kubeadm upgrade node` ([#118316](https://github.com/kubernetes/kubernetes/pull/118316), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle]
- Kubeadm: add a new "kubeadm config validate" command that can be used to validate any input config file. Use the --config flag to pass a config file to it. See the command --help screen for more information. As a result of adding this new command, enhance the validation capabilities of the existing "kubeadm config migrate" command. For both commands unknown APIs or fields will throw errors. ([#118013](https://github.com/kubernetes/kubernetes/pull/118013), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle]
- Kubernetes is now built with Go 1.20.4 ([#117744](https://github.com/kubernetes/kubernetes/pull/117744), [@xmudrii](https://github.com/xmudrii)) [SIG Release and Testing]
- Metric `scheduler_scheduler_goroutines` is removed. Use `scheduler_goroutines` instead. ([#117727](https://github.com/kubernetes/kubernetes/pull/117727), [@kerthcet](https://github.com/kerthcet)) [SIG Scheduling]
- Migrated `pkg/scheduler/framework/preemption` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116835](https://github.com/kubernetes/kubernetes/pull/116835), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling]
- Migrated `pod-security-admission` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#114471](https://github.com/kubernetes/kubernetes/pull/114471), [@Namanl2001](https://github.com/Namanl2001)) [SIG Apps and Auth]
- Migrated the noderesources scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116748](https://github.com/kubernetes/kubernetes/pull/116748), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling]
- Migrated the podtopologyspread scheduler plugins to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116797](https://github.com/kubernetes/kubernetes/pull/116797), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling]
- Set metrics-server's metric-resolution to 15s ([#117121](https://github.com/kubernetes/kubernetes/pull/117121), [@dgrisonnet](https://github.com/dgrisonnet)) [SIG Cloud Provider and Instrumentation]
- SubjectAccessReview requests sent to webhook authorizers now default `spec.resourceAttributes.version` to `*` if unset. ([#116937](https://github.com/kubernetes/kubernetes/pull/116937), [@AxeZhan](https://github.com/AxeZhan)) [SIG Apps and Auth]
- Support specifying a custom retry period for cloud load-balancer operations ([#94021](https://github.com/kubernetes/kubernetes/pull/94021), [@timoreimann](https://github.com/timoreimann)) [SIG API Machinery, Cloud Provider and Network]
- The Kubernetes apiserver now emits a warning message for Pods with a null labelSelector in podAffinity or topologySpreadConstraints. The null labelSelector means "match none". Using it in podAffinity or topologySpreadConstraint could lead to unintended behavior. ([#117025](https://github.com/kubernetes/kubernetes/pull/117025), [@sanposhiho](https://github.com/sanposhiho)) [SIG Scheduling]
- The scheduler skips the InterPodAffinity Score plugin when nothing to do with the Pod.
It will affect some metrics values related to the InterPodAffinity Score plugin. ([#117794](https://github.com/kubernetes/kubernetes/pull/117794), [@utam0k](https://github.com/utam0k)) [SIG Scheduling]
- The scheduler skips the PodTopologySpread Filter plugin if no spread constraints.
It will affect some metrics values related to the PodTopologySpread Filter plugin. ([#117683](https://github.com/kubernetes/kubernetes/pull/117683), [@utam0k](https://github.com/utam0k)) [SIG Scheduling]
- The short names vwc and mwc were introduced for the resources validatingwebhookconfigurations and mutatingwebhookconfigurations. ([#117535](https://github.com/kubernetes/kubernetes/pull/117535), [@hysyeah](https://github.com/hysyeah)) [SIG API Machinery]
- Update etcd image to 3.5.9-0 ([#117999](https://github.com/kubernetes/kubernetes/pull/117999), [@kkkkun](https://github.com/kkkkun)) [SIG API Machinery]
- Update the scheduler interface and cache methods to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116849](https://github.com/kubernetes/kubernetes/pull/116849), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Apps, Instrumentation, Scheduling and Testing]
- Updated distroless iptables to use released image `registry.k8s.io/build-image/distroless-iptables:v0.2.4` ([#117746](https://github.com/kubernetes/kubernetes/pull/117746), [@xmudrii](https://github.com/xmudrii)) [SIG Testing]
- `--version=v1.X.Y...` can now be used to set the prerelease and buildID portions of the version reported by components ([#117688](https://github.com/kubernetes/kubernetes/pull/117688), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Architecture and Release]
### Documentation
- Enhanced clarity in error messaging when waiting for volume creation ([#118262](https://github.com/kubernetes/kubernetes/pull/118262), [@torredil](https://github.com/torredil)) [SIG Apps and Storage]
### Failing Test
- Allow Azure Disk e2es to use newer topology labels if available from nodes ([#117216](https://github.com/kubernetes/kubernetes/pull/117216), [@gnufied](https://github.com/gnufied)) [SIG Storage and Testing]
- Fix nil pointer in test AfterEach volumeperf.go for sidecar release ([#117368](https://github.com/kubernetes/kubernetes/pull/117368), [@sunnylovestiramisu](https://github.com/sunnylovestiramisu)) [SIG Storage and Testing]
### Bug or Regression
- CVE-2023-27561 CVE-2023-25809 CVE-2023-28642: Bump fix runc v1.1.4 -> v1.1.5 ([#117095](https://github.com/kubernetes/kubernetes/pull/117095), [@PushkarJ](https://github.com/PushkarJ)) [SIG Architecture, Node and Security]
- Code blocks in kubectl {$COMMAND}--help will move right by 3 indentation. ([#118029](https://github.com/kubernetes/kubernetes/pull/118029), [@ardaguclu](https://github.com/ardaguclu)) [SIG CLI]
- During device plugin allocation, resources requested by the pod can only be allocated if the device plugin has registered itself to kubelet AND healthy devices are present on the node to be allocated. If these conditions are not sattsfied, the pod would fail with `UnexpectedAdmissionError` error. ([#116376](https://github.com/kubernetes/kubernetes/pull/116376), [@swatisehgal](https://github.com/swatisehgal)) [SIG Node and Testing]
- Fix Topology Aware Hints not working when the `topology.kubernetes.io/zone` label is added after Node creation ([#117245](https://github.com/kubernetes/kubernetes/pull/117245), [@tnqn](https://github.com/tnqn)) [SIG Apps and Network]
- Fix a data race in TopologyCache when `AddHints` and `SetNodes` are called concurrently ([#117249](https://github.com/kubernetes/kubernetes/pull/117249), [@tnqn](https://github.com/tnqn)) [SIG Apps and Network]
- Fix bug where `listOfStrings.join()` in CEL expressions resulted in an unexpected internal error. ([#117593](https://github.com/kubernetes/kubernetes/pull/117593), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery]
- Fix incorrect calculation for ResourceQuota with PriorityClass as its scope. ([#117677](https://github.com/kubernetes/kubernetes/pull/117677), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG API Machinery]
- Fix performance regression in scheduler caused by frequent metric lookup on critical code path. ([#117594](https://github.com/kubernetes/kubernetes/pull/117594), [@tosi3k](https://github.com/tosi3k)) [SIG Scheduling]
- Fix restricted debug profile. ([#117543](https://github.com/kubernetes/kubernetes/pull/117543), [@mochizuki875](https://github.com/mochizuki875)) [SIG CLI and Testing]
- Fix: After a Node is down and take some time to get back to up again, the mount point of the evicted Pods cannot be cleaned up successfully. (#111933) Meanwhile Kubelet will print the log `Orphaned pod "xxx" found, but error not a directory occurred when trying to remove the volumes dir` every 2 seconds. (#105536) ([#116134](https://github.com/kubernetes/kubernetes/pull/116134), [@cvvz](https://github.com/cvvz)) [SIG Node and Storage]
- Fix: the volume is not detached after the pod and PVC objects are deleted ([#116138](https://github.com/kubernetes/kubernetes/pull/116138), [@cvvz](https://github.com/cvvz)) [SIG Storage]
- Fixed a bug that unintentionally overrides your custom Accept headers in http (live-/readiness)-probes if the header is in lower casing ([#114606](https://github.com/kubernetes/kubernetes/pull/114606), [@tuunit](https://github.com/tuunit)) [SIG Network and Node]
- Fixed a bug where pv recycler failed to scrub volume with too many files in the directory due to hitting ARG_MAX limit with rm command (#117189). ([#117283](https://github.com/kubernetes/kubernetes/pull/117283), [@defo89](https://github.com/defo89)) [SIG Cloud Provider and Storage]
- Fixed a memory leak in the Kubernetes API server that occurs during APIService processing. ([#117258](https://github.com/kubernetes/kubernetes/pull/117258), [@enj](https://github.com/enj)) [SIG API Machinery]
- Fixed an issue where the API server did not send impersonated UID to authentication webhooks. ([#116681](https://github.com/kubernetes/kubernetes/pull/116681), [@stlaz](https://github.com/stlaz)) [SIG API Machinery and Auth]
- Fixed bug to correctly report `ErrRegistryUnavailable` on pulling container images for remote CRI runtimes. ([#117612](https://github.com/kubernetes/kubernetes/pull/117612), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node]
- Fixed bug where using the $deleteFromPrimitiveList directive in a strategic merge patch of certain fields would remove the other values from the list instead of the values specified. ([#110472](https://github.com/kubernetes/kubernetes/pull/110472), [@brianpursley](https://github.com/brianpursley)) [SIG API Machinery]
- Fixed issue where kubectl-convert would fail when encountering resources that could not be converted to the specified api version. New behavior is to warn the user of the failed conversions and continue to convert the remaining resources. ([#117002](https://github.com/kubernetes/kubernetes/pull/117002), [@gxwilkerson33](https://github.com/gxwilkerson33)) [SIG CLI and Testing]
- Fixed issue where there was no response or error from kubectl rollout status when there were no resources of specified kind. ([#117884](https://github.com/kubernetes/kubernetes/pull/117884), [@gxwilkerson33](https://github.com/gxwilkerson33)) [SIG CLI]
- Fixed vSphere cloud provider not to skip detach volumes from nodes at kube-controller-startup. ([#117243](https://github.com/kubernetes/kubernetes/pull/117243), [@jsafrane](https://github.com/jsafrane)) [SIG Cloud Provider]
- Fixes a bug at kube-apiserver start where APIService objects for custom resources could be deleted and recreated. ([#118104](https://github.com/kubernetes/kubernetes/pull/118104), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Testing]
- Fixes a race condition serving OpenAPI content ([#117705](https://github.com/kubernetes/kubernetes/pull/117705), [@Jefftree](https://github.com/Jefftree)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Instrumentation and Node]
- Fixes a regression in 1.27.0 that resulted in "missing metadata in converted object" errors when modifying objects for multi-version custom resource definitions with a conversion strategy of `None`. ([#117301](https://github.com/kubernetes/kubernetes/pull/117301), [@ncdc](https://github.com/ncdc)) [SIG API Machinery]
- Fixes a regression in kubectl and client-go discovery when configured with a server URL other than the root of a server. ([#117495](https://github.com/kubernetes/kubernetes/pull/117495), [@ardaguclu](https://github.com/ardaguclu)) [SIG API Machinery]
- Fixes bug that caused a resource to include patch directives when using strategic merge patch against a non-existent field ([#117568](https://github.com/kubernetes/kubernetes/pull/117568), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing]
- Fixes creationTimestamp: null causing unnecessary writes to etcd ([#116865](https://github.com/kubernetes/kubernetes/pull/116865), [@alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing]
- If `kubeadm reset` finds no etcd member ID for the peer it removes during the `remove-etcd-member` phase, it continues immediately to other phases, instead of retrying the phase for up to 3 minutes before continuing. ([#117724](https://github.com/kubernetes/kubernetes/pull/117724), [@dlipovetsky](https://github.com/dlipovetsky)) [SIG Cluster Lifecycle]
- Improved exponential backoff in Reflector, significantly reducing the load on Kubernetes apiserver in case of throttling of requests. ([#118132](https://github.com/kubernetes/kubernetes/pull/118132), [@marseel](https://github.com/marseel)) [SIG API Machinery and Scalability]
- Known issue: fixed that the PreEnqueue plugins aren't executed for Pods proceeding to activeQ through backoffQ. ([#117194](https://github.com/kubernetes/kubernetes/pull/117194), [@sanposhiho](https://github.com/sanposhiho)) [SIG Release and Scheduling]
- Kube-apiserver always removes its endpoint from kubernetes service during graceful shutdown (even if it's the only/last one) ([#116685](https://github.com/kubernetes/kubernetes/pull/116685), [@czybjtu](https://github.com/czybjtu)) [SIG API Machinery]
- Kubeadm: crictl pull should use `-i` to set the image service endpoint ([#117835](https://github.com/kubernetes/kubernetes/pull/117835), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle]
- Kubeadm: fix a bug where file copy(backup) could not be executed correctly on Windows platform during upgrade ([#117861](https://github.com/kubernetes/kubernetes/pull/117861), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle]
- Kubeadm: fix a bug where the static pod changes detection logic is inconsistent with kubelet ([#118069](https://github.com/kubernetes/kubernetes/pull/118069), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle]
- Kubeadm: speedup init by 0s or 20s. kubelet-start phase is now after etcd and control-plane phases, removing a race condition between kubelet looking for static pod manifests and kubeadm writing them. ([#117984](https://github.com/kubernetes/kubernetes/pull/117984), [@champtar](https://github.com/champtar)) [SIG Cluster Lifecycle]
- Kubeadm: throw warnings instead of errors for deprecated feature gates ([#118270](https://github.com/kubernetes/kubernetes/pull/118270), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle]
- Kubectl events --for will also support fully qualified names such as replicasets.apps, etc. ([#117034](https://github.com/kubernetes/kubernetes/pull/117034), [@ardaguclu](https://github.com/ardaguclu)) [SIG CLI and Testing]
- Kubelet now skips pod resource checks when the request is zero. ([#116408](https://github.com/kubernetes/kubernetes/pull/116408), [@ChenLingPeng](https://github.com/ChenLingPeng)) [SIG Scheduling]
- Kubelet terminates pods correctly upon restart, fixing an issue where pods may have not been fully terminated if the kubelet was restarted during pod termination. ([#117019](https://github.com/kubernetes/kubernetes/pull/117019), [@bobbypage](https://github.com/bobbypage)) [SIG Node and Testing]
- Kubelet will ensure /etc/hosts file is mode 0644 regardless of umask. ([#113209](https://github.com/kubernetes/kubernetes/pull/113209), [@luozhiwenn](https://github.com/luozhiwenn)) [SIG Node]
- Number of errors reported to the metric `storage_operation_duration_seconds_count` for emptyDir decreased significantly because previously one error was reported for each projected volume created. ([#117022](https://github.com/kubernetes/kubernetes/pull/117022), [@mpatlasov](https://github.com/mpatlasov)) [SIG Storage]
- Pod termination will be faster when the pod has a missing volume reference. ([#117412](https://github.com/kubernetes/kubernetes/pull/117412), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node and Testing]
- Recording timing traces had a race condition. Impact in practice was probably low. ([#117139](https://github.com/kubernetes/kubernetes/pull/117139), [@pohly](https://github.com/pohly)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
- Removed leading zeros from the etcd member ID in kubeadm log messages. ([#117919](https://github.com/kubernetes/kubernetes/pull/117919), [@dlipovetsky](https://github.com/dlipovetsky)) [SIG Cluster Lifecycle]
- Resolves a spurious "Unknown discovery response content-type" error in client-go discovery requests by tolerating extra content-type parameters in API responses ([#117571](https://github.com/kubernetes/kubernetes/pull/117571), [@seans3](https://github.com/seans3)) [SIG API Machinery]
- Reverted NewVolumeManagerReconstruction and SELinuxMountReadWriteOncePod feature gates to disabled by default to resolve a regression of volume reconstruction on kubelet/node restart ([#117751](https://github.com/kubernetes/kubernetes/pull/117751), [@liggitt](https://github.com/liggitt)) [SIG Storage]
- Setting a mirror pod's phase to Succeeded or Failed can prevent the corresponding static pod from restarting due mutation of a Kubelet cache. ([#116482](https://github.com/kubernetes/kubernetes/pull/116482), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node]
- Show a warning when `volume.beta.kubernetes.io/storage-class` annotation is used in pv or pvc ([#117036](https://github.com/kubernetes/kubernetes/pull/117036), [@haoruan](https://github.com/haoruan)) [SIG Storage]
- Static pods were taking extra time to be restarted after being updated. Static pods that are waiting to restart were not correctly counted in `kubelet_working_pods`. ([#116995](https://github.com/kubernetes/kubernetes/pull/116995), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node]
- This PR adds additional validation for endpoint ip configuration while iterating through queried endpoint list. ([#116749](https://github.com/kubernetes/kubernetes/pull/116749), [@princepereira](https://github.com/princepereira)) [SIG Network and Windows]
- Update etcd version to 3.5.8 ([#117335](https://github.com/kubernetes/kubernetes/pull/117335), [@kkkkun](https://github.com/kkkkun)) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing]
- Updated static pods are restarted 2s faster by correcting a safe but non-optimal ordering bug. ([#116690](https://github.com/kubernetes/kubernetes/pull/116690), [@smarterclayton](https://github.com/smarterclayton)) [SIG Node]
- [KCCM] service controller: change the cloud controller manager to make `providerID` a predicate when synchronizing nodes. This change allows load balancer integrations to ensure that the `providerID` is set when configuring
load balancers and targets. ([#117388](https://github.com/kubernetes/kubernetes/pull/117388), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Cloud Provider and Network]
### Other (Cleanup or Flake)
- A v2-level info log will be added, which will output the details of the pod being preempted, including victim and preemptor ([#117214](https://github.com/kubernetes/kubernetes/pull/117214), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG Scheduling]
- Allow container runtimes to use `ErrSignatureValidationFailed` as possible image pull failure. ([#117717](https://github.com/kubernetes/kubernetes/pull/117717), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node]
- Deprecate genericclioptions.IOStreams and use genericiooptions.IOStreams ([#117102](https://github.com/kubernetes/kubernetes/pull/117102), [@ardaguclu](https://github.com/ardaguclu)) [SIG Auth, CLI and Release]
- Enables the node-local kubelet podresources API endpoint on windows, alongside unix. ([#115133](https://github.com/kubernetes/kubernetes/pull/115133), [@ffromani](https://github.com/ffromani)) [SIG Cloud Provider, Node, Testing and Windows]
- Fixed dra e2e image build on non-amd64 architectures ([#117912](https://github.com/kubernetes/kubernetes/pull/117912), [@bart0sh](https://github.com/bart0sh)) [SIG Node and Testing]
- Kube-apiserver adds two new metrics `authorization_attempts_total` and `authorization_duration_seconds` that allow users to monitor requests to authorization webhooks, split by result. ([#117211](https://github.com/kubernetes/kubernetes/pull/117211), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG API Machinery, Auth and Instrumentation]
- Kubeadm: introduce a new feature gate UpgradeAddonsBeforeControlPlane to fix a kube-proxy skew policy misalignment. Its default value is `false`. Upgrade of the CoreDNS and kube-proxy addons will now trigger after all the control plane instances have been upgraded, unless the fearure gate is set to true. This feature gate will be removed in a future release. ([#117660](https://github.com/kubernetes/kubernetes/pull/117660), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle]
- Marked the feature gate `ExperimentalHostUserNamespaceDefaulting` as deprecated.
Enabling the feature gate already had no effect; the deprecation allows for removing the feature gate in a future release. ([#116723](https://github.com/kubernetes/kubernetes/pull/116723), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Node]
- Migrated `pkg/scheduler/framework/runtime` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116842](https://github.com/kubernetes/kubernetes/pull/116842), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling]
- Migrated the volumezone scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116829](https://github.com/kubernetes/kubernetes/pull/116829), [@mengjiao-liu](https://github.com/mengjiao-liu)) [SIG Instrumentation and Scheduling]
- Projects which use k8s.io/code-generator and invoke `generate-groups` or `generate-internal-groups.sh` have a new, simpler script (`kube_codegen.sh`) they can use. The old scripts are deprecated but remain intact. ([#117262](https://github.com/kubernetes/kubernetes/pull/117262), [@thockin](https://github.com/thockin)) [SIG API Machinery and Instrumentation]
- Remove GAed feature gate DelegateFSGroupToCSIDriver ([#117655](https://github.com/kubernetes/kubernetes/pull/117655), [@carlory](https://github.com/carlory)) [SIG Storage]
- Remove GAed feature gate DevicePlugins ([#117656](https://github.com/kubernetes/kubernetes/pull/117656), [@carlory](https://github.com/carlory)) [SIG Node]
- Remove GAed feature gate KubeletCredentialProviders ([#116901](https://github.com/kubernetes/kubernetes/pull/116901), [@pacoxu](https://github.com/pacoxu)) [SIG Cloud Provider, Node and Testing]
- Remove GAed feature gates: MixedProtocolLBService, ServiceInternalTrafficPolicy, ServiceIPStaticSubrange, and EndpointSliceTerminatingCondition ([#117237](https://github.com/kubernetes/kubernetes/pull/117237), [@yulng](https://github.com/yulng)) [SIG Network]
- Removed the deprecated `azureFile` in-tree storage plugin ([#118236](https://github.com/kubernetes/kubernetes/pull/118236), [@andyzhangx](https://github.com/andyzhangx)) [SIG API Machinery, Cloud Provider, Node and Storage]
- Structured logging of NamespacedName was inconsistent with klog.KObj. Now both use lower case field names and namespace is optional. ([#117238](https://github.com/kubernetes/kubernetes/pull/117238), [@pohly](https://github.com/pohly)) [SIG API Machinery, Architecture and Instrumentation]
- The `generate_groups.sh` and `generate_internal_groups.sh` scripts from the k8s.io/code-generator repo are deprecated (but still work) in favor of `kube_codegen.sh` in that same repo. Projects which use the old scripts are encouraged to look at adopting the new one. ([#117897](https://github.com/kubernetes/kubernetes/pull/117897), [@thockin](https://github.com/thockin)) [SIG API Machinery]
- The feature gate CSIStorageCapacity have been removed and must no longer be referenced in `--feature-gates` flags ([#118018](https://github.com/kubernetes/kubernetes/pull/118018), [@humblec](https://github.com/humblec)) [SIG Storage]
- The feature gates `DisableAcceleratorUsageMetrics` and `PodSecurity` that graduated to GA and were unconditionally enabled have been removed in v1.28 ([#114068](https://github.com/kubernetes/kubernetes/pull/114068), [@cyclinder](https://github.com/cyclinder)) [SIG API Machinery, Node, Scheduling and Storage]
- The kubelet podresources endpoint is GA and always enabled ([#116525](https://github.com/kubernetes/kubernetes/pull/116525), [@ffromani](https://github.com/ffromani)) [SIG Node]
- Updated Cluster Autosaler to version 1.26.1 ([#116526](https://github.com/kubernetes/kubernetes/pull/116526), [@pacoxu](https://github.com/pacoxu)) [SIG Autoscaling and Cloud Provider]
- Updated cri-tools to v1.26.1. ([#116649](https://github.com/kubernetes/kubernetes/pull/116649), [@saschagrunert](https://github.com/saschagrunert)) [SIG Architecture and Release]
- Updated cri-tools to v1.27.0 ([#117545](https://github.com/kubernetes/kubernetes/pull/117545), [@saschagrunert](https://github.com/saschagrunert)) [SIG Cloud Provider and Node]
- When retrieving event resources, the reportingController and reportingInstance fields in the event will contain values. ([#116506](https://github.com/kubernetes/kubernetes/pull/116506), [@HirazawaUi](https://github.com/HirazawaUi)) [SIG API Machinery and Instrumentation]
## Dependencies
### Added
- cloud.google.com/go/accessapproval: v1.6.0
- cloud.google.com/go/accesscontextmanager: v1.7.0
- cloud.google.com/go/aiplatform: v1.37.0
- cloud.google.com/go/analytics: v0.19.0
- cloud.google.com/go/apigateway: v1.5.0
- cloud.google.com/go/apigeeconnect: v1.5.0
- cloud.google.com/go/apigeeregistry: v0.6.0
- cloud.google.com/go/appengine: v1.7.1
- cloud.google.com/go/area120: v0.7.1
- cloud.google.com/go/artifactregistry: v1.13.0
- cloud.google.com/go/asset: v1.13.0
- cloud.google.com/go/assuredworkloads: v1.10.0
- cloud.google.com/go/automl: v1.12.0
- cloud.google.com/go/baremetalsolution: v0.5.0
- cloud.google.com/go/batch: v0.7.0
- cloud.google.com/go/beyondcorp: v0.5.0
- cloud.google.com/go/billing: v1.13.0
- cloud.google.com/go/binaryauthorization: v1.5.0
- cloud.google.com/go/certificatemanager: v1.6.0
- cloud.google.com/go/channel: v1.12.0
- cloud.google.com/go/cloudbuild: v1.9.0
- cloud.google.com/go/clouddms: v1.5.0
- cloud.google.com/go/cloudtasks: v1.10.0
- cloud.google.com/go/compute/metadata: v0.2.3
- cloud.google.com/go/compute: v1.19.0
- cloud.google.com/go/contactcenterinsights: v1.6.0
- cloud.google.com/go/container: v1.15.0
- cloud.google.com/go/containeranalysis: v0.9.0
- cloud.google.com/go/datacatalog: v1.13.0
- cloud.google.com/go/dataflow: v0.8.0
- cloud.google.com/go/dataform: v0.7.0
- cloud.google.com/go/datafusion: v1.6.0
- cloud.google.com/go/datalabeling: v0.7.0
- cloud.google.com/go/dataplex: v1.6.0
- cloud.google.com/go/dataproc: v1.12.0
- cloud.google.com/go/dataqna: v0.7.0
- cloud.google.com/go/datastream: v1.7.0
- cloud.google.com/go/deploy: v1.8.0
- cloud.google.com/go/dialogflow: v1.32.0
- cloud.google.com/go/dlp: v1.9.0
- cloud.google.com/go/documentai: v1.18.0
- cloud.google.com/go/domains: v0.8.0
- cloud.google.com/go/edgecontainer: v1.0.0
- cloud.google.com/go/errorreporting: v0.3.0
- cloud.google.com/go/essentialcontacts: v1.5.0
- cloud.google.com/go/eventarc: v1.11.0
- cloud.google.com/go/filestore: v1.6.0
- cloud.google.com/go/functions: v1.13.0
- cloud.google.com/go/gaming: v1.9.0
- cloud.google.com/go/gkebackup: v0.4.0
- cloud.google.com/go/gkeconnect: v0.7.0
- cloud.google.com/go/gkehub: v0.12.0
- cloud.google.com/go/gkemulticloud: v0.5.0
- cloud.google.com/go/gsuiteaddons: v1.5.0
- cloud.google.com/go/iam: v0.13.0
- cloud.google.com/go/iap: v1.7.1
- cloud.google.com/go/ids: v1.3.0
- cloud.google.com/go/iot: v1.6.0
- cloud.google.com/go/kms: v1.10.1
- cloud.google.com/go/language: v1.9.0
- cloud.google.com/go/lifesciences: v0.8.0
- cloud.google.com/go/logging: v1.7.0
- cloud.google.com/go/longrunning: v0.4.1
- cloud.google.com/go/managedidentities: v1.5.0
- cloud.google.com/go/maps: v0.7.0
- cloud.google.com/go/mediatranslation: v0.7.0
- cloud.google.com/go/memcache: v1.9.0
- cloud.google.com/go/metastore: v1.10.0
- cloud.google.com/go/monitoring: v1.13.0
- cloud.google.com/go/networkconnectivity: v1.11.0
- cloud.google.com/go/networkmanagement: v1.6.0
- cloud.google.com/go/networksecurity: v0.8.0
- cloud.google.com/go/notebooks: v1.8.0
- cloud.google.com/go/optimization: v1.3.1
- cloud.google.com/go/orchestration: v1.6.0
- cloud.google.com/go/orgpolicy: v1.10.0
- cloud.google.com/go/osconfig: v1.11.0
- cloud.google.com/go/oslogin: v1.9.0
- cloud.google.com/go/phishingprotection: v0.7.0
- cloud.google.com/go/policytroubleshooter: v1.6.0
- cloud.google.com/go/privatecatalog: v0.8.0
- cloud.google.com/go/pubsublite: v1.7.0
- cloud.google.com/go/recaptchaenterprise/v2: v2.7.0
- cloud.google.com/go/recommendationengine: v0.7.0
- cloud.google.com/go/recommender: v1.9.0
- cloud.google.com/go/redis: v1.11.0
- cloud.google.com/go/resourcemanager: v1.7.0
- cloud.google.com/go/resourcesettings: v1.5.0
- cloud.google.com/go/retail: v1.12.0
- cloud.google.com/go/run: v0.9.0
- cloud.google.com/go/scheduler: v1.9.0
- cloud.google.com/go/secretmanager: v1.10.0
- cloud.google.com/go/security: v1.13.0
- cloud.google.com/go/securitycenter: v1.19.0
- cloud.google.com/go/servicedirectory: v1.9.0
- cloud.google.com/go/shell: v1.6.0
- cloud.google.com/go/spanner: v1.45.0
- cloud.google.com/go/speech: v1.15.0
- cloud.google.com/go/storagetransfer: v1.8.0
- cloud.google.com/go/talent: v1.5.0
- cloud.google.com/go/texttospeech: v1.6.0
- cloud.google.com/go/tpu: v1.5.0
- cloud.google.com/go/trace: v1.9.0
- cloud.google.com/go/translate: v1.7.0
- cloud.google.com/go/video: v1.15.0
- cloud.google.com/go/videointelligence: v1.10.0
- cloud.google.com/go/vision/v2: v2.7.0
- cloud.google.com/go/vmmigration: v1.6.0
- cloud.google.com/go/vmwareengine: v0.3.0
- cloud.google.com/go/vpcaccess: v1.6.0
- cloud.google.com/go/webrisk: v1.8.0
- cloud.google.com/go/websecurityscanner: v1.5.0
- cloud.google.com/go/workflows: v1.10.0
- github.com/googleapis/enterprise-certificate-proxy: [v0.2.3](https://github.com/googleapis/enterprise-certificate-proxy/tree/v0.2.3)
- go.etcd.io/gofail: v0.1.0
- google.golang.org/genproto/googleapis/api: dd9d682
- google.golang.org/genproto/googleapis/rpc: 28d5490
### Changed
- cloud.google.com/go/bigquery: v1.8.0 → v1.50.0
- cloud.google.com/go/datastore: v1.1.0 → v1.11.0
- cloud.google.com/go/firestore: v1.1.0 → v1.9.0
- cloud.google.com/go/pubsub: v1.3.1 → v1.30.0
- cloud.google.com/go: v0.97.0 → v0.110.0
- github.com/Azure/azure-sdk-for-go: [v55.0.0+incompatible → v68.0.0+incompatible](https://github.com/Azure/azure-sdk-for-go/compare/v55.0.0...v68.0.0)
- github.com/Azure/go-autorest/autorest/adal: [v0.9.20 → v0.9.23](https://github.com/Azure/go-autorest/autorest/adal/compare/v0.9.20...v0.9.23)
- github.com/Azure/go-autorest/autorest/validation: [v0.1.0 → v0.3.1](https://github.com/Azure/go-autorest/autorest/validation/compare/v0.1.0...v0.3.1)
- github.com/Azure/go-autorest/autorest: [v0.11.27 → v0.11.29](https://github.com/Azure/go-autorest/autorest/compare/v0.11.27...v0.11.29)
- github.com/Microsoft/go-winio: [v0.4.17 → v0.6.0](https://github.com/Microsoft/go-winio/compare/v0.4.17...v0.6.0)
- github.com/cenkalti/backoff/v4: [v4.1.3 → v4.2.1](https://github.com/cenkalti/backoff/v4/compare/v4.1.3...v4.2.1)
- github.com/census-instrumentation/opencensus-proto: [v0.2.1 → v0.4.1](https://github.com/census-instrumentation/opencensus-proto/compare/v0.2.1...v0.4.1)
- github.com/cespare/xxhash/v2: [v2.1.2 → v2.2.0](https://github.com/cespare/xxhash/v2/compare/v2.1.2...v2.2.0)
- github.com/cilium/ebpf: [v0.7.0 → v0.9.1](https://github.com/cilium/ebpf/compare/v0.7.0...v0.9.1)
- github.com/cncf/udpa/go: [04548b0 → c52dc94](https://github.com/cncf/udpa/go/compare/04548b0...c52dc94)
- github.com/cncf/xds/go: [cb28da3 → 06c439d](https://github.com/cncf/xds/go/compare/cb28da3...06c439d)
- github.com/cockroachdb/datadriven: [bf6692d → v1.0.2](https://github.com/cockroachdb/datadriven/compare/bf6692d...v1.0.2)
- github.com/container-storage-interface/spec: [v1.7.0 → v1.8.0](https://github.com/container-storage-interface/spec/compare/v1.7.0...v1.8.0)
- github.com/containerd/cgroups: [v1.0.1 → v1.1.0](https://github.com/containerd/cgroups/compare/v1.0.1...v1.1.0)
- github.com/containerd/ttrpc: [v1.1.0 → v1.2.2](https://github.com/containerd/ttrpc/compare/v1.1.0...v1.2.2)
- github.com/coredns/caddy: [v1.1.0 → v1.1.1](https://github.com/coredns/caddy/compare/v1.1.0...v1.1.1)
- github.com/coreos/go-oidc: [v2.1.0+incompatible → v2.2.1+incompatible](https://github.com/coreos/go-oidc/compare/v2.1.0...v2.2.1)
- github.com/coreos/go-semver: [v0.3.0 → v0.3.1](https://github.com/coreos/go-semver/compare/v0.3.0...v0.3.1)
- github.com/coreos/go-systemd/v22: [v22.4.0 → v22.5.0](https://github.com/coreos/go-systemd/v22/compare/v22.4.0...v22.5.0)
- github.com/docker/distribution: [v2.8.1+incompatible → v2.8.2+incompatible](https://github.com/docker/distribution/compare/v2.8.1...v2.8.2)
- github.com/envoyproxy/go-control-plane: [49ff273 → v0.10.3](https://github.com/envoyproxy/go-control-plane/compare/49ff273...v0.10.3)
- github.com/envoyproxy/protoc-gen-validate: [v0.1.0 → v0.9.1](https://github.com/envoyproxy/protoc-gen-validate/compare/v0.1.0...v0.9.1)
- github.com/frankban/quicktest: [v1.11.3 → v1.14.0](https://github.com/frankban/quicktest/compare/v1.11.3...v1.14.0)
- github.com/fvbommel/sortorder: [v1.0.1 → v1.1.0](https://github.com/fvbommel/sortorder/compare/v1.0.1...v1.1.0)
- github.com/go-logr/logr: [v1.2.3 → v1.2.4](https://github.com/go-logr/logr/compare/v1.2.3...v1.2.4)
- github.com/go-task/slim-sprig: [348f09d → 52ccab3](https://github.com/go-task/slim-sprig/compare/348f09d...52ccab3)
- github.com/gofrs/uuid: [v4.0.0+incompatible → v4.4.0+incompatible](https://github.com/gofrs/uuid/compare/v4.0.0...v4.4.0)
- github.com/golang-jwt/jwt/v4: [v4.4.2 → v4.5.0](https://github.com/golang-jwt/jwt/v4/compare/v4.4.2...v4.5.0)
- github.com/google/gofuzz: [v1.1.0 → v1.2.0](https://github.com/google/gofuzz/compare/v1.1.0...v1.2.0)
- github.com/googleapis/gax-go/v2: [v2.1.1 → v2.7.1](https://github.com/googleapis/gax-go/v2/compare/v2.1.1...v2.7.1)
- github.com/inconshreveable/mousetrap: [v1.0.1 → v1.1.0](https://github.com/inconshreveable/mousetrap/compare/v1.0.1...v1.1.0)
- github.com/mitchellh/go-wordwrap: [v1.0.0 → v1.0.1](https://github.com/mitchellh/go-wordwrap/compare/v1.0.0...v1.0.1)
- github.com/onsi/ginkgo/v2: [v2.9.1 → v2.9.4](https://github.com/onsi/ginkgo/v2/compare/v2.9.1...v2.9.4)
- github.com/onsi/gomega: [v1.27.4 → v1.27.6](https://github.com/onsi/gomega/compare/v1.27.4...v1.27.6)
- github.com/opencontainers/runc: [v1.1.4 → v1.1.7](https://github.com/opencontainers/runc/compare/v1.1.4...v1.1.7)
- github.com/rogpeppe/go-internal: [v1.10.0 → v1.6.1](https://github.com/rogpeppe/go-internal/compare/v1.10.0...v1.6.1)
- github.com/seccomp/libseccomp-golang: [f33da4d → v0.10.0](https://github.com/seccomp/libseccomp-golang/compare/f33da4d...v0.10.0)
- github.com/spf13/cobra: [v1.6.0 → v1.7.0](https://github.com/spf13/cobra/compare/v1.6.0...v1.7.0)
- github.com/stretchr/testify: [v1.8.1 → v1.8.2](https://github.com/stretchr/testify/compare/v1.8.1...v1.8.2)
- github.com/vishvananda/netns: [v0.0.2 → v0.0.4](https://github.com/vishvananda/netns/compare/v0.0.2...v0.0.4)
- github.com/xlab/treeprint: [v1.1.0 → v1.2.0](https://github.com/xlab/treeprint/compare/v1.1.0...v1.2.0)
- go.etcd.io/bbolt: v1.3.6 → v1.3.7
- go.etcd.io/etcd/api/v3: v3.5.7 → v3.5.9
- go.etcd.io/etcd/client/pkg/v3: v3.5.7 → v3.5.9
- go.etcd.io/etcd/client/v2: v2.305.7 → v2.305.9
- go.etcd.io/etcd/client/v3: v3.5.7 → v3.5.9
- go.etcd.io/etcd/pkg/v3: v3.5.7 → v3.5.9
- go.etcd.io/etcd/raft/v3: v3.5.7 → v3.5.9
- go.etcd.io/etcd/server/v3: v3.5.7 → v3.5.9
- go.opencensus.io: v0.23.0 → v0.24.0
- go.uber.org/atomic: v1.7.0 → v1.10.0
- go.uber.org/multierr: v1.6.0 → v1.11.0
- golang.org/x/crypto: v0.1.0 → v0.6.0
- golang.org/x/mod: v0.9.0 → v0.10.0
- golang.org/x/net: v0.8.0 → v0.9.0
- golang.org/x/oauth2: ee48083 → v0.6.0
- golang.org/x/sys: v0.6.0 → v0.7.0
- golang.org/x/term: v0.6.0 → v0.7.0
- golang.org/x/text: v0.8.0 → v0.9.0
- golang.org/x/time: 90d013b → v0.3.0
- golang.org/x/tools: v0.7.0 → v0.8.0
- google.golang.org/api: v0.60.0 → v0.114.0
- google.golang.org/genproto: c8bf987 → 0005af6
- google.golang.org/grpc: v1.51.0 → v1.54.0
- google.golang.org/protobuf: v1.28.1 → v1.30.0
- gopkg.in/gcfg.v1: v1.2.0 → v1.2.3
- gopkg.in/natefinch/lumberjack.v2: v2.0.0 → v2.2.1
- gopkg.in/warnings.v0: v0.1.1 → v0.1.2
- k8s.io/klog/v2: v2.90.1 → v2.100.1
- k8s.io/kube-openapi: 15aac26 → 7828149
- k8s.io/utils: a36077c → d93618c
- sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.1.1 → v0.1.2
### Removed
- github.com/certifi/gocertifi: [2c3bb06](https://github.com/certifi/gocertifi/tree/2c3bb06)
- github.com/cockroachdb/errors: [v1.2.4](https://github.com/cockroachdb/errors/tree/v1.2.4)
- github.com/cockroachdb/logtags: [eb05cc2](https://github.com/cockroachdb/logtags/tree/eb05cc2)
- github.com/getsentry/raven-go: [v0.2.0](https://github.com/getsentry/raven-go/tree/v0.2.0)
Codebase Browser
kubernetes