feat: update v1beta2

[jschoone]

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squash commits
• include documentation
• add unit tests

[garloff]

Well, this is scs3 then, because we change all the variable names again.
"Old scs2" and "new scs2" is just trying to maximize customer confusion.
I honestly would hate this as customer.

Question:
Can we keep variable names etc. stable with going to v1beta2 or do we need to change anyways? In the latter case, I'd say let's call it scs3.
Otherwise let's avoid another round of variable renames.
There is really only one variable missing from scs2 in my practical usage of the stack: serviceLoadBalancer. But adding things is not a problem (as long as the defaults are in line with old behavior).

[garloff]

Is this the rootDisk for the workers? That would make sense then: SCS-4V-8 plus a 50GB cinder disk.
If it is for the controlPlane, then the example is a very bad idea: We have the flavor SCS-2V-4-20s which has a fast local SSD/NVMe which costs a bit more than a 20GB cinder disk (in most clouds), but has the big advantage of low write latency and thus a stable etcd cluster. If you now attach a 50GB cinder root disk, you get the worst of both worlds: You pay for a local SSD/NVMe which you can't even see and your etcd is not stable :-(
We had the variables controlPlaneFlavor, controlPlaneRootDisk and workerFlavor and workerRootDisk in scs2 clusterClass. I would expect to see these variables here ...

[jschoone]

It's for all machines and can/should be configured for different control planes and machine deployments using variables.override.
This allows us to have the same variables for clusters with a hosted control plane where controlPlane* variables make no sense

[garloff]

OK, so we need scs3 if we change the variable names again.
Why oh why?
Also, I consider it a regression.
I regularly chose SCS-2V-4-20s flavors for the control plane, (no rootDisk) and a flavor without root disk (and potentially more vCPUs/RAM) for workers.

[jschoone]

This allows us to have the same variables for clusters with a hosted control plane where controlPlane* variables make no sense.
Variables can be overridden per machineDeployment and control plane

[garloff]

Sorry for my rant, but changing the clusterClass variables all the time looks like a very confused way of doing product management for cluster stacks. Communicating to users that you don't care about their pain for handling spurious changes all the time. Trying to get a Lennart-medal? Hiding it behind "old scs2" and "new scs2" adds insult to injury.
If the conversion to v1beta2 makes this hard to avoid, then let's use the opportunity to design a stable setup for the variables that we hopefully can keep stable for more than half a year and call this scs3. I would also object to merging flavor and rootDisk for workers and controlPlane. They have very different requirements.

[jschoone]

Sorry for my rant, but changing the clusterClass variables all the time looks like a very confused way of doing product management for cluster stacks. Communicating to users that you don't care about their pain for handling spurious changes all the time. Trying to get a Lennart-medal? Hiding it behind "old scs2" and "new scs2" adds insult to injury. If the conversion to v1beta2 makes this hard to avoid, then let's use the opportunity to design a stable setup for the variables that we hopefully can keep stable for more than half a year and call this scs3. I would also object to merging flavor and rootDisk for workers and controlPlane. They have very different requirements.

This must be a new Cluster Stack anyway as we said for v1beta2 and the changes we want to implement for the Workload Clusters default loadbalancer configuration.
Sorry for not marking it as Draft, I did it now.

[garloff]

OK, good, so let's work on a good scs3 clusterStack with v1beta2 then.

[jschoone]

There were just many discussions on the configuration at the hackathon which are not yet visible in this PR.
We already reviewed it together and it will change even more.
To avoid confusion I will find a working title for the new Cluster Stack.

Well, this is scs3 then, because we change all the variable names again. "Old scs2" and "new scs2" is just trying to maximize customer confusion. I honestly would hate this as customer.

Question: Can we keep variable names etc. stable with going to v1beta2 or do we need to change anyways? In the latter case, I'd say let's call it scs3. Otherwise let's avoid another round of variable renames. There is really only one variable missing from scs2 in my practical usage of the stack: serviceLoadBalancer. But adding things is not a problem (as long as the defaults are in line with old behavior).

Basically adding variables breaks the idea of Cluster Stacks the same since the description does not match anymore, but can be mentioned in the docs, with "available from..." or something

[Nils98Ar]

@jschoone @garloff @janiskemper

Side note: The cso validating webhook does not allow changing the ClusterStack of an existing cluster (e.g. scs -> scs2) and therefore needs to be disabled for cluster resources during the upgrade.

[janiskemper]

In general, my very strong opinion is that if we want people to upgrade their clusters from one version of a cluster stack to another, then it must be the same cluster stack.

I don't see any reason in creating different cluster stacks, if we want people to upgrade their clusters from one to another.

The only reason for me to create a new cluster stack, is if we want to maintain them in parallel because they both have valid, but different configuration.

If we introduce breaking changes in the configuration, then we need to think about upgrade paths (if we want people to upgrade) independently of the name of the cluster stack.

[Nils98Ar]

@janiskemper Would it be possible in general to use SemVer for ClusterStacks, for example, to distinguish between patch, minor, and major updates with breaking changes?

[janiskemper]

not right now. But I don't think it is necessary. How we do it at Syself is that we use the Kubernetes minor versions to introduce some new stuff. Within one Kubernetes minor version we stick to small changes and patches, while with the new minor versions we introduce more things.

We haven't actually had breaking changes so far, but if they are necessary, and people should still be able to upgrade their clusters, I suggest to introduce these breaking changes with a new Kubernetes minor version.

In this way, there is no need for semver versioning. This is in fact the reason why we didn't introduce it, because the versioning based on the Kubernetes version already gives you a new "major" version every few months (based on e.g. "openstack-scs-1-34-v1"

[jschoone]

not right now. But I don't think it is necessary. How we do it at Syself is that we use the Kubernetes minor versions to introduce some new stuff. Within one Kubernetes minor version we stick to small changes and patches, while with the new minor versions we introduce more things.

We haven't actually had breaking changes so far, but if they are necessary, and people should still be able to upgrade their clusters, I suggest to introduce these breaking changes with a new Kubernetes minor version.

In this way, there is no need for semver versioning. This is in fact the reason why we didn't introduce it, because the versioning based on the Kubernetes version already gives you a new "major" version every few months (based on e.g. "openstack-scs-1-34-v1"

Ah, so if we follow that pattern we need the one dir per Kubernetes version or at least per all Kubernetes versions with the same features. Could work, probably not more or less messy than now :D
Let's see how this could look like

[janiskemper]

we do in fact have on directory per Kubernetes minor version!

[jschoone]

we do in fact have on directory per Kubernetes minor version!

Yes I know, in this repo we replaced it because of code repetition, because we always had the idea of dealing with breaking changes in different cluster stacks not their minor versions, where the subdirectories make much more sense now. I think that can really solve the problems we now have, even it still looks wrong to have the version directories in a git repo :D
I'll add a possible structure soon