Packaging: reimplement packagecloud release number script as a pants rule

[cognifloyd]

This PR is working towards doing packaging via pantsbuild. Eventually, I hope to archive and stop using st2-packages.git.

One of the key parts of packaging is assigning a release number for a given build. The only way to do that is to query packagecloud to see what the last release was and add 1 to that. We use a shell script to do that in st2-packages.git:
https://github.com/StackStorm/st2-packages/blob/master/.circle/packagecloud.sh

In this PR I rewrote that shell script in python in pants-plugins/release as pantsbuild @rules. Then this builds on #6321 by Injecting the release number from packagecloud as the release field on nfpm_deb_package and nfpm_rpm_package targets.

Comparison of old packagecloud shell script with the python functions in this PR

latest_revision is the primary function for looking up the release number in the old script. This function calls: get_repo_name, get_pkg_os, and get_revision (which calls get_versions_url).

The old script recalculates PKG_IS_UNSTABLE in several places. In this PR, it is calculated only once in the @rule inject_nfpm_fields:

st2/pants-plugins/release/rules.py

Lines 232 to 237 in 0df575c

	version: str = extracted_version.value
	is_dev = "dev" in version
	if is_dev and "-dev" not in version:
	# nfpm parses this into version[-version_prerelease][+version_metadata]
	# that dash is required to be a valid semver version (3.9dev => 3.9-dev).
	version = version.replace("dev", "-dev")

To replace the get_repo_name function, I used a simple f-string (we do not use enterprise repos any more, so I didn't bother to copy that):

st2/pants-plugins/release/packagecloud_rules.py

Line 123 in 0df575c

repo = f"{'' if request.production else 'staging-'}{'unstable' if pkg_is_unstable else 'stable'}"

To replace the get_pkg_os function, I used some dictionaries instead of functions to link an os-release to its package type. These include all OS-versions that have release in packagecloud (It should always include that so we can look up old versions if we need to), as well as a few more that we will or hope to support. In particular, these dictionaries:

• DISTROS_BY_PKG_TYPE which has {pkg_type: {distro: {distro_id: distro_version}}}. Here pkg_type is deb or rpm; distro is the OS (debian, ubuntu, el); distro_id is the name we use internally to identify a particular os-version (el8, el9, focal, jammy, etc); and distro_version is an actual version number.
  

  st2/pants-plugins/release/packagecloud_rules.py

  Line 38 in 0df575c

  DISTROS_BY_PKG_TYPE = { # {pkg_type: {distro: {distro_id: distro_version}}}

• DISTRO_INFO is a dictionary comprehension that reorganizes the metadata in DISTROS_BY_PKG_TYPE to make it easier to look it up using our internal distro_id name.
  

  st2/pants-plugins/release/packagecloud_rules.py

  Lines 65 to 74 in 0df575c

  	DISTRO_INFO = {
  	distro_id: {
  	"distro": distro,
  	"version": distro_version,
  	"pkg_type": pkg_type,
  	}
  	for pkg_type, distros in DISTROS_BY_PKG_TYPE.items()
  	for distro, distro_ids in distros.items()
  	for distro_id, distro_version in distro_ids.items()
  	}

  
  DISTRO_INFO is used in the @rule packagecloud_get_next_release here:
  

  st2/pants-plugins/release/packagecloud_rules.py

  Lines 117 to 118 in 0df575c

  	distro_id = request.distro_id
  	distro_info = DISTRO_INFO[distro_id]

The get_revision function calls get_versions_url which is reimplemented with an f-string and a requests session (with some additional safety checks around the API call):

st2/pants-plugins/release/packagecloud_rules.py

Lines 130 to 133 in 0df575c

	# https://packagecloud.io/docs/api#resource_packages_method_index (api doc incorrectly drops /:package)
	# /api/v1/repos/:user_id/:repo/packages/:type/:distro/:version/:package/:arch.json
	index_url = f"/api/v1/repos/{org}/{repo}/packages/{pkg_type}/{distro}/{distro_version}/{pkg_name}/{arch}.json"
	package_index: list[dict[str, Any]] = get(index_url)

Finally, we reimplement the packcloud API calls in the get_revision function (with some additional safety checks around the API call) in the @rule packagecloud_get_next_release.

st2/pants-plugins/release/packagecloud_rules.py

Lines 137 to 149 in 0df575c

	versions_url: str = package_index[0]["versions_url"]
	versions: list[dict[str, Any]] = get(versions_url)
	releases = [
	version_info["release"]
	for version_info in versions
	if version_info["version"] == request.package_version
	]
	if not releases:
	return PackageCloudNextRelease()
	max_release = max(int(release) for release in releases)
	next_release = max_release + 1
	return PackageCloudNextRelease(next_release)

Additional safety checks around API calls and paging in those API calls are implemented here:

st2/pants-plugins/release/packagecloud_rules.py

Lines 105 to 115 in 0df575c

	def get(url_path: str) -> list[dict[str, Any]]:
	response = client.get(f"https://packagecloud.io{url_path}")
	response.raise_for_status()
	ret: list[dict[str, Any]] = response.json()
	next_url = response.links.get("next", {}).get("url")
	while next_url:
	response = client.get(f"https://packagecloud.io{next_url}")
	response.raise_for_status()
	ret.extend(response.json())
	next_url = response.links.get("next", {}).get("url")
	return ret

Overview of changes to pants-plugins/release

This uses requests to make packagecloud API calls, so tell pants about our plugin's requirement by adding a requirements.txt file in the plugin directory (pants-plugins/release/requirements.txt):

st2/pants-plugins/release/requirements.txt

Line 1 in 0df575c

requests

Then add that requirements.txt file in this BUILD file entry so we can regenerate lockfiles/pants-plugins.lock:

st2/pants-plugins/release/BUILD

Lines 7 to 9 in 0df575c

	python_requirements(
	name="reqs",
	)

And implement the packagecloud release lookup in the @rule packagecloud_get_next_release here (Note that we use @_uncacheable_rule to make sure this gets recalculated every time we build a package. It is uncacheable since we depend on state in an external system.):

st2/pants-plugins/release/packagecloud_rules.py

Lines 91 to 94 in 0df575c

	@_uncacheable_rule
	async def packagecloud_get_next_release(
	request: PackageCloudNextReleaseRequest,
	) -> PackageCloudNextRelease:

Then we call the @rule packagecloud_get_next_release here (Note the TODO: add field for distro ID will be addressed in a follow-up PR after this PR gets merged.):

st2/pants-plugins/release/rules.py

Lines 239 to 249 in 0df575c

	# this is specific to distro-version (EL8, EL9, Ubuntu Focal, Ubuntu Jammy, ...)
	next_release = await packagecloud_get_next_release(
	PackageCloudNextReleaseRequest(
	nfpm_arch=target[NfpmArchField].value,
	distro_id="", # TODO: add field for distro ID
	package_name=target[NfpmPackageNameField].value,
	package_version=version,
	production=not is_dev,
	)
	)
	release = 1 if next_release.value is None else next_release.value

This rule has to be registered with pants, so do that here:

st2/pants-plugins/release/rules.py

Lines 259 to 261 in 0df575c

	def rules():
	return [
	*packagecloud_rules(),

And finally, we inject the nfpm release field into nfpm_deb_package and nfpm_rpm_package targets here:

st2/pants-plugins/release/rules.py

Line 254 in 0df575c

NfpmVersionReleaseField(release, address=address),