A few months ago, I found myself hacking chunks out of a monolithic Python repo to break out a few generic reusable packages. Splitting out the libraries had a number of benefits beyond being able to reuse the code in other places: it gave a much clearer picture of the test coverage, provided a chance to ditch some vestigial code, as well as some impetus to pay off roll over some old technical debt. But one area that gave me a few headaches was getting the packages to play nicely with existing CI/CD pipelines.

I tend to rely on Poetry for managing requirements, even when I’m not explicitly building packages. Whilst adding the new packages to the pyproject.toml was simple enough for local development, I ran into some permissions issues. Originally I used a trick described by Edmundo Sanchez here to modify the source of custom packages as part of the workflow, but unfortunately it seems to break with more recent versions of Poetry, complaining quite reasonably that the poetry.lock file is out of date.

So rather than find another workaround to fix this already somewhat hacky workaround, I decided it was time for a proper fix. The packages belong in a (private) registry where they can be installed irrespective of source. Here are the two basic steps.

Building Packages

Getting the packages built and added to their own registries was fairly straightforward, basically adding a new version on tagged commits with something like the following:1

build:
  stage: build
    image: python:3.11
    rules:
      - if: $CI_COMMIT_TAG
    script:
      - poetry config repositories.gitlab ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi
      - poetry build
      - poetry publish --repository gitlab -u gitlab-ci-token -p ${CI_JOB_TOKEN}
YAML

Installing Packages

In the tools actually consuming these packages, the additional registries are then added as extra Poetry sources, explicitly adding PyPI as the primary source for all other packages:

[tool.poetry.dependencies]
python = "^3.11"
mypackage = { version = "^0.1.0", source = "gitlab" }

[[tool.poetry.source]]
name = "PyPI"
priority = "primary"

[[tool.poetry.source]]
name = "gitlab"
url = "https://<gitlab_url>/api/v4/projects/<project_id>/packages/pypi/simple"
priority = "supplemental"
TOML

Giving the packages access to one another in Gitlab was then just a case of providing them access via Settings → CI/CD → Token Access → Limit access to this project → Groups and projects with access.

However, installing packages locally proved to be a pain to set up. It should simply have been a case of creating a Personal Access Token and then adding the relevant credentials to Poetry:

poetry config http-basic.gitlab <username> <password>
TOML

Following several comments online, I tried various combinations of username, PAT name as username, blank username etc. which all failed, despite being able to install the package via pip as expected:

pip install -i https://<username:password>@<gitlab_url>/pypi/simple/mypackage
TOML

Ultimately the problem turned out to be my PAT permissions. While read_api is the only scope required, it appears that the default Guest role provides insufficient permissions:

On self-managed GitLab instances, users with the Guest role are able to perform this action only on public and internal projects (not on private projects). External users must be given explicit access (at least the Reporter role) even if the project is internal.2

Creating a new PAT with the Reporter role was all it took to get the pipelines pumping again, ensuring local dev and online tests are all singing from the same codesheet. Now I just need to turn all them tests green!

  1. A more in-depth workflow might also add automated version bumping and release management. ↩︎
  2. Apparently reading the docs can reveal the answer. ↩︎