With the recent drama of GitLab, both with the CI/CD changes and then more recent possible threat of deleting old repositories, I continue my migration to a local Gitea instance, https://src.mfgames.com/ for the bulk of my code and writing.
For the most part, migrating is just a matter of shuffling data. I have a lot of repositories, both active and inactive, and it will take me months to move them over. Plus I haven't decided if I'm going to purge them from my GitLab account so there is a single source of truth or just mirror back to them.
Currently, the most difficult task was figuring out how to handle the build processing. I've mentioned previously that I use Conventional Commits and Semantic Release fairly heavily. I've branched out a little from there using Lefthook and my project layout.
Currently, the CI does the following:
- Build the project
- Test various conditions including valid commit messages
- If the commits indicate a new build:
- Tag it
- Build the release version
- Create a release on Gitea
This changes over time, but it is the basic pattern.
Tags and Git Depth
Woodpecker does not automatically download the needed tags for
semantic-release (and GitVersion). This means that the
.woodpecker.yml file needs to include tags.
clone: git: image: woodpeckerci/plugin-git settings: tags: true pipeline: # The pipeline elements
Unike GitLab, which only limits to the last ten commits, it appears that Woodpecker downloads the full repository by default which is also needed by GitVersion because it calculates every version. Not entirely sure about
semantic-release logging indicates it doesn't need the full repository, just enough back to find a version.
Building and Testing
To support task branches, I have a basic build and test code that runs on pushes and pull requests. This lets me identify bugs earlier and catch typos with my commits.
build: image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest commands: - nix develop --command scripts/build.sh when: # We need both "tag" for the next section. event: [push, pull_request, tag] tag: v* test: image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest commands: - nix develop --command scripts/test.sh when: event: [push, pull_request]
From the build tasks, you can see that I'm using my current project layout which uses scripts in the
scripts/ folder instead of
npm run or
dotnet run. This is to make it easier to work with polyglot plus works around the issue that I need to use
nix develop to get into my reproducible environment since the Docker image doesn't automatically do that. This is because both Gitlab and Woodpecker use the image which bypasses initialization files and I couldn't have it run
direnv allow automatically to set up environment variables.
One thing that is missing is that Woodpecker doesn't have a clean mechanism for temporary build artifacts. I can't upload the build files and then download them so I can see the final results. Instead, I have to script it out or use a S3 plugin.
Building on Versions
With most cases, I build the release version of the project when the conventional commits indicate that there is a new version (
fix). This is an additional pipeline that comes after the
release-main: image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest commands: - export DRONE="true" # Required to convince `env-ci` # semantic-release needs this locally - git branch $DRONE_BRANCH origin/$DRONE_BRANCH - nix develop --command scripts/release.sh secrets: - gitea_token - git_credentials when: event: push branch: main
There are a number of things in this block that took me a while. The first is the
branch. We only do releases on the
main (I'm still moving away from
master as racist language).
The second is the
export DRONE line. At the time I set this up, env-ci wasn't aware of Woodpecker, but it was recently added thanks to 6543 on the Woodpecker Matrix channel,
#woodpecker-ci:matrix.org. I don't know when the latest
semantic-release will have it, but it shouldn't be needed soon, if not already.
The third is the
git branch line in the above script. Woodpecker creates a detached head, as does Gitlab. But when it doesn't do is also create a local branch for the one being created. This causes a problem because the release process appears to “jump” to the branch to figure out the changes between the detached head (the commit being built) and the actual branch.
Finally, we have the secrets.
semantic-release automatically picks up $GITEA_TOKEN for the release process but also needs $GIT_CREDENTIALS to verify Git access.
The token is easy, that is what given by Gitea for the user.
GIT_CREDENTIALS is slightly harder, it is a colon-separate tuple of the user name and the Gitea access token. From observations, the Gitea is basically just a bunch of URL-safe characters, so the URL-escaping isn't needed in my case (you need to URL-escape the left and right of the colon but not the colon itself).
$ export GITEA_TOKEN=9fc6d72c72e4b149f07491a0b2d3ec9215d57caf $ export GIT_CREDENTIALS="dmoonfire:$GITEA_TOKEN"
These need to be set on a per-project basis since Woodpecker, unlike GitLab and sourcehut, there doesn't appear to be a good way of having a shared set of secrets for projects (GitLab has organization/group level secrets, sourcehut has the secret storage). This means I have to set the same GITEA_TOKEN and GIT_CREDENTIALS for all 80+ of my Fedran repositories†.
† Woodpecker has a CLI,
woodpecker-cli which will let me automate that. I will use that.
Create Release on Tag
One thing I'm moving toward is creating a release entry on the forge. Since this happens after the build process and Woodpecker uses a different Docker image, I need it to be a post-release event so I hang it off the tagging process instead of the push to
6543 came to my rescue again with this one, so that is why there are those two “tag” elements in the script above. I also have a new stanza for the release process:
release-gitea: image: plugins/gitea-release settings: base_url: https://src.mfgames.com files: - "*.pdf" - "*.epub" api_key: from_secret: gitea_token when: event: tag tag: v*
If I didn't have the
tag: in the
build: stanza, it wasn't working for the tags. This caused me some difficulties because I usually treat hashes as being unordered, but Woodpecker uses file order for processing pipelines. So, I needed to have the
build: target build the file (with the correct version because it was tagged) and then
release-gitea: to use that output for the release process. The
release-main: are skipped because they don't have those events listed.
In addition, secrets are handled differently when done as a parameter for a plugin. That is why I have the
from_secret: element in the above script. This inconsistency threw me for a few days.
Putting it Together
If you want to see the final version, check out this example which has my current version as a single file.
I'm happy to move over to Woodpecker (you know, except for the cost of hosting) both because of the control and the challenge. I also don't have a need for speed, so if it takes a while to get through the queue, I'm okay.