Lefthook2024-03-28T17:39:17Zhttps://d.moonfire.us/tags/lefthook/D. MoonfireCreative Commons Attribution-NonCommercial-ShareAlike 4.0 InternationalSemantic Release and Woodpecker CI2022-08-07T05:00:00Zhttps://d.moonfire.us/blog/2022/08/07/semantic-release-and-woodpecker-ci/In my migration from GitLab to Gitea, I've started moving my CI/CD server over to Woodpecker. Here is some of the struggles I've done through in the process of getting it to work.
<p>With the recent drama of <a href="/tags/gitlab/">GitLab</a>, both with the CI/CD changes and then more recent possible threat of deleting old repositories, I continue my migration to a local <a href="/tags/gitea/">Gitea</a> instance, <a href="https://src.mfgames.com/">https://src.mfgames.com/</a> for the bulk of my code and writing.</p>
<p>For the most part, migrating is just a matter of shuffling data. I have a <em>lot</em> of repositories, both active and inactive, and it will take me months to move them over. Plus I haven't decided if I'm going to purge them from my GitLab account so there is a single source of truth or just mirror back to them.</p>
<p>Currently, the most difficult task was figuring out how to handle the build processing. I've mentioned previously that I use <a href="/tags/conventional-commits/">Conventional Commits</a> and <a href="/tags/semantic-release/">Semantic Release</a> fairly heavily. I've branched out a little from there using <a href="/tags/lefthook/">Lefthook</a> and my <a href="/garden/project-layout/">project layout</a>.</p>
<p>Currently, the CI does the following:</p>
<ol>
<li>Build the project</li>
<li>Test various conditions including valid commit messages</li>
<li>If the commits indicate a new build:
<ol>
<li>Tag it</li>
<li>Build the release version</li>
<li>Create a release on Gitea</li>
</ol>
</li>
</ol>
<p>This changes over time, but it is the basic pattern.</p>
<h1>Tags and Git Depth</h1>
<p>Woodpecker does not automatically download the needed tags for <code>semantic-release</code> (and <a href="/tags/gitversion/">GitVersion</a>). This means that the <code>.woodpecker.yml</code> file needs to include tags.</p>
<pre><code class="language-yaml">clone:
git:
image: woodpeckerci/plugin-git
settings:
tags: true
pipeline:
# The pipeline elements
</code></pre>
<p>Unike GitLab, which only limits to the last ten commits, it appears that Woodpecker <a href="https://woodpecker-ci.org/plugins/plugin-git">downloads the full repository</a> by default which is also needed by GitVersion because it calculates every version. Not entirely sure about <code>semantic-release</code> logging indicates it doesn't need the full repository, just enough back to find a version.</p>
<h1>Building and Testing</h1>
<p>To support task branches, I have a basic build and test code that runs on pushes and pull requests. This lets me identify bugs earlier and catch typos with my commits.</p>
<pre><code class="language-yaml">build:
image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
commands:
- nix develop --command scripts/build.sh
when:
# We need both "tag" for the next section.
event: [push, pull_request, tag]
tag: v*
test:
image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
commands:
- nix develop --command scripts/test.sh
when:
event: [push, pull_request]
</code></pre>
<p>From the build tasks, you can see that I'm using my current project layout which uses scripts in the <code>scripts/</code> folder instead of <code>npm run</code> or <code>dotnet run</code>. This is to make it easier to work with polyglot plus works around the issue that I need to use <code>nix develop</code> to get into my reproducible environment since the Docker image doesn't automatically do that. This is because both Gitlab and Woodpecker use the image which bypasses initialization files and I couldn't have it run <code>direnv allow</code> automatically to set up environment variables.</p>
<p>One thing that is missing is that Woodpecker doesn't have a clean mechanism for temporary build artifacts. I can't upload the build files and then download them so I can see the final results. Instead, I have to script it out or use a S3 plugin.</p>
<h1>Building on Versions</h1>
<p>With most cases, I build the release version of the project when the conventional commits indicate that there is a new version (<code>feat</code> and <code>fix</code>). This is an additional pipeline that comes after the <code>test:</code> line.</p>
<pre><code class="language-yaml">release-main:
image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
commands:
- export DRONE="true" # Required to convince `env-ci`
# semantic-release needs this locally
- git branch $DRONE_BRANCH origin/$DRONE_BRANCH
- nix develop --command scripts/release.sh
secrets:
- gitea_token
- git_credentials
when:
event: push
branch: main
</code></pre>
<p>There are a number of things in this block that took me a while. The first is the <code>event</code> and <code>branch</code>. We only do releases on the <code>main</code> (I'm still moving away from <code>master</code> as racist language).</p>
<p>The second is the <code>export DRONE</code> line. At the time I set this up, <a href="https://www.npmjs.com/package/env-ci">env-ci</a> wasn't aware of Woodpecker, but it was <a href="https://github.com/woodpecker-ci/woodpecker/pull/1035">recently added</a> thanks to 6543 on the Woodpecker Matrix channel, <code>#woodpecker-ci:matrix.org</code>. I don't know when the latest <code>semantic-release</code> will have it, but it shouldn't be needed soon, if not already.</p>
<p>The third is the <code>git branch</code> line in the above script. Woodpecker creates a detached head, as does Gitlab. But when it doesn't do is also create a local branch for the one being created. This causes a problem because the release process appears to “jump” to the branch to figure out the changes between the detached head (the commit being built) and the actual branch.</p>
<p>Finally, we have the secrets. <code>semantic-release</code> automatically picks up $GITEA_TOKEN for the release process but also needs $GIT_CREDENTIALS to verify Git access.</p>
<p>The token is easy, that is what given by Gitea for the user.</p>
<p><a href="https://github.com/semantic-release/semantic-release/blob/master/docs/usage/ci-configuration.md">GIT_CREDENTIALS</a> is slightly harder, it is a colon-separate tuple of the user name and the Gitea access token. From observations, the Gitea is basically just a bunch of URL-safe characters, so the URL-escaping isn't needed in my case (you need to URL-escape the left and right of the colon but not the colon itself).</p>
<pre><code class="language-shell">$ export GITEA_TOKEN=9fc6d72c72e4b149f07491a0b2d3ec9215d57caf
$ export GIT_CREDENTIALS="dmoonfire:$GITEA_TOKEN"
</code></pre>
<p>These need to be set on a per-project basis since Woodpecker, unlike GitLab and <a href="/tags/sourcehut/">sourcehut</a>, there doesn't appear to be a good way of having a shared set of secrets for projects (GitLab has organization/group level secrets, sourcehut has the secret storage). This means I have to set the same GITEA_TOKEN and GIT_CREDENTIALS for all 80+ of my <a href="/tags/fedran/">Fedran</a> repositories†.</p>
<p>† Woodpecker has a CLI, <code>woodpecker-cli</code> which will let me automate that. I will use that.</p>
<h1>Create Release on Tag</h1>
<p>One thing I'm moving toward is creating a release entry on the forge. Since this happens after the build process and Woodpecker uses a different Docker image, I need it to be a post-release event so I hang it off the tagging process instead of the push to <code>main</code>.</p>
<p>6543 came to my rescue again with this one, so that is why there are those two “tag” elements in the script above. I also have a new stanza for the release process:</p>
<pre><code class="language-yaml">release-gitea:
image: plugins/gitea-release
settings:
base_url: https://src.mfgames.com
files:
- "*.pdf"
- "*.epub"
api_key:
from_secret: gitea_token
when:
event: tag
tag: v*
</code></pre>
<p>If I didn't have the <code>event:</code> and <code>tag:</code> in the <code>build:</code> stanza, it wasn't working for the tags. This caused me some difficulties because I usually treat hashes as being unordered, but Woodpecker uses file order for processing pipelines. So, I needed to have the <code>build:</code> target build the file (with the correct version because it was tagged) and then <code>release-gitea:</code> to use that output for the release process. The <code>test:</code> and <code>release-main:</code> are skipped because they don't have those events listed.</p>
<p>In addition, secrets are handled differently when done as a parameter for a plugin. That is why I have the <code>from_secret:</code> element in the above script. This inconsistency threw me for a few days.</p>
<h1>Putting it Together</h1>
<p>If you want to see the final version, check out <a href="https://src.mfgames.com/dmoonfire-garden/project-layout/src/branch/main/.woodpecker.yml">this example</a> which has my current version as a single file.</p>
<h1>Conclusion</h1>
<p>I'm happy to move over to Woodpecker (you know, except for the cost of hosting) both because of the control and the challenge. I also don't have a need for speed, so if it takes a while to get through the queue, I'm okay.</p>