Semantic Release

Semantic Release and Woodpecker CI

2022-08-07T05:00:00Z

With the recent drama of GitLab, both with the CI/CD changes and then more recent possible threat of deleting old repositories, I continue my migration to a local Gitea instance, https://src.mfgames.com/ for the bulk of my code and writing.

For the most part, migrating is just a matter of shuffling data. I have a lot of repositories, both active and inactive, and it will take me months to move them over. Plus I haven't decided if I'm going to purge them from my GitLab account so there is a single source of truth or just mirror back to them.

Currently, the most difficult task was figuring out how to handle the build processing. I've mentioned previously that I use Conventional Commits and Semantic Release fairly heavily. I've branched out a little from there using Lefthook and my project layout.

Currently, the CI does the following:

Build the project
Test various conditions including valid commit messages
If the commits indicate a new build:
1. Tag it
2. Build the release version
3. Create a release on Gitea

This changes over time, but it is the basic pattern.

Tags and Git Depth

Woodpecker does not automatically download the needed tags for semantic-release (and GitVersion). This means that the .woodpecker.yml file needs to include tags.

clone:
    git:
        image: woodpeckerci/plugin-git
        settings:
            tags: true
pipeline:
    # The pipeline elements

Unike GitLab, which only limits to the last ten commits, it appears that Woodpecker downloads the full repository by default which is also needed by GitVersion because it calculates every version. Not entirely sure about semantic-release logging indicates it doesn't need the full repository, just enough back to find a version.

Building and Testing

To support task branches, I have a basic build and test code that runs on pushes and pull requests. This lets me identify bugs earlier and catch typos with my commits.

build:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - nix develop --command scripts/build.sh
    when:
        # We need both "tag" for the next section.
        event: [push, pull_request, tag]
        tag: v*

test:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - nix develop --command scripts/test.sh
    when:
        event: [push, pull_request]

From the build tasks, you can see that I'm using my current project layout which uses scripts in the scripts/ folder instead of npm run or dotnet run. This is to make it easier to work with polyglot plus works around the issue that I need to use nix develop to get into my reproducible environment since the Docker image doesn't automatically do that. This is because both Gitlab and Woodpecker use the image which bypasses initialization files and I couldn't have it run direnv allow automatically to set up environment variables.

One thing that is missing is that Woodpecker doesn't have a clean mechanism for temporary build artifacts. I can't upload the build files and then download them so I can see the final results. Instead, I have to script it out or use a S3 plugin.

Building on Versions

With most cases, I build the release version of the project when the conventional commits indicate that there is a new version (feat and fix). This is an additional pipeline that comes after the test: line.

release-main:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - export DRONE="true" # Required to convince `env-ci`
        # semantic-release needs this locally
        - git branch $DRONE_BRANCH origin/$DRONE_BRANCH
        - nix develop --command scripts/release.sh
    secrets:
        - gitea_token
        - git_credentials
    when:
        event: push
        branch: main

There are a number of things in this block that took me a while. The first is the event and branch. We only do releases on the main (I'm still moving away from master as racist language).

The second is the export DRONE line. At the time I set this up, env-ci wasn't aware of Woodpecker, but it was recently added thanks to 6543 on the Woodpecker Matrix channel, #woodpecker-ci:matrix.org. I don't know when the latest semantic-release will have it, but it shouldn't be needed soon, if not already.

The third is the git branch line in the above script. Woodpecker creates a detached head, as does Gitlab. But when it doesn't do is also create a local branch for the one being created. This causes a problem because the release process appears to “jump” to the branch to figure out the changes between the detached head (the commit being built) and the actual branch.

Finally, we have the secrets. semantic-release automatically picks up $GITEA_TOKEN for the release process but also needs $GIT_CREDENTIALS to verify Git access.

The token is easy, that is what given by Gitea for the user.

GIT_CREDENTIALS is slightly harder, it is a colon-separate tuple of the user name and the Gitea access token. From observations, the Gitea is basically just a bunch of URL-safe characters, so the URL-escaping isn't needed in my case (you need to URL-escape the left and right of the colon but not the colon itself).

$ export GITEA_TOKEN=9fc6d72c72e4b149f07491a0b2d3ec9215d57caf
$ export GIT_CREDENTIALS="dmoonfire:$GITEA_TOKEN"

These need to be set on a per-project basis since Woodpecker, unlike GitLab and sourcehut, there doesn't appear to be a good way of having a shared set of secrets for projects (GitLab has organization/group level secrets, sourcehut has the secret storage). This means I have to set the same GITEA_TOKEN and GIT_CREDENTIALS for all 80+ of my Fedran repositories†.

† Woodpecker has a CLI, woodpecker-cli which will let me automate that. I will use that.

Create Release on Tag

One thing I'm moving toward is creating a release entry on the forge. Since this happens after the build process and Woodpecker uses a different Docker image, I need it to be a post-release event so I hang it off the tagging process instead of the push to main.

6543 came to my rescue again with this one, so that is why there are those two “tag” elements in the script above. I also have a new stanza for the release process:

release-gitea:
    image: plugins/gitea-release
    settings:
        base_url: https://src.mfgames.com
        files:
            - "*.pdf"
            - "*.epub"
        api_key:
            from_secret: gitea_token
    when:
        event: tag
        tag: v*

If I didn't have the event: and tag: in the build: stanza, it wasn't working for the tags. This caused me some difficulties because I usually treat hashes as being unordered, but Woodpecker uses file order for processing pipelines. So, I needed to have the build: target build the file (with the correct version because it was tagged) and then release-gitea: to use that output for the release process. The test: and release-main: are skipped because they don't have those events listed.

In addition, secrets are handled differently when done as a parameter for a plugin. That is why I have the from_secret: element in the above script. This inconsistency threw me for a few days.

Putting it Together

If you want to see the final version, check out this example which has my current version as a single file.

Conclusion

I'm happy to move over to Woodpecker (you know, except for the cost of hosting) both because of the control and the challenge. I also don't have a need for speed, so if it takes a while to get through the queue, I'm okay.

Integrating Semantic Versioning into MfGames Writing

2018-08-27T05:00:00Z

The final component of this MfGames Writing series is how to integrate semantic releases into the publication process.

If you want to use sentimental versioning, then you can probably skip this. I like having automatic versioning because it helps me identify the version that beta readers or an editor has when I go to integrate the changes.

Series

I appear to be writing a short series of post about the tools I use for publication and writing.

Semantic Versions and Releases: Why semantic versioning helps with the writing process.
Evolution of MfGames Writing: A brief history and reasoning behind the tools.
First Steps Using MfGames Writing: Starting a new project with MfGames Writing.
Adding Content to MfGames Writing: Adding front and back matter to novels.
Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.
Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.
Theming for MfGames Writing: A light introduction on how to customize the output.
Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.

NPM Packages

Like everything else, we pull in a number of packages from NPM to handle the release process.

$ npm install
npm install \
  @commitlint/cli \
  @commitlint/config-conventional \
  @semantic-release/changelog \
  @semantic-release/git \
  commitizen \
  cz-conventional-changelog \
  husky \
  semantic-release

This is a pretty big and scary list, but there isn't a lot of clean ways to do this. I'll give a brief summary of them.

The @commitlint is to make sure we have a consistent commit messages (the feat: and fix: stuff). This makes sure everything else works smoothly.

The @semantic-release packages are to do the release process.

The two packages, commitizen and cz-conventional-changelog, are used to help guide you through creating the commits if you are unfamiliar with it. When these are installed, you can use git cz instead of git commit.

Finally, husky is used to make sure you follow the commits correctly because it will reject the commit if you don't follow conventions.

Configuration

The bulk of the configuration happens inside package.json. This file can get pretty big, so I'm only going to list the differences.

{
    "scripts": {
        "commitmsg": "commitlint -E GIT_PARAMS"
    },
    "release": {
        "branch": "master",
        "message": "chore(release): v${nextRelease.version}\n\n${nextRelease.notes}",
        "verifyConditions": [
            "@semantic-release/changelog",
            "@semantic-release/git"
        ],
        "analyzeCommits": ["@semantic-release/commit-analyzer"],
        "prepare": [
            "@semantic-release/changelog",
            "@semantic-release/npm",
            "@semantic-release/git"
        ],
        "publish": [],
        "success": [],
        "fail": []
    },
    "commitlint": {
        "extends": ["@commitlint/config-conventional"]
    }
}

Updating .gitlab-ci.yml

To actually use this, we have to modify the GitLab setup slightly.

image: dmoonfire/mfgames-writing-js:1.1.1

stages:
    - publish

publish:
    stage: publish
    tags:
        - docker
    script:
        - npm ci
        - npx commitlint --from=master to=CI_BUILD_REF_NAME
        - npx semantic-release
        - npm run build

    artifacts:
        expire_in: 1 week
        paths:
            - "*.pdf"
            - "*.epub"
            - "*.mobi"
            - "*.docx"
            - "*.html"

The two new lines are what does the release process.

- npx commitlint --from=master to=CI_BUILD_REF_NAME
- npx semantic-release

What Does This Give You

So, given the amount of setup, what does this give you? Every time you push up a feat: or fix:, it will do the following:

Bump the version to the next appropriate one (1.0.0 to 1.1.0 for example).
Generate the EPUB, MOBI, etc file.
Create or update the CHANGELOG.md file with the summary of your changes.
Tag the version in Git with v1.1.0 (for example).

That means, every change you do will have a distinct version. With everything else tied together, you could put it in a header and use that to figure out where to make the changes (my writing group gives me 4-12 sets of corrections, frequently overlapping).

This isn't for everyone but I have found it very helpful when working with others.

Semantic Versions and Releases for Publishing

2018-08-13T05:00:00Z

Over the last few weeks, I've been tweaking my publishing process. I think I have something pretty stable and started using it for my next novel. After talking about it on social networks, a number of people suggested I write up what I've learned.

Series

I appear to be writing a short series of post about the tools I use for publication and writing.

Semantic Versions and Releases: Why semantic versioning helps with the writing process.
Evolution of MfGames Writing: A brief history and reasoning behind the tools.
First Steps Using MfGames Writing: Starting a new project with MfGames Writing.
Adding Content to MfGames Writing: Adding front and back matter to novels.
Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.
Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.
Theming for MfGames Writing: A light introduction on how to customize the output.
Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.

Software Processes

Much of how I manage writing projects is to treat them like software. It works well for framing what I have done, need to get done, and what the end goal is. Because of this, I use various features of GitLab pretty heavily: issue tracking for arcs, issues for the publication process, and even more to remind me to order books; milestones to keep me encouraged; continuous integration (CI) to simplify the production of EPUB, MOBI, and PDF versions of the file.

I've also written a number of framework tools such as MfGames Writing to work with CI services for the publication or help me manage stuff (markdowny).

Recently, I got into a discussion about treating novels as software. Some of the things I do are… strict and not everyone will care for them, but I like to think I'm making my reader's lives easier by the little details that go into my processes.

Semantic Versioning

Somewhere in 2014 or so, I fell in love with semantic versioning. Semantic versioning is a movement to create a common understanding (a grammar) of version numbers that can be understood by anyone to understand the significance of any given release.

A semantic version comes in three parts: major.minor.patch.

If a major number increases, it means there was a “breaking change”. In writing terms, it would be rewriting sections, changing motivations, removing scenes, and generally changing the meaning of the novel. I would say adding a new epilogue would be reasonably called a major change because it changes the meaning and plot of the novel.
A minor version is adding features or improvements. As I see it, a minor version is one that I would be adding details to clarify a scene, make it more obvious or more understandable. Maybe adding a paragraph or so would be a minor change.
Finally, there is a patch. This is fixing typos, character names, or the missing quotes. Sadly, I seem to have a lot of patches as I write, because I only notice those annoying little things days or weeks later.

I put the version on the legal page so it is present but not obvious unless one is looking for it. There are a number of reasons I do this. The biggest is The Deeds of Paksenarrion by Elizabeth Moon. I have two copies of this omnibus book, mainly because I love it so much. They are two different printings. One of them had a few typos in it that they corrected with a later printing. Having an indicator that there were changes (in this case, a “patch”) would tell me there is something different about these two books.

Lucas, with his reworking of the first three Star Wars movies, would have either new features (the new crowd scenes at the end of The Return of the Jedi) or breaking changes (Greedo shooting first). An example of a patch is redoing the unintended transparency of the X-Wing dash during the fight scenes or reworking the undercarriage of the hover car in the desert (Vaseline verses CGI).

I want to know those changes so when I look at a copy and notice that only the last number changed, it just means there were some typo fixes. If the first, then I start to ask questions, why?

This is also important because some authors will rewrite their books after publication. I also use it when sending books out to alpha and beta readers. Since I get the responses in a semi-random order, it is easier to know that I'm getting feedback from version 0.4.0 instead of 0.5.0. With Git, I can create a branch, manually handle the edits, and then pull them back into the master branch which will then help me consolidate changes between two versions.

Figuring out what goes where (major, minor, or patch) can be a rough.

In Sand and Blood, I'm on version 3.1.0. I switched to version 2.0.0 when I made the book Creative Commons. A license change (from “All Rights Reserved”) felt like a breaking change. I bumped to 3.0.0 when I realized there was a major plot hole about how the clans dealt with the dead (they don't talk about it), so I removed some conversations because I had a better understanding of the world after Sand and Ash.

Changes

There is an important part about version numbers: why. Knowing that someone bought version 1.1.2 of a book and now I'm on 3.1.0 would beg the question: what major changes have happened.

That is where the change log comes in. I'm still working on the format, but I try to document every version of the book, from major to minor to patch. That way, someone can look at the versions and decide for themselves if they want a copy (or to ask for a replacement, which I've done also).

Semantic Releases

Of course, the problem with change logs is that they require some effort to maintain. Over the last year, I stumbled on the concept of a semantic-release. A semantic release is an extension of the version but it automates much of the processes that are involved with figuring out what type of number has to be bumped in the version and creating the appropriate log entries.

One of the goals is to be unromantic about version numbers. I'll admit, I've been romantic about the versions up to this point, but there is something about the systems approach to versions that takes the human factor out and reduces the amount of work.

Basically, semantic-release is a workflow for NPM packages (and novels in my case) that automates the calculations of verison numbers and also updates the change. It has plugins for publishing that I could use to update my website with the latest versions whenever I make a minor change.

It also means that the versions are much smaller. Instead of having 5-10 items in a release, there is probably just one change between versions.

Commit Conventions

Since I work with Git, there is one component that needs to be done to make semantic-release: conventions. In specific, I use conventional commits which is a standards to identify if something is a minor (a “feature”) or a patch (“fix”). These are done by a convention of the commit messages:

feat: integrated edits from Marta B
fix: fixed a typo with Rutejimo's missing accent
ci: trying to get it to build on GitLab

The semantic-release package parses the commit messages (in specific, the Angular conventions) and figures out if it should be releases (any “BREAKING CHANGES”, "feat:", or “fix:” that it sees). Other commits don't bump up the version but still give me the ability to make changes.

I like the structure of using conventional commits and having it automatically feed into new version numbers and creating the change log. Later, it can be used to automate my website and maybe even publish books on various vendors.

This automation is also important for authors published by the various Typewriter Press imprints. All of them have access to their source files in GitLab. They can make changes without me approving of them. If they follow the same conventions, then it will bump up to the versions for their corrections without me tracking them or asking what changed. In that way, it allows non-technical writers to also participate in this process; all without getting overwhelmed by the technical details while still not requiring them to keep track of changes.

Tracking

Since the version number is pulled into the legal page, it also means that I can easily manage alpha and beta readers, different rounds of editors, and even coordinate changes after giving out multiple copies at the same time (my alpha and beta readers are concurrent).

Software

Much of these concepts also apply to my tools, so I also switched all of MfGames Writing tools to use semantic-release and automate the deployment. Like writing, it lets me focus on what I want to do (write) and automate the tedious stuff (release management).