Gitlab

Semantic Release and Woodpecker CI

2022-08-07T05:00:00Z

With the recent drama of GitLab, both with the CI/CD changes and then more recent possible threat of deleting old repositories, I continue my migration to a local Gitea instance, https://src.mfgames.com/ for the bulk of my code and writing.

For the most part, migrating is just a matter of shuffling data. I have a lot of repositories, both active and inactive, and it will take me months to move them over. Plus I haven't decided if I'm going to purge them from my GitLab account so there is a single source of truth or just mirror back to them.

Currently, the most difficult task was figuring out how to handle the build processing. I've mentioned previously that I use Conventional Commits and Semantic Release fairly heavily. I've branched out a little from there using Lefthook and my project layout.

Currently, the CI does the following:

Build the project
Test various conditions including valid commit messages
If the commits indicate a new build:
1. Tag it
2. Build the release version
3. Create a release on Gitea

This changes over time, but it is the basic pattern.

Tags and Git Depth

Woodpecker does not automatically download the needed tags for semantic-release (and GitVersion). This means that the .woodpecker.yml file needs to include tags.

clone:
    git:
        image: woodpeckerci/plugin-git
        settings:
            tags: true
pipeline:
    # The pipeline elements

Unike GitLab, which only limits to the last ten commits, it appears that Woodpecker downloads the full repository by default which is also needed by GitVersion because it calculates every version. Not entirely sure about semantic-release logging indicates it doesn't need the full repository, just enough back to find a version.

Building and Testing

To support task branches, I have a basic build and test code that runs on pushes and pull requests. This lets me identify bugs earlier and catch typos with my commits.

build:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - nix develop --command scripts/build.sh
    when:
        # We need both "tag" for the next section.
        event: [push, pull_request, tag]
        tag: v*

test:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - nix develop --command scripts/test.sh
    when:
        event: [push, pull_request]

From the build tasks, you can see that I'm using my current project layout which uses scripts in the scripts/ folder instead of npm run or dotnet run. This is to make it easier to work with polyglot plus works around the issue that I need to use nix develop to get into my reproducible environment since the Docker image doesn't automatically do that. This is because both Gitlab and Woodpecker use the image which bypasses initialization files and I couldn't have it run direnv allow automatically to set up environment variables.

One thing that is missing is that Woodpecker doesn't have a clean mechanism for temporary build artifacts. I can't upload the build files and then download them so I can see the final results. Instead, I have to script it out or use a S3 plugin.

Building on Versions

With most cases, I build the release version of the project when the conventional commits indicate that there is a new version (feat and fix). This is an additional pipeline that comes after the test: line.

release-main:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - export DRONE="true" # Required to convince `env-ci`
        # semantic-release needs this locally
        - git branch $DRONE_BRANCH origin/$DRONE_BRANCH
        - nix develop --command scripts/release.sh
    secrets:
        - gitea_token
        - git_credentials
    when:
        event: push
        branch: main

There are a number of things in this block that took me a while. The first is the event and branch. We only do releases on the main (I'm still moving away from master as racist language).

The second is the export DRONE line. At the time I set this up, env-ci wasn't aware of Woodpecker, but it was recently added thanks to 6543 on the Woodpecker Matrix channel, #woodpecker-ci:matrix.org. I don't know when the latest semantic-release will have it, but it shouldn't be needed soon, if not already.

The third is the git branch line in the above script. Woodpecker creates a detached head, as does Gitlab. But when it doesn't do is also create a local branch for the one being created. This causes a problem because the release process appears to “jump” to the branch to figure out the changes between the detached head (the commit being built) and the actual branch.

Finally, we have the secrets. semantic-release automatically picks up $GITEA_TOKEN for the release process but also needs $GIT_CREDENTIALS to verify Git access.

The token is easy, that is what given by Gitea for the user.

GIT_CREDENTIALS is slightly harder, it is a colon-separate tuple of the user name and the Gitea access token. From observations, the Gitea is basically just a bunch of URL-safe characters, so the URL-escaping isn't needed in my case (you need to URL-escape the left and right of the colon but not the colon itself).

$ export GITEA_TOKEN=9fc6d72c72e4b149f07491a0b2d3ec9215d57caf
$ export GIT_CREDENTIALS="dmoonfire:$GITEA_TOKEN"

These need to be set on a per-project basis since Woodpecker, unlike GitLab and sourcehut, there doesn't appear to be a good way of having a shared set of secrets for projects (GitLab has organization/group level secrets, sourcehut has the secret storage). This means I have to set the same GITEA_TOKEN and GIT_CREDENTIALS for all 80+ of my Fedran repositories†.

† Woodpecker has a CLI, woodpecker-cli which will let me automate that. I will use that.

Create Release on Tag

One thing I'm moving toward is creating a release entry on the forge. Since this happens after the build process and Woodpecker uses a different Docker image, I need it to be a post-release event so I hang it off the tagging process instead of the push to main.

6543 came to my rescue again with this one, so that is why there are those two “tag” elements in the script above. I also have a new stanza for the release process:

release-gitea:
    image: plugins/gitea-release
    settings:
        base_url: https://src.mfgames.com
        files:
            - "*.pdf"
            - "*.epub"
        api_key:
            from_secret: gitea_token
    when:
        event: tag
        tag: v*

If I didn't have the event: and tag: in the build: stanza, it wasn't working for the tags. This caused me some difficulties because I usually treat hashes as being unordered, but Woodpecker uses file order for processing pipelines. So, I needed to have the build: target build the file (with the correct version because it was tagged) and then release-gitea: to use that output for the release process. The test: and release-main: are skipped because they don't have those events listed.

In addition, secrets are handled differently when done as a parameter for a plugin. That is why I have the from_secret: element in the above script. This inconsistency threw me for a few days.

Putting it Together

If you want to see the final version, check out this example which has my current version as a single file.

Conclusion

I'm happy to move over to Woodpecker (you know, except for the cost of hosting) both because of the control and the challenge. I also don't have a need for speed, so if it takes a while to get through the queue, I'm okay.

semantic-release-nuget v1.1.0

2021-09-04T05:00:00Z

Last week, I wrote semantic-release-dotnet which was a semantic-release plugin to automatically set the version to the appropriate one before building.

One of the key parts missing from the normal .NET development cycle was also publishing the packages. I decided to break that into a separate plugin because I have a number of places where I don't want to publish but I do want something versioned (internal projects and customized deployments). So, in the essence of the Single Responsibility Principle, I created a second utility which does one thing: build and publish NuGet packages.

Introducing semantic-release-nuget. It doesn't have a lot of configurations, but the documentation covers all of them. Basically, it does one thing.

I mostly tested with MfGames.Locking, my CIL library for some thread-locking patterns, because I'm the process of carving out Gallium and Nitride into their own packages. I'm just not sure where to put them, so they are probably going in my Gitlab organization until I find a “better” organization/home.

generator-mfgames-writing v0.3.2

2021-07-31T05:00:00Z

Related to process of writing from a few days ago, I've been working on another tool that I find myself using fairly often: Yeoman. This is a scaffolding tool, which basically means it asks a few questions and then sets up a project. In my case, it does a lot of the drudge work of creating a writing project and getting it ready for Gitlab's CI which I also use heavily (and why I wrote commitlint-gitlab-ci).

Like many of the writing tools, this is based on Javascript (but not Typescript this time).

$ sudo npm install -g yo generator-mfgames-writing

The first thing it does is ask a number of questions about the project. I'll use Nor Curse Be Found, my Beauty and the Beast sequel, as an example.

$ mkdir nor-curse-be-found
$ cd nor-curse-be-found
$ yo mfgames-writing
? The title for your novel or story: (Nor Curse Be Found)

All of my writing projects have a unique “slug” to describe them. This is used for the website's URL, such as https://fedran.com/nor-curse-be-found, and also the name of the output (dmoonfire-nor-curse-be-found-0.0.1.epub). I use a library to pull out the current directory's name and try to make a reasonable title out of it. I can change the title to something more accurate, maybe with apostrophes or possessives, but generally I keep the slug pretty even. In this case, the parenthetical item is the suggested output, so I can just hit return.

? The title for your novel or story: Nor Curse Be Found
? The slug based on the title: (nor-curse-be-found)

Just in case the directory doesn't match the directory, I use a library to take that given title and create a slug out of it. One of the biggest things, my Fast Trip rule, is that I don't want to tell someone how to work. That is why I called this semi-opinionated in that there are a lot of options but the default is how I work.

There is also a question about bylines, but then we start to get into the technical bits.

? The title for your novel or story: Nor Curse Be Found
? The slug based on the title: nor-curse-be-found
? Author's name or byline: D. Moonfire
? Do you want to create the Git repository? (Y/n)

This came from some years ago, but I do one story per repository, more so if I think the story has a possibility of being a bigger piece (which many of mine are), I want to track the versions independently, or I might be sending to an editor or submitting it somewhere (not likely these days, I self-censor a lot there). Since this is so common, I put it in as part of the questions.

I have a slowly evolving set of practices when it comes to writing processes. One of the first ones is setting up a Git repository. This always starts with configuring husky, semantic-release, conventional-commits, and commitlint. (I'm working on updating my Yeoman generator to make this a lot easier, but that will be later this week.)

? Do you want to create the Git repository? Yes
? Any command to run after `git init`? set-moonfire-git-user

I also write for a number of different projects (and bylines) which means I don't always use the same Gitlab login or even email. To faciliate this, I do not have a global user.name and user.email set up in Git so it blows up if I try to make a commit without setting a local one. Since that is tedious, I write a bunch of scripts that set the user information and SSH keys (set-moonfire-git-user for writing, set-mfgames-git-user for programming for example) and I want to call them before I do that first commit. If I just hit return to have a blank, then nothing extra will happen.

? Any command to run after `git init`? set-moonfire-git-user
? Do you want to create the initial commit? Yes
? Do you want to create the initial Git tag? Yes
? What version do you want to start? 0.0.1

Now, the initial commit is a messy one and a philosophical difference between me and the Semantic Release folks. They have their reasons to hard-code the value to “1.0.0” but I'm specifically start this process before a release, so I feel it should be “0.0.1” because I want that tracking as soon as possible.

Since I don't have the ability to set it via a configuration file, I get around it by creating the initial commit and then allowing it to be tagged so it can be pushed up and get the starting version I want.

This is also because I tag versions I give to editors, beta readers, and even my writing group (when I went to it). The tagged version also shows up on the various Fedran pages and mainly it shows progression as I write.

So, Fast Trip so I jump through a few hoops. It should also be clear why I needed that command to run after git init to set my user, otherwise this part would blow up with a “user not set” message.

? What version do you want to start? 0.0.1
? Which package manager do you use? (Use arrow keys)
❯ NPM
  Yarn

Moving away from Git and into the build process, the first question is package management. I have a love/hate relationships with npm. The command works great, but I've been struggling with it (more on than off lately) so I've been using yarn fairly heavily in the last year or so. But, with Yeoman I can give both so the user can use up or down keys to select and hit return to choose. Once they do, the menu goes away and is replaced with the choice.

? Which package manager do you use? Yarn
? Do you want to use semantic releases? Yes
? Do you want to use conventional commits? Yes
? Do you want to use Husky? Yes

The next three are to set up the common functionality I use with releases. They set up semantic-release, conventional commits, commitlint, and Husky which I use to make sure all the commit messages are clean. These four program have been a stable in most of my development lately, for coding and writing, for Typescript and even C#.

? Do you want to use Husky? Yes
? Do you want to use CI/CD? Yes
? Do you use Gitlab for your CI/CD? Yes

I can't say enough for Gitlab's CI. Whenever I push up code to the server, it runs the entire publication process. Assuming it works, it then produces a zip archive with EPUB, MOBI, PDF, and other formats without me doing anything else. Every single time. With sematic releases and the rest, each one has a unique version so I don't have “final”, “final-2”, “final-2a”, etc. Just a clean version number that shows up everywhere, starting at 0.0.1.

? Do you use Gitlab for your CI/CD? Yes
? Do you use asdf? No

This is another tool I use, asdf. This was also something that pretty much started last year, but one of the goals of mfgames-writing was to be able to produce consistent output of novels even years later. Well, what I didn't realize is how much the Node ecosystem would change and Node 15 doesn't exactly run Node 8 packages very well due to deprecations and changing libraries.

This is where asdf comes in. It puts a small file (.tool-versions) which says which version of Node, Yarn, C#, Python, whatever and will automatically change to those versions when I cd into the directory. It basically rolls in all the functionality of nvm, virtualenv, and a bunch of other localized version libraries into one. Since I write and work across many languages, having one tool to manage that is easier for me. So, saying “yes” to the question copies that .tool-versions into the resulting directory.

? Do you use asdf? Yes
? Which formats do you want to use? (Press <space> to select, <a> to toggle all, <i> to invert selection)
❯◉ PDF
 ◉ EPUB
 ◯ DOCX
 ◯ HTML

With this next question, we are out of the build section and into the formats. Basically, this just queries which types of files will be written out. The directions are pretty clear, but basically this will determine which of the format libraries are included.

? Which formats do you want to use? PDF, EPUB, DOCX, HTML
? The name of the theme package to use: (@mfgames-writing/clean-theme)

I only have three themes right now, two are “generic” and one I prefer just for Fedran stories (but obviously don't prevent anyone from using it). Later, I hope to have more themes, but basically the choices are @mfgames-writing/clean-theme and @mfgames-writing/greekil-theme (a Gentium-based theme).

? The name of the theme package to use: @mfgames-writing/clean-theme
Creating initial Git repository
Initialized empty Git repository in fedran/nor-curse-be-found/.git/
Running custom Git command


I'm all done. Running npm install for you to install the required dependencies. If this fails, try running the command yourself.


   create package.json
    force ../../../.yo-rc-global.json
    force .yo-rc.json
   create release.config.js
   create commitlint.config.js
   create .husky/commit-msg
   create .gitlab-ci.yml
   create .tool-versions
   create publication.yaml
   create yarn.lock
   create README.md
   create .gitignore
   create chapters/chapter-01.md

Changes to package.json were detected.

Running yarn install for you to install the required dependencies.

With that, it just producing a wall of tell as it creates all the files, hooks up everything, and basically gets my entire project ready, including a sample first chapter, a really basic README.md, and a Git repository ready to push.

$ ls -a
.                     .git            node_modules       README.md
..                    .gitignore      package.json       release.config.js
chapters              .gitlab-ci.yml  package-lock.json  .yo-rc.json
commitlint.config.js  .husky          publication.yaml

Right off the bat, I can build the files and see the output.

$ yarn build:pdf
... lots of output
$ ls *.pdf
nor-curse-be-found-0.0.1.pdf
$ yarn build
... huge amounts of output
$ ls nor-curse-be-found*
nor-curse-be-found-0.0.1.docx  nor-curse-be-found-0.0.1.epub
nor-curse-be-found-0.0.1.html  nor-curse-be-found-0.0.1.pdf

I'm also ready to push up to Gitlab.

$ git remote add origin git@gitlab.com:fedran/nor-curse-be-found.git
$ git push --tags
$ git push -u origin main

Hopping over to Gitlab, I can see it building immediately.

As soon as it is done, it uploads the output as an artifact.

I can then click on the browse (or download) button on the right.

Browsing lets me see a list of all the outputs that I've added.

And I can even click on the PDF to preview it.

commitlint-gitlab-ci v0.0.4

2021-07-27T05:00:00Z

Sadly, my Git host of choice, Gitlab, has a little problem with commitlint. It blows up on the first few commits which lead to me reporting this issue.

Since I'm usually creating a number of new Git repositories in a month and commitlint is one of the first things I set up, I ended up writing a little NPM utility to handle the bug for me since the original package (rightfully, I feel) don't want to add Gitlab-specific code:

npx commitlint-gitlab-ci -x @commitlint/config-conventional

It seems to work fairly well for me, I've been messing with it for a little while and haven't had too many complaints.

Gallium Nitride and Gemini

2021-07-10T05:00:00Z

Last November, I switched my static site generator from CobblestoneJS (my homebrew Gulp-based one) to Statiq. There were a number of reasons, all of them still good but mainly to support what I want to do with fedran.com and my other sites.

This holiday weekend, I ended up ripping out all of the Statiq and wrote another static site generator, this time in C# and using an Entity-Component-System (ECS). I also migrated my site over to Gemini as part of the effort.

Why?

I liked many of the ideas that Statiq provided:

Treating the website generation as code.
Having pipelines with dependencies for generation.
Immutable objects on the pipeline.

However, I found myself struggling with concepts. It just didn't sit well with me and I would spend two weeks trying to implement a feature and getting stuck. When I realized I had spent a month not fixing something that was bothering me because I didn't want to delve into the code, I knew it was time to change.

That isn't to say Statiq isn't bad. It just isn't for me. That's it.

About once a year, I get 4-7 days of “alone time” to do what I want. This year, I decided to work on a new static site generator that did work the way I work today and that I hoped would carry me over for the next five years or so.

Entity-Component-Systems (ECS)

One thing that Statiq did (but differently) was implemented the system as an ECS. Basically, you have a lightweight object (the “entity”) and add various components into it. Those components are what provide the features: the text content, flags to say if it is HTML or Markdown, or the path.

While Statiq had a number of these elements built-into the Document call (basically their entity), there were a lot of assumptions that didn't always fit. Likewise, Statiq.Web had some nice opinionated ways of handling it, including a document type (binary, text, etc), but I couldn't find an easy way to extend it.

Gallium

With my ECS, Entity is only a collection plus an integer identifier. Components can be added, removed, and replaced easily using generics to determine the type. Methods are chained together but not pseudo-English fluent (which I'm also not fond of). Entities are immutable, so all the operations return a clone of that entity with the new components added, removed, or otherwise changed. (Thanks to functional programming for some of those ideas.)

Entity entity = new();
Entity entityWithTwoComponents = entity
  .Set(new Uri("https://d.moonfire.us"))
  .Set<FileSystemInfo>(new FileInfo("/bob.txt"));
Entity entityWithReplacedUri = entityWithTwoComponents
  .Set(new Uri("http://path-to-replace-d.moonfire.us/"));
Entity entityWithOnlyUri = entityWithReplacedUri
  .Remove<FileSystemInfo>();

I also really like chained operations, so most of the processing looks like this:

IEnumerable<Entity> input;
var onlyHtmlOutputs = input
  .OrderBy(x => x.Get<FileInfo>().FullPath)
  .ForComponents<Uri>((entity, uri) => this.Uri(entity, uri))
  .WhereComponents<IsHtml>();

The whole idea is ForComponents<T1, T2, T3> will go through the list and for all entities that have T1, T2, and T3, it will do the callback, otherwise it will passs it on. Likewise WhereComponents<T1> is basically a Where that says only the entities with the given components.

Those ideas really simplified a lot of the difficulties I had with CobblestoneJS. Overall, most of the logic “felt” right for me, so I'm really happy with the results. Plus, it is based on a far more stable package ecosystem (NuGet) and in a language I enjoy greatly (C#).

Also, the language uses Autofac as my preferred dependency injection of choice. I really like the library plus NodaTime when coding, so I went with these. It's a bit opinionated, but… only a few people ever used Cobblestone, so I'm going to assume very few are going to use this.

Once I get it cleaned up, I'll probably call the ECS “Gallium” because my original name was “Gallium Nitride” (GaN, because it's a cool name and I like what the molecule does). The static site generator would be named Nitride.

Nitride

Nitride is just a multi-threaded pipeline static generator. It uses pipelines much like Statiq but based on C#'s thread control (ManualReset, ReaderWriterLockSlim).

The rough code looks like this:

List<Entity> list = entities
    .Run(new IdentifyMarkdown())
    .Run(new ParseYamlHeader<PageModel>())
    .ForComponents<PageModel>(this.SetHandlebarsFromPageModel)
    .Run(this.setInstantFromComponent)
    .Run(this.filterOutFutureInstants)
    .Run(this.createCategoryIndexes)
    .Run(this.createTagIndexes)
    .ToList();

Again, DI or direct instantiate of modules, it all works the same and really ties into using Linq and C# generics. All of the pipelines are async but most of the operations (createTagIndexes, ParseYamlHeader) are not. But since the pipelines are, it is easy to make something await without changing signatures.

I really like the pipelines. For my site, I have the following:

PostsPipeline: Load posts and create blog archives
PagesPipeline: Load static pages.
ContentPipeline: Combines PostsPipeline and PagesPipeline, creates categories/tag indexes, handle filtering out future pages.
BareHtmlPipeline: Takes ContentPipeline and converts into bare, unstyled HTML. Also creates the RSS and Atom feeds from the bare HTML.
WebpackPipeline: Runs Webpack.
StyledHtmlPipeline: Takes BareHtmlPipeline and WebpackPipeline and makes it styled using handlebars.
GeminiPipeline: Takes ContentPipeline and turns the Markdown into Gemini, applies some simple styling, and generates Gemtext pages.

That's it, but I'm happy with the result because I've taken lessons learned from my previous attempts to created something that will handle Fedran's massive cross-linking and project pages, MfGames's pulling in of separate Git repositories, and also some of the more complex formatting of my new sci-fi fiction website.

Gemini

I like the idea of Gemini. It is a low-overhead protocol that has almost no extra features, no cookies, and basically focused on presenting content. In my case, I really want to see all of my sites on Gemini because I think it has some significant merits, more so as I want to get away from heavily styled content written by people who like tiny fonts or don't have my color contrast issues.

To do that, I ended up taking inspiration from md2gemini and wrote a C# library that converts Markdown into Gemtext (Gemini's markup format inspired by Markdown). The end result is pretty nice, I think and I'm really happy with the results.

Of course, it meant I had to get a virtual machine to host a Gemini server next to a HTTP one, but that was going to happen sooner or later anyways.

Next Steps

I write a lot libraries that I think are interesting but very few people worry about. They rise up, either I stick with them or I trail off, but they always scratch my itches. On that front, the following things are left to do:

Split out Gallium more formally into MfGames.Gallium or Gallium, not sure.
Clean up the API for Nitride and make sure everything is consistent. Some of the names are a bit… off, but the functionality is good.
Break all the modules into separate Gitlab projects and set up CI for releasing.
Document everything.
Convert five other websites sites to use it to figure out what is missing.

I haven't found the money to get a developer's signing certificate, so I'll probably just put everything up on my public MyGet repository.

March in Writing

2020-04-01T05:00:00Z

This was a rough month. As pretty much everyone knows, COVID-19 is throwing wrenches into everyone's plans. Between the anxiety of a pandemic, having children at home, and the regular disruptions to my schedules, writing has been difficult. Oh well, I'm going to adapt like everything else.

I'm still maintained my weekly chapter posting. I managed to get a six week buffer though it's down to two weeks at this point due to distractions of various natures (the big one is below). March covered consecutive weeks 245 through 249 of my weekly chapter goal so I'm excited that I'm halfway done through the 200s.

Allegro 11-15: I'm really enjoying this part of the story. It is the early days of Linsan on her own, mainly about those first days of being on her own and the frustrations of trying to find the murderers and thieves.

On the short story side, I had three at the end of the month:

Illusions of the Dead was a inspired by Victoria Corva. This is another of the stories about the dead of the world, but not undead. In this case, it is about an illusionist giving someone closer.
Confessions Among Friends is a short story about two old men who find out they were on opposite sides of the same war.
Brilliant Luck is the sequel to a flash fiction I wrote a while ago, Brilliant Manifestation. Told from the point of view of a father (Jacom) who's son had gotten light powers (one of the rarer forms of magic in my world), it is just another view of a parent seeing how their child changes after going away for a year. (Something that I'll probably go through myself, not sure).

Last year, I did a poem a day for National Poetry Month. As much as I'd like to do that, I'm thinking it is too much load at this point. Likewise, I have given up trying to write a money post this year. I'm also going to skip writing processes post, but maybe only for a month or two.

On the other hand, I did hit a lovely distraction. About a week ago, my Gitlab stopped deploying. I have no clue what changed and I spent two days trying to figure it out without any success. This was based on a static site generator I had written (Cobblestone) but I decided to try switching over Fedran's (no other site has a problem) to Gatsby.

The last time I tried Gatsby, it was a disaster. It took seven hours to generate the website after three days of trying. This time, however, it has been going much faster and I've been rather happy with the results. I'm about 19% done with the conversion, but I'm aiming for having a limited version of the site on https://fedran.com/ by the end of Sunday.

It doesn't hurt that I understand CSS Grid and Flexbox a lot better, so a lot of the hoops I had jumped through are much easier. However, I have years of custom code behind this website so it will take a while to produce a good looking and useful site.

In April, my goals are:

Continue Allegro and get a few more weeks buffer going.
Three short stories.
Get the 80% point on the Fedran website conversion.

As usual, let me know if you like any stories or want to see new topics or themes. I'm deeply thankful for my subscribers through Patreon and other places. Your help has been wonderful in encouraging me.

Thank you.

Integrating Semantic Versioning into MfGames Writing

2018-08-27T05:00:00Z

The final component of this MfGames Writing series is how to integrate semantic releases into the publication process.

If you want to use sentimental versioning, then you can probably skip this. I like having automatic versioning because it helps me identify the version that beta readers or an editor has when I go to integrate the changes.

Series

I appear to be writing a short series of post about the tools I use for publication and writing.

Semantic Versions and Releases: Why semantic versioning helps with the writing process.
Evolution of MfGames Writing: A brief history and reasoning behind the tools.
First Steps Using MfGames Writing: Starting a new project with MfGames Writing.
Adding Content to MfGames Writing: Adding front and back matter to novels.
Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.
Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.
Theming for MfGames Writing: A light introduction on how to customize the output.
Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.

NPM Packages

Like everything else, we pull in a number of packages from NPM to handle the release process.

$ npm install
npm install \
  @commitlint/cli \
  @commitlint/config-conventional \
  @semantic-release/changelog \
  @semantic-release/git \
  commitizen \
  cz-conventional-changelog \
  husky \
  semantic-release

This is a pretty big and scary list, but there isn't a lot of clean ways to do this. I'll give a brief summary of them.

The @commitlint is to make sure we have a consistent commit messages (the feat: and fix: stuff). This makes sure everything else works smoothly.

The @semantic-release packages are to do the release process.

The two packages, commitizen and cz-conventional-changelog, are used to help guide you through creating the commits if you are unfamiliar with it. When these are installed, you can use git cz instead of git commit.

Finally, husky is used to make sure you follow the commits correctly because it will reject the commit if you don't follow conventions.

Configuration

The bulk of the configuration happens inside package.json. This file can get pretty big, so I'm only going to list the differences.

{
    "scripts": {
        "commitmsg": "commitlint -E GIT_PARAMS"
    },
    "release": {
        "branch": "master",
        "message": "chore(release): v${nextRelease.version}\n\n${nextRelease.notes}",
        "verifyConditions": [
            "@semantic-release/changelog",
            "@semantic-release/git"
        ],
        "analyzeCommits": ["@semantic-release/commit-analyzer"],
        "prepare": [
            "@semantic-release/changelog",
            "@semantic-release/npm",
            "@semantic-release/git"
        ],
        "publish": [],
        "success": [],
        "fail": []
    },
    "commitlint": {
        "extends": ["@commitlint/config-conventional"]
    }
}

Updating .gitlab-ci.yml

To actually use this, we have to modify the GitLab setup slightly.

image: dmoonfire/mfgames-writing-js:1.1.1

stages:
    - publish

publish:
    stage: publish
    tags:
        - docker
    script:
        - npm ci
        - npx commitlint --from=master to=CI_BUILD_REF_NAME
        - npx semantic-release
        - npm run build

    artifacts:
        expire_in: 1 week
        paths:
            - "*.pdf"
            - "*.epub"
            - "*.mobi"
            - "*.docx"
            - "*.html"

The two new lines are what does the release process.

- npx commitlint --from=master to=CI_BUILD_REF_NAME
- npx semantic-release

What Does This Give You

So, given the amount of setup, what does this give you? Every time you push up a feat: or fix:, it will do the following:

Bump the version to the next appropriate one (1.0.0 to 1.1.0 for example).
Generate the EPUB, MOBI, etc file.
Create or update the CHANGELOG.md file with the summary of your changes.
Tag the version in Git with v1.1.0 (for example).

That means, every change you do will have a distinct version. With everything else tied together, you could put it in a header and use that to figure out where to make the changes (my writing group gives me 4-12 sets of corrections, frequently overlapping).

This isn't for everyone but I have found it very helpful when working with others.

Working with MfGames Writing, CI, and Docker

2018-08-24T05:00:00Z

One of the main points of having MfGames Writing is to automate the publication process.

Series

I appear to be writing a short series of post about the tools I use for publication and writing.

Semantic Versions and Releases: Why semantic versioning helps with the writing process.
Evolution of MfGames Writing: A brief history and reasoning behind the tools.
First Steps Using MfGames Writing: Starting a new project with MfGames Writing.
Adding Content to MfGames Writing: Adding front and back matter to novels.
Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.
Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.
Theming for MfGames Writing: A light introduction on how to customize the output.
Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.

Adding Scripts

We've installed the commands in a previous step, but hooking them up in the package.json makes life a lot easier.

{
  "name": "test-project",
  "private": true,
  "version": "0.0.0"
  "scripts": {
    "build:epub": "mfgames-writing-format build epub",
    "build:pdf": "mfgames-writing-format build pdf",
    "build:html": "mfgames-writing-format build html",
    "build:docx": "sed 's@&#173;@@g' < $npm_package_name-$npm_package_version.html | pandoc -f html -t docx -o $npm_package_name-$npm_package_version.docx",
    "build:mobi": "kindlegen $npm_package_name-$npm_package_version.epub",
    "build": "npm run build:epub && npm run build:mobi && npm run build:pdf && npm run build:html && npm run build:docx"
  }
}

Basically this sets up my basic suite of scripts that let me generate everything or one specific file.

$ npm run build:epub
$ npm run build

You can see we use a couple other programs like pandoc to make DOCX files from HTML or MOBI files using the EPUB file and kindlegen.

I use $npm_package_name to pull in the package name (as I said, I use them for consistency) which lets me produce files with a consistent pattern including the version number.

The other formats (HTML, PDF, etc) will be talked about in tomorrow's post.

CI Configuration

I use GitLab as my primary development platform. I like the company, the fact they open-source much of their code, and their features. Most important is that they provide private repos, which means I can have one repo for every novel I work on.

GitLab has a continual integration (CI) service which will automatically run whenever I push up code. This is controlled by the .gitlab-ci.yml file.

# This Docker image contains all the libraries and tools to generate files.
image: dmoonfire/mfgames-writing-js:1.1.1

stages:
    - publish

publish:
    stage: publish
    tags:
        - docker

    # Generate the various publication files.
    script:
        - npm ci
        - npm run build

    # Keeping artifacts means we can download them after the fact.
    artifacts:
        expire_in: 1 week
        paths:
            - "*.pdf"
            - "*.epub"
            - "*.mobi"
            - "*.docx"
            - "*.html"

If CI is turned on in GitLab and this file is present, then it will automatically run the commands in the script section. It then takes the resulting PDF, EPUB, MOBI, DOCX, and HTML files and zips them up to be found on the website. Basically this runs the same command that you can run manually which makes it easier to test.

Docker

In the .gitlab-ci.yml file, you may notice we use a Docker image dmoonfire/mfgames-writing-js:1.1.1. This is up on DockerHub and includes the various programs and utilities needed to generate the files. Some of the formats, in specific WeasyPrint, have specific installations and I found that having a Docker image makes life a lot easier.

The image contains:

NPM and required packages
WeasyPrint
KindleGen
Pandoc
PdfTk (pdfcat)

Semantic Versions and Releases for Publishing

2018-08-13T05:00:00Z

Over the last few weeks, I've been tweaking my publishing process. I think I have something pretty stable and started using it for my next novel. After talking about it on social networks, a number of people suggested I write up what I've learned.

Series

I appear to be writing a short series of post about the tools I use for publication and writing.

Semantic Versions and Releases: Why semantic versioning helps with the writing process.
Evolution of MfGames Writing: A brief history and reasoning behind the tools.
First Steps Using MfGames Writing: Starting a new project with MfGames Writing.
Adding Content to MfGames Writing: Adding front and back matter to novels.
Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.
Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.
Theming for MfGames Writing: A light introduction on how to customize the output.
Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.

Software Processes

Much of how I manage writing projects is to treat them like software. It works well for framing what I have done, need to get done, and what the end goal is. Because of this, I use various features of GitLab pretty heavily: issue tracking for arcs, issues for the publication process, and even more to remind me to order books; milestones to keep me encouraged; continuous integration (CI) to simplify the production of EPUB, MOBI, and PDF versions of the file.

I've also written a number of framework tools such as MfGames Writing to work with CI services for the publication or help me manage stuff (markdowny).

Recently, I got into a discussion about treating novels as software. Some of the things I do are… strict and not everyone will care for them, but I like to think I'm making my reader's lives easier by the little details that go into my processes.

Semantic Versioning

Somewhere in 2014 or so, I fell in love with semantic versioning. Semantic versioning is a movement to create a common understanding (a grammar) of version numbers that can be understood by anyone to understand the significance of any given release.

A semantic version comes in three parts: major.minor.patch.

If a major number increases, it means there was a “breaking change”. In writing terms, it would be rewriting sections, changing motivations, removing scenes, and generally changing the meaning of the novel. I would say adding a new epilogue would be reasonably called a major change because it changes the meaning and plot of the novel.
A minor version is adding features or improvements. As I see it, a minor version is one that I would be adding details to clarify a scene, make it more obvious or more understandable. Maybe adding a paragraph or so would be a minor change.
Finally, there is a patch. This is fixing typos, character names, or the missing quotes. Sadly, I seem to have a lot of patches as I write, because I only notice those annoying little things days or weeks later.

I put the version on the legal page so it is present but not obvious unless one is looking for it. There are a number of reasons I do this. The biggest is The Deeds of Paksenarrion by Elizabeth Moon. I have two copies of this omnibus book, mainly because I love it so much. They are two different printings. One of them had a few typos in it that they corrected with a later printing. Having an indicator that there were changes (in this case, a “patch”) would tell me there is something different about these two books.

Lucas, with his reworking of the first three Star Wars movies, would have either new features (the new crowd scenes at the end of The Return of the Jedi) or breaking changes (Greedo shooting first). An example of a patch is redoing the unintended transparency of the X-Wing dash during the fight scenes or reworking the undercarriage of the hover car in the desert (Vaseline verses CGI).

I want to know those changes so when I look at a copy and notice that only the last number changed, it just means there were some typo fixes. If the first, then I start to ask questions, why?

This is also important because some authors will rewrite their books after publication. I also use it when sending books out to alpha and beta readers. Since I get the responses in a semi-random order, it is easier to know that I'm getting feedback from version 0.4.0 instead of 0.5.0. With Git, I can create a branch, manually handle the edits, and then pull them back into the master branch which will then help me consolidate changes between two versions.

Figuring out what goes where (major, minor, or patch) can be a rough.

In Sand and Blood, I'm on version 3.1.0. I switched to version 2.0.0 when I made the book Creative Commons. A license change (from “All Rights Reserved”) felt like a breaking change. I bumped to 3.0.0 when I realized there was a major plot hole about how the clans dealt with the dead (they don't talk about it), so I removed some conversations because I had a better understanding of the world after Sand and Ash.

Changes

There is an important part about version numbers: why. Knowing that someone bought version 1.1.2 of a book and now I'm on 3.1.0 would beg the question: what major changes have happened.

That is where the change log comes in. I'm still working on the format, but I try to document every version of the book, from major to minor to patch. That way, someone can look at the versions and decide for themselves if they want a copy (or to ask for a replacement, which I've done also).

Semantic Releases

Of course, the problem with change logs is that they require some effort to maintain. Over the last year, I stumbled on the concept of a semantic-release. A semantic release is an extension of the version but it automates much of the processes that are involved with figuring out what type of number has to be bumped in the version and creating the appropriate log entries.

One of the goals is to be unromantic about version numbers. I'll admit, I've been romantic about the versions up to this point, but there is something about the systems approach to versions that takes the human factor out and reduces the amount of work.

Basically, semantic-release is a workflow for NPM packages (and novels in my case) that automates the calculations of verison numbers and also updates the change. It has plugins for publishing that I could use to update my website with the latest versions whenever I make a minor change.

It also means that the versions are much smaller. Instead of having 5-10 items in a release, there is probably just one change between versions.

Commit Conventions

Since I work with Git, there is one component that needs to be done to make semantic-release: conventions. In specific, I use conventional commits which is a standards to identify if something is a minor (a “feature”) or a patch (“fix”). These are done by a convention of the commit messages:

feat: integrated edits from Marta B
fix: fixed a typo with Rutejimo's missing accent
ci: trying to get it to build on GitLab

The semantic-release package parses the commit messages (in specific, the Angular conventions) and figures out if it should be releases (any “BREAKING CHANGES”, "feat:", or “fix:” that it sees). Other commits don't bump up the version but still give me the ability to make changes.

I like the structure of using conventional commits and having it automatically feed into new version numbers and creating the change log. Later, it can be used to automate my website and maybe even publish books on various vendors.

This automation is also important for authors published by the various Typewriter Press imprints. All of them have access to their source files in GitLab. They can make changes without me approving of them. If they follow the same conventions, then it will bump up to the versions for their corrections without me tracking them or asking what changed. In that way, it allows non-technical writers to also participate in this process; all without getting overwhelmed by the technical details while still not requiring them to keep track of changes.

Tracking

Since the version number is pulled into the legal page, it also means that I can easily manage alpha and beta readers, different rounds of editors, and even coordinate changes after giving out multiple copies at the same time (my alpha and beta readers are concurrent).

Software

Much of these concepts also apply to my tools, so I also switched all of MfGames Writing tools to use semantic-release and automate the deployment. Like writing, it lets me focus on what I want to do (write) and automate the tedious stuff (release management).

MfGames Tasks v0.0

2018-02-23T06:00:00Z

As some of you know, I have a lot of irons in the fire. Some of them are the natural problem of programmers where many interesting things never take root but sometimes there are a few that do and they quickly blossom in needing more time. In other cases, it is commissions due, weekly chapters, and the normal household to do list.

I keep most of my issues in GitLab including things I have to do for sales tax, filing dates, and even per-project items. I also keep my novels and commissions there, along with associated issues to keep each moving.

So, when I get stressed out, I write things to help me. In this case, I decided to write mfgames-tasks which pulls all of those assigned issues and throws it into a single Markdown file which I keep on Dropbox across all my machines. This way, it picks up the items from each of those repositories and jams it into a file that looks like this:

# dmoonfire / mfgames-culture-js

-   Errors representing some seconds and milliseconds
    -   https://api.github.com/repos/dmoonfire/mfgames-culture-js/issues/3

# dmoonfire / mfgames-locking-cil

-   [simple] Add/update unit tests
    -   https://gitlab.com/dmoonfire/mfgames-locking-cil/issues/2
-   [simple] Get this packaged and uploaded to NuGet
    -   https://gitlab.com/dmoonfire/mfgames-locking-cil/issues/1

# typewriter-press / events

-   {2018-08-30} Register as an attendee
    -   https://gitlab.com/typewriter-press/events/issues/2
-   {2018-03-30} Register as a vendor and get a table
    -   https://gitlab.com/typewriter-press/events/issues/1

It handles due dates (in {2018-03-30}) and labels (such as [simple]). If there are none, it skips them. On GitLab, if the issue is assigned to a milestone and doesn't have a due date of its own, it uses the milestone's due date.

Another big feature is that GitLab issues that are assigned to a milestone that hasn't started, it doesn't include them. This lets me set up quarterly or monthly milestones (e.g., taxes) and have them pop up when they are needed.

Installation

I wrote this in Typescript because I really like it as a scripting language and it has a nice ecosystem. Installation is pretty simple:

npm install --global mfgames-tasks-cli

Configuration

As my preferred configuration file format, I have a mfgames-tasks.yaml file on my laptop. It looks somewhat like this:

# This is what defines the sources to scan for outstanding tasks.
sources:
    - id: github-dmoonfire
      type: github
      token: TOKEN # Get from GitHub settings
    - id: gitlab-dmoonfire
      type: gitlab
      token: TOKEN # Get from GitLab settings
    # You can have multiple GitLab and GitHub accounts.

# This defines how the files are written out. There is only one choice and it doesn't have formatting.
outputs:
    - type: markdown
      id: md
      path: /path/to/output

With the above file, I just have this added to my hourly cron:

mfgames-tasks write path/to/mfgames-tasks.yaml

Documentation

Yeah, there isn't a lot of documentation here.

GitLab
NPM

Now, to work on the 332 lines of items on my list.

NaNoGenMo, ICON, New Laptop, and NSFW Second-Hand Dresses 19

2017-10-04T05:00:00Z

I had some technical difficulties this week, so it is a short posting again. This week, Lily is happily working on her dress when she is interrupted once again. This time, it is from Margiold who is in a fury because her daughter's dress isn't done weeks before it was done. Unreasonable? Probably. However when Lily breaks down, she finds comfort from an unexpected source.

NaNoGenMo

My plans to enjoy NaNoGenMo might be in jeopardy. Over the convention, I had a series of expensive problems (laptop got destroyed, car's oil pain got cracked) so I might have to focus on completing more commissions. That will suck away the energy I have to do NaNoGenMo, which is heartbreaking. However, I have to be “responsible” before having fun; that is what I keep telling my boy.

I won't have a laptop until Wednesday at the earliest which means I won't be able to write anything until then. Thursday is my writing group night so… nothing for almost an entire week. We'll find out, but if it comes to push, I have to do obligations first. The cool part is that I'm writing this blog post off Gitlab and the CI setup I've been using does all the rest.

New Laptop

I am excited about the new laptop. I haven't had one in about eight years. While it was a 17" with a large keyboard, it wasn't large enough for me. This time, I'm trying a 2 in 1 that folds back and then also bought a ten keyless Cherry Brown mechanical keyboard which should make it more comfortable when I get up to my full typing speeding. Anything around 100 wpm starts to cause me a lot of pain. The monitor is small but I'm hoping that it will work out if my wrists don't hurt as much.

ICON

Baring the above bad news, ICON was a fun convention. I got to chat with a lot of people and maybe made a few fans. My reading wasn't really attended (my wife and a good friend came but no one else) so the bump in book sales didn't quite happen. I am happy to see that my new covers were pretty much universally preferred over the old one.

NSFW

This chapter isn't that raunchy but there are a couple details that probably aren't work-friendly. After Lily breaks down, someone comes to her rescue from an unexpected quarter and Lily is thrust into a position where her fantasties become a bit more real, not to mention her hopes are confirmed as she is comforted from the abuse.

Second-Hand Dresses 19: Fury

Once again, Lily finally gets into the groove of working on the dresses when Marigold comes in, furious and ready to fight. Despite Lily's attempt to placate her, the far more powerful woman rants about Lily's skill and effort because Lily has not been working exclusively on Nirih's dress. When she leaves, Lily is in tears.

Read the chapter at https://fedran.com/second-hand-dresses/chapter-19/.

Patrons

My completed works and various works-in-progress are available for free on my fantasy website. If you like what you've read and want to see more, consider becoming a patron.

State of writing with Markdown, YAML, and Git 2017

2017-05-24T05:00:00Z

A year ago, at one of my more successful panels at WisCon, I was on a panel with K. Tempest Bradford and Kristine Smith talking about writing processes. I got to see a lot of cool gadgets but I also got a chance to talk about my processes of writing with Markdown, YAML, and Git.

I'm not going to WisCon this year, but I thought this would be a good opportunity to write up where I am using all those technologies to write, both for my personal projects and as a publisher.

This is a long post in a single part. If you've read my blog, there is some rehashed information.

Markdown and YAML

Probably the one part that hasn't changed is my use of Markdown and YAML. I originally used Creole with a makeshift header but after playing with Jekyll for a while, I jumped whole-heartedly on Markdown with a YAML header.

Below is an example of one of the files. I'll be referencing it a bit in this post so please forgive the length.

---
availability: public
when: 1471/3/28 MTR
duration: 25 gm
date: 2012-02-18
title: Rutejìmo
locations:
  primary:
    - Shimusogo Valley
characters:
  primary:
    - Shimusogo Rutejìmo
  secondary:
    - Shimusogo Hyonèku
  referenced:
    - Funikogo Ganósho
    - Shimusogo Gemènyo
    - Shimusogo Chimípu
    - Shimusogo Yutsupazéso
concepts:
  referenced:
    - The Wait in the Valleys
purpose:
  - Introduce Rutejìmo
  - Introduce Hyonèku
  - Introduce naming conventions
  - Introduce formality rules
  - Introduce the basic rules of politeness
summary: >
  Rutejìmo was on top of the clan's shrine roof trying to sneak in and steal his grandfather's ashes. It was a teenage game, but also one to prove that he was capable of becoming an adult. He ended up falling off the roof.

  The shrine guard, Hyonèku, caught him before he hurt himself. After a few humiliating comments, he gave Rutejìmo a choice: tell the clan elder or tell his grandmother. Neither choice was good, but Rutejìmo decided to tell his grandmother.
---

> When a child is waiting to become an adult, they are subtly encouraged to prove themselves ready for the rites of passage. In public, however, they are to remain patient and respectful. --- Funikogo Ganóshyo, *The Wait in the Valleys*

Rutejìmo's heart slammed against his ribs as he held himself still. The cool desert wind blew across his face, teasing his short, dark hair. In the night, his brown skin was lost to the shadows, but he would be exposed if anyone shone a lantern toward the top of the small building. Fortunately, the shrine house was at the southern end of the Shimusogo Valley, the clan's ancestral home, and very few of the clan went there except for meetings and prayers.

I seem to be moving from the .markdown to .md extension. It's minor but I haven't quite jumped on it; I've only done it for the last two new projects.

Atom

My current editor of choice for novels is Atom with a backup of Emacs. Unlike Emacs, Atom has standard keyboards and looks good with the fonts I use. It is a little slow, but for the most part, works pretty well for belting out words.

I'm writing some extensions for it including a modified spell check (project-spell) that can handle project-specific dictionaries (language.json). This lets me tell the spell-checker not to mark character names as misspelled.

Later, I'll integrate my work into Author Intrusion with this but that isn't nearly ready for prime time even for my purposes.

Metadata

Over the last year I noticed I am putting lot more information in the header than before. Some of it is done long after the project but even while writing, I'll put in notes about the characters, the reason I'm writing the chapter, time of day, even the outline which I remove as I write.

I just redid the Fedran website, otherwise I'd show you how the characters and related fields show up in the sidebar. I'll get that worked in the next few months, but it let me cross-link things into my wiki. Having it in the header also means I can query it to get a summary of the page using a TypeScript tool I wrote called markdowny.

$ markdowny table *.markdown -f _basename title when
| _basename           | title                   | when          |
| :------------------ | :---------------------- | :------------ |
| chapter-01.markdown | Rutejìmo                | 1471/3/28 MTR |
| chapter-02.markdown | Confession              | 1471/3/28 MTR |
| chapter-03.markdown | Morning                 | 1471/3/29 MTR |
| chapter-04.markdown | Rivals                  | 1471/3/29 MTR |
| chapter-05.markdown | Decisions               | 1471/3/29 MTR |
$

Another nice thing about markdowny is that it also lets me show those YAML lists in a useful manner.

$ markdowny table -f _basename characters.secondary
| _basename           | characters.secondary
| :------------------ | :-------------------------------------------------
| chapter-01.markdown | Hyonèku
| chapter-02.markdown | Somiryòki, Tejíko, Gemènyo
| chapter-03.markdown | Desòchu, Gemènyo, Hyonèku, Mapábyo, Opōgyo, Panédo
| chapter-04.markdown | Desòchu, Karawàbi, Tsubàyo
$

I also has writing synopsis at the end, so what I can do is put each individual chapter in its header (in my case summary) and then use markdowny to pull them out. That way, I can write the synopsis as I go and not be overwhelemed at the end.

$ markdowny sections *.markdown | head -n 5
# Rutejìmo

Rutejìmo was on top of the clan's shrine roof trying to sneak in and steal his grandfather's ashes. It was a teenage game, but also one to prove that he was capable of becoming an adult. He ended up falling off the roof.

The shrine guard, Hyonèku, caught him before he hurt himself. After a few humiliating comments, he gave Rutejìmo a choice: tell the clan elder or tell his grandmother. Neither choice was good, but Rutejìmo decided to tell his grandmother.
$

The final bit is word counting. Because of the YAML metadata, I can't get good word counts using most tools becaues the header skews the number. So I use the tool to get me counts of the content minus the YAML header.

$ markdowny count *.markdown
chapter-01.markdown:   1520
chapter-02.markdown:   2144
chapter-03.markdown:   2905
chapter-04.markdown:   1173
chapter-05.markdown:   2570

I also have an alias for mdwc to the count since I use it pretty heavily.

Directory Structure

I'm leaning toward a semi-standard layout for my projects. Some of this was built up over the last few years but even my short stories have been migrating over to it.

* `README.md` contains a summary of the project and my tasks list. * `chapters/` contains all the chapters in `chapter-01.md`. I have never had a project with more than 89 chapters, so I stick with two digit zero pad so it remains alphabetical. * `frontmatter/` contains the frontmatter (dedication, legal) files. * `backmatter/` contains the backmatter (colophon, about, also by). * `characters/` has one Markdown file per character, for future use with [Author Intrusion](/tags/author-intrusion/). * `covers/v1`: The first variant of the cover. I have a couple and each one goes into a different `vX` folder so I can keep them apart.

Even with short stories, I have a chapters/ folder. I'm just picking up a “muscle memory” of going into chapters. I consider switching to putting things into a src/ like most of my programming projects I just haven't because I couldn't see the advantage other than having a neat folder.

Renumbering

One drawback of having chapter-01.md, chapter-02.md, etc is when I have to add a new chapter. I wrote a renumber script that lets me inject a chapter.

$ ls -l
total 8
-rw-r--r-- 1 dmoonfire dmoonfire  22 May 24 08:06 chapter-01.md
-rw-r--r-- 1 dmoonfire dmoonfire 127 May 24 08:06 chapter-02.md
$ touch chapter-01a.md
$ renumber *.md
$ ls -l
total 8
-rw-r--r-- 1 dmoonfire dmoonfire   0 May 24 08:07 chapter-01.md
-rw-r--r-- 1 dmoonfire dmoonfire  22 May 24 08:06 chapter-02.md
-rw-r--r-- 1 dmoonfire dmoonfire 127 May 24 08:06 chapter-03.md
$

As you can see, using 01a puts it before the first chapter. It works the same with deleting chapters. Sadly, this script isn't very clean but eventually I'll merge it into markdowny.

Git

One of the biggest advantages of using Markdown (text files in general) is how well is plays with source control, Git in specific. I can't stress how much Git has helped me over the years. I still remember accidentally losing a chapter because I copied the wrong file in the wrong direction. Or the painful way of tracking versions (chapter-01a.doc, chapter01b.doc, chapter-01b-final.doc, etc.).

At one point, I had a single master repository which all completed pieces in the master branch and branches off that for the works-in-progress. That actually worked out pretty well… until I started publishing. While working with text is great, the binary files of working through the covers ended up creating huge repositories which took forever to clone.

Last year, I hadn't quite split apart all of my repositories but that's pretty much done now. This also meant I can give access to editors and others to a single novel without exposing the other ones. I don't have to worry about Git modules or jumping through hoops to have the binaries.

I tried LFS (large file system) for a while but then dropped it. With it not baked in, it was a little difficult to coordinate with continuous integration servers. I think in another year or so, it could work out fairly well. I also suspect it has to do with my comfort with LFS more than technical issues.

Gitlab

Related to LFS is where I'm hosting Git. A year ago, I used Git over SSH with my ISP. However, Gitlab has been fantastic, both as their hosted environment and also on an instance running on Dreamhost (my provider). There are a lot of reasons to use Gitlab as a writer (and a publisher).

Probably the biggest is private repositories. Even if I didn't how it on my own site, being able to make each of the novels private is fantastic. Github, which is only slightly slicker, doesn't allow private repositories. Even if they did, I have over a hundred projects when you count works-in-progress, completed novels, and websites. Hosting that on Github would be expensive.

I also have a Gitlab instance. The Broken Typewriter Press business and my private novels on there. This gives me full control over the site (though I trust Gitlab) and I like it. The only people on the site are me, authors, and editors.

Like Github, Gitlab has some pretty nice features. When publishing the last few books, we've used the issue tab for authors to ask for me to order books or make corrections to their book. I can give them access to make their own typo corrections, which reduces the amount of work. At the same time, because of Git, I can be doing other changes and I don't have to worry about losing or screwing up their work. For the latest book, Sins of Intent, the author did a fantastic job of using the features.

I use the milestones for the various tasks of publishing a book. That way I can set up milestones for when the book has to be done, when the release it done, and various conventions. The issues are assigned to the milestones for both the author and myself and we can track what is missing or remaining to do. In effect, we can use a public project management to handle a book's release.

I normally turn off wiki and snippets though, they usually don't help.

One of the things with using Gitlab this way is that the author has full access to the raw files. Typically, I get a Word document and break it into individual Markdown files. Corrections are done against the Markdown as my baseline format for everything else. If they want to leave BTP or something catastropic happens, they can have exactly what I published. This is because I've been through cases where local edits were made but I couldn't get them back myself. In this case, they get and can see everything I do.

Which leads to the last feature of Gitlab which I use heavily, their Continious Integration service. This was difficult to set up at first, mainly because I had to educate myself. For most novels, the .gitlab-ci.yml looks like this:

image: dmoonfire/mfgames-writing-js:0.4.0

stages:
    - review
    - publish

review:
    stage: review
    only:
        - master
    tags:
        - docker
    script:
        - git lfs pull # Yeah, I'm using LFS with this project.
        - npm install
        - npm run build
    artifacts:
        expire_in: 1 week
        paths:
            - "*.pdf"
            - "*.epub"
            - "*.mobi"

publish:
    stage: publish
    script: "echo published"
    when: manual
    artifacts:
        paths:
            - "*.pdf"
            - "*.epub"
            - "*.mobi"
    dependencies:
        - review

Every time I check in, this rebuilds the project. For the little changes, the “staging” lets me and authors test the results of the file which show up underneath the build. They expire after a week but that's okay. “Releases” are done by moving a task to “publish”.

The nice part is I can grab the resulting PDF, MOBI, and PDF about 5-10 minutes after I check in. This is good to ensure that not only I have a reproducible build and also that I won't lost the ability to recover the output if my laptop catches on fire (it is almost seven years old, four majors cracks, and has a hernia).

This ability to see the final version is great because me or another author can make changes and see it, test it, and make sure it is exactly what they want. If I change formatting, I can see the results without overloading my computer.

mfgames-writing-js

Of course, using CI requires some way of actually formatting the books. Markdown is fantastic for some things but there are relatively few tools to format it into good-looking EPUB, MOBI, and PDF files.

Over the years, I have many variants of this. The original few were based on Makefiles and various Python or Perl tools. I've also integrated pandoc into the mix. Nothing ever worked quite the way I wanted for what I considered to be a “proper” book. Files were put in the wrong place, dedications don't need titles, making sure chapters start on the right. There were little things that pushed me closer to making something more specific.

I also had a problem that extending the features for a later book broke the generation of older books. I had to start over or rebuild (usually copy/paste/edit). Now this is a problem that has been addressed by NuGet and NPM, having the build process inside the project instead of using shared programs and libraries.

Between this and using Gitlab CI runner, I decided to create a new framework that specifically was geared toward having specific versions that could be used to reproduce the book even if the underlying libraries were updated. I ended up using NPM, mainly because I'm learning TypeScript for the last few months. It also had a better story for installing (npm install), wasn't whitespace-based, and specifically designed to be isolated to the project.

The result is mfgames-writing-js, a framework for creating EPUB and PDF files from Markdown files. This entire thing is controlled by a single file checked into the Git repository.

editions:
    epub:
        format:             mfgames-writing-epub
        theme:              ./lib/efferding
        outputDirectory:    .
        outputFilename:     sins-of-intent-{{edition.version}}.epub

    pdf:
        format:             mfgames-writing-weasyprint
        theme:              ./lib/efferding
        outputDirectory:    .
        outputFilename:     sins-of-intent-{{edition.version}}.pdf
        isbn:               978-1-940509-24-2

metadata:
    title:      Sins of Intent
    series:     Cletus Efferding
    author:     Randy Roeder
    language:   en

contents:
    - element: cover
      source: covers/v2/front.jpg
      linear: false
      exclude:
        editions: [pdf]
        toc: true
    - element: bastard
      source: frontmatter/bastard.html
      linear: false
      exclude:
        toc: true
    - element: title
      source: frontmatter/title.html
      linear: false
      exclude:
        toc: true
    - element: legal
      source: frontmatter/legal.md
      liquid: true
      linear: false
      exclude:
        toc: true
    - element: dedication
      source: frontmatter/dedication.md
      linear: false
      exclude:
        toc: true
    - element: toc
      linear: false
      exclude:
        editions: [pdf]
    - element: chapter
      number: 1
      directory: chapters
      source: /^chapter-\d+.md$/
      start: true
      page: 1
      pipeline: &pipelines
          - module: mfgames-writing-hyphen
            exceptions:
                # https://www.hyphenation24.com/word/driving/ says "driv-ing"
                - driving
                - drive
    - element: acknowledgement
      source: backmatter/acknowledgments.md
      pipeline: *pipelines

This actually has turned out better than I expected. Yeah, I had trouble with using WeasyPrint for PDF generation, but it has almost all the features I needed to do right-side chapter openings, first page headers differently, and stitching everything into a single PDF with properly embedded fonts.

The one thing I can't do is create Smashword's Microsoft Word. I'm still trying to decide if it is worth it.

Website

I also release chapters every week. I used to use WordPress but it got cumbersome to add new chapters so I switched to a static site generator over the years. I started with DocPad, then Jekyll, and then extended Jekyll with Python and Perl programs to handle my requirements.

Well… I decided to write my own. This uses the same input files as the rest of my system (Markdown + YAML). It also lets me change a header to release the chapter and include it into the build.

One of the complexities I had to figure out is the different repository for each novel. In the building process, I actually clone every project I'm publishing or have published, then use scripts to pull them into a single website before generating it. Jekyll couldn't handle it easily which is one reason I created my own.

The tool for making the website is CobblestoneJS but it has no documentation and I'm still fumbling through it. On the other hand, it handles the ten or so repositories needed to build fedran.com without my conlang, world data, and individual novels.

As part of the weekly release, I've gotten the tasks down to:

Update a header in the YAML (or use the scheduler to do it once)
Write a blog post about the chapter
Copy the chapter and the post over to Ello
Copy the chapter from fedran.com to Wattpad
Copy the post over to Patreon
Run the diaspora sync post

It takes about 1-2 hours which is much shorter than the 4 is used to be.

Summary

Well, there it is, my current state of writing with Git. You have everything from writing the chapters, supplying metadata, how to store it, various tools for getting through the publication process, and even formatting it for the various vendors.

I've gotten lost trying to automate and simplify a lot of this, but I think the current state is mostly usable by others and has been pretty solid for my own needs. I'm sure I'll improve and expand on it.

If you have question or comments, please ask. I love talking about processes, looking for improvements, or explaining in more detail why I do these things.

Thank you.

The Overhead of Blogging

2017-03-02T06:00:00Z

After nearly burning myself out to finish 10k words before the end of the month, I decided to switch back over to working on my publishing business and getting ready to release my own books but also the next book I'm publishing.

With some recent realizations, I found out my current incarnation of generating EPUB, MOBI, and PDF files was fragile. In specific, it required my laptop to be set up properly. If I lost it—not an unreasonable suggestion given that it has a hernia and a cracked shell—then I would be significantly impacted.

To fix that, I decided to rewrite the system (again) using TypeScript and NodeJS. This time, the packages are pushed up to shared servers which means others can contribute (yeah, right) but I also can rebuild the entire environment without having my specific machine.

This also means that I can set up my GitLab installation to automatically build the EPUB, MOBI, and PDF files whenever I check files in. This also means the authors who have access to their own books can also make small corrections and see the results. In the end, it should reduce the work it takes for me to publish which helps more than just myself.

To simplify the logic, I'm building everything with HTML (EPUB's native format and MOBI's second language). However, there is not a good XeLaTeX conversion from HTML so I decided to try out wkhtmltopdf which I use for BTP's monthly statements. That worked out pretty well… up to the point I found out I couldn't do inner and outer margins.

Those are kind of important for books.

When one thing creates a block, try a different one. After a while, I found a nice library called WeasyPrint. This handled inner and outer margins but… it couldn't handle recto chapter pages (chapters opening on the right side) nor could it handle chapter-specific leading pages (no headers).

So I have two ways of getting past this. I figured I could either hack the code by assembling the book and doing a bit of magic with the WeasyPrint code or do the “right” thing and teach that system how to handle recto and page numbers. I don't know which one is “easier” but I'm pretty sure there is a good chance I won't have time, but I'm willing to attempt it.

I can always go back to XeLaTeX, I know how to convert that relatively quickly.

Which… leads me to my 974 Problem. I wanted to do a quick blog post about the stuff above. However, I didn't want to spent twenty minutes looking for an image to tag the post. My alternative was to create a “generic” image for posts that I didn't find an image. My novel posts will still probably have appropriate images.

Instead of spending twenty minutes for an image, I spent two days writing a generic system including properly documented and tested library to integrate into my build system. Because ten hours verses twenty minutes is somehow better when I get lost in the forest.

Gulping down websites

2016-11-04T05:00:00Z

For the last few years, I've been using a Jekyll-based website. I started with a mostly stock setup, but it wasn't long before I was supplementing the results with Perl, Python, and a fairly complicated Makefile. This worked (and sucked out many hours out of my life) and I was pretty happy.

As I've been posting repeatedly, the entire process has been getting slower and slower. The Jekyll version was taking ten to fifteen minute, which was frustrating when I'm trying to write a post (no preview).

I also started doing serials. The weekly nature of posting could really use some scheduling. That way if I have a few hours to work ahead of time, I could write up the posts and get them queued up. Yes, I'm lazy but I also know that I've had more than a few postings minutes why of midnight.

This year, I'm also planning on doing Lexember, a conlang-a-day challenge for December. This could really use scheduled posts.

Over the last week, I've created a Gulp-based system that creates the entire website. This will make it easier to work with Gitlab's CI service, which I trigger with a cron job. It also runs a lot faster.

The really hard part is to migrate the Fedran site over to this system because I have a lot of logic, but that is an adventure for another year.

Eventually, I'll actually write up the twenty Gulp plugins I've created to make this. Mainly, I just wanted to test the CI and scheduled posts.

Changes

2015-06-22T05:00:00Z

It's been a while since I've posted, I have to stop doing that. But, most of the time it is because I'm working on some project or dealing with life's little trials that make it hard to sit down and talk about them (or they are ones I choose not to share online).

Trials have an interesting affect on me. I break out of my routine to deal with it and, when I go back, I realize that things I've done religiously are no longer things I want to do. A good example is that I used to play Kingdom of Loathing. I love that game and played it every morning as part of my ritual. But, then I was flooded out of my apartment and spent three months without a laptop. When I came back, the joy had disappearing. I realized I was no longer playing the game but enjoying the puzzle of automating playing the game. So I dropped it.

The latest trial popped out a few things that don't really work.

Penflip and Wattpad

I love the idea of Penflip and Wattpad. I've gotten a number of fans from posting frequently back before I wanted to be serious about writing.

The problem is simple: pride. I want to produce the best piece of writing I can, but I simply don't have the skills to self-edit. Which means I need beta readers and editors before I feel I'm not throwing out garbage. At the moment, I don't have an army of fans who work for free, which means if I post my works-in-progress (WIP) on those sites, I'm posting “not my best quality.”

This is a purely mental issue, I'm sure, but I don't want to be an author who puts out things with a “few mistakes.” I try to hold myself to a higher standard of not having mistakes or having very few (I still remember the single typo I found in my copy of Deeds of Paksenarrion in 1,024 pages).

Before I had Sand and Blood re-edited, my dad read a copy. When he said “there weren't that many typos,” it actually bothered me a lot. And it was the other reason, besides the Immerse or Die review, that I decided to have it re-edited.

The only way I could see posting on either of these sites is to complete them, have them edited, and then post online as a weekly or monthly serial until it completes. I'm considering doing it with Sand and Blood if I ever make it Creative Commons licensed.

Until then, I'll probably removing the pieces I have up already. I don't really have anyone reading them, so I don't mind. I'll keep the Journals of Fedran issue #0 up until I get it edited, but issue #1 (which will be based on in-world religion) will remain offline until I can afford to get it edited.

Mailing Lists, Disqus, and Discourse

Another thing that is changing is my site comments. Previously, I used Disqus for the page comments. It is a useful service for smaller sites (which I'm very much one of them), but they also have some limitations such as being non-commerical for the free services, a constant push to buy their product, and linking all the sites together. Since they are free, I'm actually okay with that but that doesn't mean I have to stay with them.

What pushed me over was getting DreamCompute from DreamHost, where I host my sites. It was mainly to have a GitLab instance but I had a second instance and I figured I'd go with the free alternative I've seen on a number of sites I liked, Discourse. It took me a little while to get it installed (had to learn Docker and a bit of Ruby) but I'm pretty happy with the result.

If you are on my site, you can see it on the bottom of a number of pages. When you comment, it shows up at discuss.moonfire.us but you can add new topics over on the discussion site.

Having Discourse installed also meant that I could unify my mailing lists from Google Groups over to Discourse. That way, anyone interesting in commenting or reading or just talking about my world can post over there. And it handles email replies, which means it can also function as a full-blown mailing list for those who don't care for web forums.

I'll be shutting down (i.e., deleting) the Google Groups probably by the end of the month or so. Or at least marking them as “read only.” If I have contests, they'll be posted over there (along with a blog post).

The one drawback of Discourse is that it creates a topic as someone visits one of my pages. The steady stream of new posts has dwindled lately, but there is probably another 300-400 empty posts going to show up in the next few months. I never realized how much I wrote until I saw them popped up. It was also disheartening to see all my failed projects too.

My Time Has Value

The last one is a mental framework and less changes. I have a small publishing company called Broken Typewriter Press (BTP). It is what I use for doing ebook formatting, print typesetting, and all the little writing-related services that I offer (including commissioned writing, actually). I only have to pieces there, but my goal is to have a 1:10 ratio of my stuff to other author's.

I have three books in the pipeline which will bring me to a 1:2 ratio (if you including my short story, Casting Call).

I'm treating it as a proper publishing company, which means I that if I'm publishing it, I don't charge the author and I recoup my costs via sales. Previously, I never included my time in making covers, typesetting, or formatting books I'm publishing. In effect, treating my personal time as having no value.

I didn't have a problem when it was a one-off, those have invoices and I treat them as such.

I can't really treat my time as zero anymore, it isn't fair to me or my family.

This weekend, I went through and created some accounting sheets (avoiding doing the “proper” GnuCash until this becomes more of a business) to handle the profit/loss reports for individual authors. This time, I'm putting in items that I work on (in six minute intervals) so I can tell when I'm truly in the black verses when I've simply paid for the printing costs.