MfGames Writing Upkeep

2021-02-05T06:00:00Z

As usual between major projects, I try to spend a little time working on maintenance and upkeep of the multitude of projects that I've written or contributed too. It is somewhat random, but I had a task to move two of my Node packages, @mfgames-writing/ncx and @mfgames-writing/opf into the mfgames-writing monorepo when I had done the rest of the projects.

The decision to convert to a single repository for a multiple of projects took me quite a while. Mostly it came from the problem I have with package-based arrangements, the “rail car”. When I have to make a change to a library but that change has to be propagated through a number of packages until the one that needed the change. It looks like a chain and it's tedious when I'm doing rapid sets of changes trying to figure something out.

Now, I could just do everything in a single monolithic project, basically have a mfgames-writing package that does everything, but I still follow the Unix Philosophy: Do One Thing Well. The individual packages handle EPUB, PDF, and HTML generation. It feels “wrong” to force all of them to be combined together.

In the end, I want separate packages but I dislike the process of working with them. With Node packages, I found Lerna which handles a lot of that forwarding, plus plays well with semantic-release and conventional commits. I've been happy, but it was a learning process and I missed two packages.

I also reformatting the code and switched testing over to Jest. It bumped up a few versions, but that always seem to happen when I'm learning things.

While that cleans up most of the packages, there are a few others related packages I still need to convert, namely the Docker image and the Yoman generator. Those are completely different than the core packages, so they shouldn't be in the monorepo in my opinion.

But those can wait until next around.

Working with MfGames Writing, CI, and Docker

2018-08-24T05:00:00Z

One of the main points of having MfGames Writing is to automate the publication process.

Series

I appear to be writing a short series of post about the tools I use for publication and writing.

Semantic Versions and Releases: Why semantic versioning helps with the writing process.
Evolution of MfGames Writing: A brief history and reasoning behind the tools.
First Steps Using MfGames Writing: Starting a new project with MfGames Writing.
Adding Content to MfGames Writing: Adding front and back matter to novels.
Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.
Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.
Theming for MfGames Writing: A light introduction on how to customize the output.
Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.

Adding Scripts

We've installed the commands in a previous step, but hooking them up in the package.json makes life a lot easier.

{
  "name": "test-project",
  "private": true,
  "version": "0.0.0"
  "scripts": {
    "build:epub": "mfgames-writing-format build epub",
    "build:pdf": "mfgames-writing-format build pdf",
    "build:html": "mfgames-writing-format build html",
    "build:docx": "sed 's@&#173;@@g' < $npm_package_name-$npm_package_version.html | pandoc -f html -t docx -o $npm_package_name-$npm_package_version.docx",
    "build:mobi": "kindlegen $npm_package_name-$npm_package_version.epub",
    "build": "npm run build:epub && npm run build:mobi && npm run build:pdf && npm run build:html && npm run build:docx"
  }
}

Basically this sets up my basic suite of scripts that let me generate everything or one specific file.

$ npm run build:epub
$ npm run build

You can see we use a couple other programs like pandoc to make DOCX files from HTML or MOBI files using the EPUB file and kindlegen.

I use $npm_package_name to pull in the package name (as I said, I use them for consistency) which lets me produce files with a consistent pattern including the version number.

The other formats (HTML, PDF, etc) will be talked about in tomorrow's post.

CI Configuration

I use GitLab as my primary development platform. I like the company, the fact they open-source much of their code, and their features. Most important is that they provide private repos, which means I can have one repo for every novel I work on.

GitLab has a continual integration (CI) service which will automatically run whenever I push up code. This is controlled by the .gitlab-ci.yml file.

# This Docker image contains all the libraries and tools to generate files.
image: dmoonfire/mfgames-writing-js:1.1.1

stages:
    - publish

publish:
    stage: publish
    tags:
        - docker

    # Generate the various publication files.
    script:
        - npm ci
        - npm run build

    # Keeping artifacts means we can download them after the fact.
    artifacts:
        expire_in: 1 week
        paths:
            - "*.pdf"
            - "*.epub"
            - "*.mobi"
            - "*.docx"
            - "*.html"

If CI is turned on in GitLab and this file is present, then it will automatically run the commands in the script section. It then takes the resulting PDF, EPUB, MOBI, DOCX, and HTML files and zips them up to be found on the website. Basically this runs the same command that you can run manually which makes it easier to test.

Docker

In the .gitlab-ci.yml file, you may notice we use a Docker image dmoonfire/mfgames-writing-js:1.1.1. This is up on DockerHub and includes the various programs and utilities needed to generate the files. Some of the formats, in specific WeasyPrint, have specific installations and I found that having a Docker image makes life a lot easier.

The image contains:

NPM and required packages
WeasyPrint
KindleGen
Pandoc
PdfTk (pdfcat)