Markdown

Capabilities

2022-12-21T06:00:00Z

Lately, I've been thinking about capabilities of server APIs and how the clients interact with them. This is driven by a couple things I'm considering doing in the future but also with the frustration that certain features in the fediverse are difficult to implement for clients.

In specific, I'm very fond of Markdown posts and emoji responses. I love Markdown for all things and I like the low-attention responses of emojis when you don't need to have a big post to say “I'm sorry” or “that blows”. My family and work buddies use emoji responses fairly heavily, and it is well supported with Slack, Discord, and even Microsoft Teams as of this week.

Mastodon decided not to implement them which is one reason I like Pleroma and Misskey, though Misskey has historically been too heavy to put on my servers so one of my accounts uses Pleroma.

However, my favorite Android client, Tusky handles Markdown fairly well on the Glitch instance I'm on, but doesn't for Pleroma and it doesn't do emoji responses at all. Huksy supports these as a fork of Tusky specifically for Pleroma. But Husky doesn't really play well with the Glitch, which means I struggle to find One Client to Rule Them All™.

Likewise, I want to switch my Pleroma for GoToSocial for various reasons, but they also support Markdown but not emoji responses. Mainly because there is no standard for emoji responses and if I want to get emoji responses, I need broader acceptance or patterns to work with.

So, this comes down to capabilities. I want there was a way of having Tusky be able to query the server to determine it can handle emoji responses, so it can turn on that logic so I don't have to have Tusky and Husky. Simple, but I also want it to have the ability to know if the server can handle Markdown (base Mastodon can't, Glitch can) without doing various server identification scans.

This means a fedi server should have some ability to say “I can handle this feature” that can be queried. If there was a standard for that, then Tusky or a CLI or any other client could selectively enable or disable the feature as needed.

Fine-Grained

This is talking about a fine-grained approach to APIs. We have many different fedi servers out there, being developed and also in production. Likewise, we have a number of clients that are also growing as the fedi becomes more popular. I think we're going to see features moving from server to server as they become interesting and there is no way every client can make that choice for every server.

Identifying those abilities will make it easier to say “this client supports Markdown for those servers who have it”. It decouples the tie between the servers and the clients by giving a bit of an abstraction, not unlike how the Language Server Protocol has made developing language refactoring a lot easier across multiple clients by working to an interface.

Approach 1 - Instance Information

The first approach, which is somewhat specific to fedi servers, is to update the instance information (Mastodon, Pleroma doesn't have it implemented) to provide that information to clients. This would require updating Glitch and submitting a patch to Pleroma (and maybe Misskey) once some sort of understand/agreement could be found on how to identify these features.

This is probably the easiest and I think would really benefit the client ecosystem as a whole, not only to give me the features I want.

Approach 2 - OpenAPI Support

A more generic approach, which is beyond just fedi servers, is to encourage most API servers to also expose their OpenAPI v3.1.0 file in a well-known location that describes all the endpoints they support for that specific instance. This has benefits beyond just describing what they can handle.

However, knowing there is an endpoint that can handle something doesn't mean it works the same way. Pleroma may have the same abilities as Glitch for Markdown, but they may not be communicating them in the same way or require different inputs. There needs to be a signal that indicates a specification is in use.

References

OpenAPI has a $ref which can be used to describe an endpoint; this reference is a JSON Reference that allows an endpoint to point to another specification either in the same document or to an external URL.

It is the external URL that interests me. If that points to a fragment of OpenAPI specification that is documented for a specific feature, that would be a signal that the endpoint impelments that specific feature. In other words, if there was an /api/emoji endpoint that is described as:

{
    "$ref": "https://well.known.location/emoji.xml"
}

Then a client could use that information to assume that the endpoint has those abilities and can handle those requests.

Capabilities

As mentioned before, Mastodon doesn't have that ability since a posting is a posting, be it a response or a full text. OpenAPI also has the ability to add additional specifications to an endpoint, such as a x-additional-refs that has an array of $ref object that do point to specific ones.

That way, then Mastodon post status endpoint could add a x-additional-$refs that would allow it to describe the ability to have polls (added in 2.8.0), schedules (2.7.0), in a clear manner that clients would know it could handle those correctly without guessing or inferring from server information. Or… emoji.

{
    "paths": {
        "/api/v1/statuses": {
            "post": {
                "operationId": "postStatus",
                "parameters": [],
                "x-additional-$refs": [
                    "https://well.known.location/emoji.xml"
                ]
            }
        }
    }
}

This approach also appeals to some other projects I have in mind, so I'm curious to see what others think of it.

Thoughts

Obviously, I have a specific goal in mind: Markdown and emoji. But hopefully there is also a benefit for allowing other servers to provide this information and make it easier to create clients and interfaces to their servers.

Integrating Semantic Versioning into MfGames Writing

2018-08-27T05:00:00Z

The final component of this MfGames Writing series is how to integrate semantic releases into the publication process.

If you want to use sentimental versioning, then you can probably skip this. I like having automatic versioning because it helps me identify the version that beta readers or an editor has when I go to integrate the changes.

Series

I appear to be writing a short series of post about the tools I use for publication and writing.

Semantic Versions and Releases: Why semantic versioning helps with the writing process.
Evolution of MfGames Writing: A brief history and reasoning behind the tools.
First Steps Using MfGames Writing: Starting a new project with MfGames Writing.
Adding Content to MfGames Writing: Adding front and back matter to novels.
Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.
Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.
Theming for MfGames Writing: A light introduction on how to customize the output.
Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.

NPM Packages

Like everything else, we pull in a number of packages from NPM to handle the release process.

$ npm install
npm install \
  @commitlint/cli \
  @commitlint/config-conventional \
  @semantic-release/changelog \
  @semantic-release/git \
  commitizen \
  cz-conventional-changelog \
  husky \
  semantic-release

This is a pretty big and scary list, but there isn't a lot of clean ways to do this. I'll give a brief summary of them.

The @commitlint is to make sure we have a consistent commit messages (the feat: and fix: stuff). This makes sure everything else works smoothly.

The @semantic-release packages are to do the release process.

The two packages, commitizen and cz-conventional-changelog, are used to help guide you through creating the commits if you are unfamiliar with it. When these are installed, you can use git cz instead of git commit.

Finally, husky is used to make sure you follow the commits correctly because it will reject the commit if you don't follow conventions.

Configuration

The bulk of the configuration happens inside package.json. This file can get pretty big, so I'm only going to list the differences.

{
    "scripts": {
        "commitmsg": "commitlint -E GIT_PARAMS"
    },
    "release": {
        "branch": "master",
        "message": "chore(release): v${nextRelease.version}\n\n${nextRelease.notes}",
        "verifyConditions": [
            "@semantic-release/changelog",
            "@semantic-release/git"
        ],
        "analyzeCommits": ["@semantic-release/commit-analyzer"],
        "prepare": [
            "@semantic-release/changelog",
            "@semantic-release/npm",
            "@semantic-release/git"
        ],
        "publish": [],
        "success": [],
        "fail": []
    },
    "commitlint": {
        "extends": ["@commitlint/config-conventional"]
    }
}

Updating .gitlab-ci.yml

To actually use this, we have to modify the GitLab setup slightly.

image: dmoonfire/mfgames-writing-js:1.1.1

stages:
    - publish

publish:
    stage: publish
    tags:
        - docker
    script:
        - npm ci
        - npx commitlint --from=master to=CI_BUILD_REF_NAME
        - npx semantic-release
        - npm run build

    artifacts:
        expire_in: 1 week
        paths:
            - "*.pdf"
            - "*.epub"
            - "*.mobi"
            - "*.docx"
            - "*.html"

The two new lines are what does the release process.

- npx commitlint --from=master to=CI_BUILD_REF_NAME
- npx semantic-release

What Does This Give You

So, given the amount of setup, what does this give you? Every time you push up a feat: or fix:, it will do the following:

Bump the version to the next appropriate one (1.0.0 to 1.1.0 for example).
Generate the EPUB, MOBI, etc file.
Create or update the CHANGELOG.md file with the summary of your changes.
Tag the version in Git with v1.1.0 (for example).

That means, every change you do will have a distinct version. With everything else tied together, you could put it in a header and use that to figure out where to make the changes (my writing group gives me 4-12 sets of corrections, frequently overlapping).

This isn't for everyone but I have found it very helpful when working with others.

Theming for MfGames Writing

2018-08-26T05:00:00Z

One of the big parts about MfGames Writing is the ability to customize the appearance of the generated files. If we didn't have that, then every book would look the same and I don't care for that. Books should allow for different themes, chapter numbers, and even spacing. To that regard, everything is funneled through a theme for formatting.

Series

I appear to be writing a short series of post about the tools I use for publication and writing.

Semantic Versions and Releases: Why semantic versioning helps with the writing process.
Evolution of MfGames Writing: A brief history and reasoning behind the tools.
First Steps Using MfGames Writing: Starting a new project with MfGames Writing.
Adding Content to MfGames Writing: Adding front and back matter to novels.
Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.
Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.
Theming for MfGames Writing: A light introduction on how to customize the output.
Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.

HTML

The entire theme system is built on HTML and CSS. In previous versions of this, I generated XeLaTeX files for PDF and used a custom conversion utility for Word documents. It also increased the complexity significantly. Switching to a HTML-based system simplified the theme creation and made debugging easier.

This is one case where looking at a reference implementation will be helpful.

Components

In the content post, there are quite a few references to element on each of the content items. These elements are Liquid-based stylesheets that are provided by the theme.

For example, the title.xhtml in the clean theme looks like this:

<div class="element-page-break element-page-break-title">&#160;</div>
{{html}}

The {{html}} is where the contents in the source given on the content will be placed. In this case, the element-page-break is how I handle page rendering with WeasyPrint.

A more complicated example would be chapter.xhtml:

<div class="element-wrapper {{content.element}}">
    <div class="element-page-break element-page-break-right">&#160;</div>
    <div class="element-page-pad-right">
        {% if content.number %}
        <div class="element-number">Chapter {{content.number}}</div>
        {% endif %}
        <h1 class="element-title">{{content.title}}</h1>
        {{html}}
    </div>
</div>

You can see more uses of the Liquid tags. In this case, if we have a number property in the content, it will insert the Chapter # above the title of the chapter (from the YAML header) and the contents of the body into {{html}}.

Recursive Templates

To reduce copy/paste, there are also recursive templates. For example, the colophon template looks like this:

---
extends: simple-title
---
{{html}}

Once this page is rendered, the results are formatted by the simple-title

<div class="element-wrapper {{content.element}}">
<div class="element-page-break element-page-break-right">&#160;</div>
<h1 class="initial">{{content.title}}</h1>
{{html}}
</div>

It is recursive, so you can have one template extending another extending another.

Styling

You'll notice that there is very little styling in the HTML. Instead, most of it is done with a SASS template in the theme. The base style is called stylesheet.scss:

p {
    margin-bottom: 0;
    margin-top: 0;
    text-align: left;
    text-indent: 2em;
}

Stylesheets can be overridden by specific formats. The most common is having a WeasyPrint-specific stylesheet:

@import "stylesheet";

@page {
    @bottom-center {
        content: counter(page);
        vertical-align: top;
        padding-top: 1em;
    }
    size: letter portrait;
    margin: 3cm 2cm;
}

To figure out overrides, the system looks for the format name first (html.scss or weasyprint.scss) and if it doesn't find one, goes to the base (stylesheet.scss).

Stylesheet Variables

Because I'm fond of single points of truth, SASS also lets us have variables so we can pull the version, or information from the edition data.

@page :right {
    @top-right {
        content: themeString("edition.title");
        vertical-align: bottom;
        padding-bottom: 1em;
    }
    padding-left: 1cm;
}

The themeString works just like the Liquid templates.

Creating New Themes

To create a new theme, I basically just copy all of the clean theme and start customizing.

Additional Formats for MfGames Writing

2018-08-25T05:00:00Z

Not everyone in the world uses EPUB. Yes, I consider it one of the best formats for me because I can change font size or layout, there is still a big need for print books and Amazon is the source of most ebook sales (though I don't care for that either).

To handle that, MfGames Writing provides a pluggable collection of different output formats. This allows you to use only the formats that you way (EPUB2, PDF, HTML) and they can be updated separately.

Series

I appear to be writing a short series of post about the tools I use for publication and writing.

Semantic Versions and Releases: Why semantic versioning helps with the writing process.
Evolution of MfGames Writing: A brief history and reasoning behind the tools.
First Steps Using MfGames Writing: Starting a new project with MfGames Writing.
Adding Content to MfGames Writing: Adding front and back matter to novels.
Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.
Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.
Theming for MfGames Writing: A light introduction on how to customize the output.
Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.

Using Formats

Formats are consumed in the edition section of the publication.yaml file.

metadata:
    title: Test Project
    author: D. Moonfire
    language: en

    theme: "@mfgames-writing/clean"
    outputDirectory: .
    outputFilename: test-project-{{edition.version}}.{{edition.editionName}}

editions:
    epub:
        format: "@mfgames-writing/epub2"
    pdf:
        format: "@mfgames-writing/weasyprint"
        isbn: 978-1-999999
    html:
        format: "@mfgames-writing/html"
        images:
            scale: 0.25

The format is the name of the package installed with npm install.

EPUB

The easiest format to use is EPUB2.

$ npm install @mfgames-writing/epub2
$ npm run build:html

The setup for the NPM commands can be found yesterday.

When this runs, it will generate an EPUB2 file that will be verifiable with epubcheck.

HTML

HTML is also pretty easy.

$ npm install @mfgames-writing/html

PDF

PDF is the complicated one and why I use a Docker image. I decided to use WeasyPrint for PDF generation because it creates good-quality PDF files and works with HTML, the format themes use.

$ npm install @mfgames-writing/weasyprint

This assumes that weasyprint and pdftk are both in the PATH.

MOBI

There isn't a good package for generating MOBI files directly, so the best approach to creating a Amazon MOBI file for uploading is to create a EPUB file and then use kindlegen to convert it.

$ npm run build:epub
$ kindlegen *.epub

DOCX

Like MOBI, it is hard to write a DOCX in a clean manner. To do that, we create a HTML and then convert it.

$ npm run build:html
$ sed 's@&#173;@@g' < *.html \
  | pandoc -f html -t docx -o test-package-0.0.0.docx

We have the extra sed command to remove the hypenation from @mfgames-writing/hyphen which makes a cleaner file.

Working with MfGames Writing, CI, and Docker

2018-08-24T05:00:00Z

One of the main points of having MfGames Writing is to automate the publication process.

Series

I appear to be writing a short series of post about the tools I use for publication and writing.

Semantic Versions and Releases: Why semantic versioning helps with the writing process.
Evolution of MfGames Writing: A brief history and reasoning behind the tools.
First Steps Using MfGames Writing: Starting a new project with MfGames Writing.
Adding Content to MfGames Writing: Adding front and back matter to novels.
Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.
Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.
Theming for MfGames Writing: A light introduction on how to customize the output.
Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.

Adding Scripts

We've installed the commands in a previous step, but hooking them up in the package.json makes life a lot easier.

{
  "name": "test-project",
  "private": true,
  "version": "0.0.0"
  "scripts": {
    "build:epub": "mfgames-writing-format build epub",
    "build:pdf": "mfgames-writing-format build pdf",
    "build:html": "mfgames-writing-format build html",
    "build:docx": "sed 's@&#173;@@g' < $npm_package_name-$npm_package_version.html | pandoc -f html -t docx -o $npm_package_name-$npm_package_version.docx",
    "build:mobi": "kindlegen $npm_package_name-$npm_package_version.epub",
    "build": "npm run build:epub && npm run build:mobi && npm run build:pdf && npm run build:html && npm run build:docx"
  }
}

Basically this sets up my basic suite of scripts that let me generate everything or one specific file.

$ npm run build:epub
$ npm run build

You can see we use a couple other programs like pandoc to make DOCX files from HTML or MOBI files using the EPUB file and kindlegen.

I use $npm_package_name to pull in the package name (as I said, I use them for consistency) which lets me produce files with a consistent pattern including the version number.

The other formats (HTML, PDF, etc) will be talked about in tomorrow's post.

CI Configuration

I use GitLab as my primary development platform. I like the company, the fact they open-source much of their code, and their features. Most important is that they provide private repos, which means I can have one repo for every novel I work on.

GitLab has a continual integration (CI) service which will automatically run whenever I push up code. This is controlled by the .gitlab-ci.yml file.

# This Docker image contains all the libraries and tools to generate files.
image: dmoonfire/mfgames-writing-js:1.1.1

stages:
    - publish

publish:
    stage: publish
    tags:
        - docker

    # Generate the various publication files.
    script:
        - npm ci
        - npm run build

    # Keeping artifacts means we can download them after the fact.
    artifacts:
        expire_in: 1 week
        paths:
            - "*.pdf"
            - "*.epub"
            - "*.mobi"
            - "*.docx"
            - "*.html"

If CI is turned on in GitLab and this file is present, then it will automatically run the commands in the script section. It then takes the resulting PDF, EPUB, MOBI, DOCX, and HTML files and zips them up to be found on the website. Basically this runs the same command that you can run manually which makes it easier to test.

Docker

In the .gitlab-ci.yml file, you may notice we use a Docker image dmoonfire/mfgames-writing-js:1.1.1. This is up on DockerHub and includes the various programs and utilities needed to generate the files. Some of the formats, in specific WeasyPrint, have specific installations and I found that having a Docker image makes life a lot easier.

The image contains:

NPM and required packages
WeasyPrint
KindleGen
Pandoc
PdfTk (pdfcat)

Adding Content to MfGames Writing

2018-08-23T05:00:00Z

In my continuing introduction to MfGames Writing, I've only given a very brief introduction to adding content to the publication. Like everything else, there is a lot more complexity in the process to support various components of a novel: bastard and full titles, table of contents, appendixes, and even colophons.

Series

I appear to be writing a short series of post about the tools I use for publication and writing.

Semantic Versions and Releases: Why semantic versioning helps with the writing process.
Evolution of MfGames Writing: A brief history and reasoning behind the tools.
First Steps Using MfGames Writing: Starting a new project with MfGames Writing.
Adding Content to MfGames Writing: Adding front and back matter to novels.
Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.
Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.
Theming for MfGames Writing: A light introduction on how to customize the output.
Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.

More Than Chapters

In the previous blog post, I gave a short example of publication.yaml that included all the chapters in a folder.

contents:
    - element: chapter
      number: 1
      directory: chapters
      source: /^chapter-\d+.md$/
      start: true
      page: 1

This would include any number of chapters, from 01 to 99.

Covers

Most EPUB files need a good graphical image for readers. Creating MOBI (via kindlegen) from the EPUB also requires a graphical image. Adding one is pretty easy.

- element: cover
  source: build/repos/fedran-covers/dist/full/0100-00-sand-and-blood-front.jpg
  linear: false
  exclude:
      editions: [pdf]
      toc: true

The element is the name of the component in the theme. Basically that determines how it is rendered when we generate the files.

The source is the relative path to the image. It should be a JPEG image. In my example, I have it pulling from another Git repository but it can be a file checked into the Git. In general, it needs to be a file that will be accessed while building.

We use linear: false because covers are one of those things that reader software will occasionally rearrange. In other words, I don't know, it seems to be required.

We also use exclude: to exclude it from the table of contents (toc) and from the PDF version.

Titles

- element: bastard
  source: frontmatter/bastard.html
  linear: false
  exclude:
      editions: [epub]
      toc: true
- element: title
  source: frontmatter/title.html
  linear: false
  exclude:
      toc: true

In this case, we are using the bastard and title templates (nearly identical) from the theme. I put my front matter files in a directory creatively called frontmatter. You can probably guess where the back matter goes. I could use a number in the front (like chapters) and a pattern but I like to be explicit with my matters.

The exclude.toc is because title pages don't show up in the table of contents.

Legal Page

- element: legal
  source: frontmatter/legal.markdown
  liquid: true
  linear: false

The legal page introduces a new concept. The liquid flag lets me use Liquid templates on the page. The variables available for the tags are combined from three places: the package.json file, the metadata section in publication.yaml, and then the edition being generated from the same file. This means you can store variables like the version from package.json but an ISBN from the edition.

Since we are using Liquid, we can also have some simple logic such as the optional ISBN which only shows up in the pdf edition. This would allow us to have two different versions of the book (say hardcover and paperback) with different formats and edition names.

---
title: Legal
---

Broken Typewriter Press\
[https://broken.typewriter.press/](https://broken.typewriter.press/)

{% if edition.isbn %}
ISBN {{ edition.isbn }}
{% endif %}

Version {{ edition.version }}

Using versions like this could also be used to create site-specific buy links for the different editions (such as a Amazon or Smashwords-specific edition of the same file).

The table of contents (element: toc) has a special code to generate one if a source isn't given. However, the generated code doesn't do page numbers which is why I currently have it turned off in PDF. For EPUB and MOBI, it will generated a properly linked table for each chapter.

- element: toc
  linear: false
  title: Contents
  exclude:
      editions: [pdf]

Pipelines

The last interesting component to point out is the pipelines. A pipeline is something that modifies the original Markdown before it is converted into HTML (I've needed a post conversion one also but I haven't coded it). These are used to handle word hyphenation (we don't do it by default), either for generic languages or for specific languages (such as the Fedran one to handle Miwāfu names). There is also one to convert << and >> into « and » (I use this for telepathy in my books).

- element: chapter
  number: 1
  directory: chapters
  source: /^chapter-\d+.markdown$/
  start: true
  page: 1
  pipeline: &pipelines
      - module: "@fedran/writing-hyphen"
      - module: mfgames-writing-hyphen

Sand and Blood example

Using Sand and Blood as an example:

contents:
    - element: cover
      source: build/repos/fedran-covers/dist/full/0100-00-sand-and-blood-front.jpg
      linear: false
      exclude:
          editions: [pdf]
          toc: true
    - element: bastard
      source: frontmatter/bastard.html
      linear: false
      exclude:
          editions: [epub]
          toc: true
    - element: title
      source: frontmatter/title.html
      linear: false
      exclude:
          toc: true
    - element: legal
      source: frontmatter/legal.markdown
      liquid: true
      linear: false
    - element: dedication
      source: frontmatter/dedication.markdown
      linear: false
    - element: toc
      linear: false
      title: Contents
      exclude:
          editions: [pdf]
    - element: preface
      source: frontmatter/miwafu.html
      linear: false
    - element: chapter
      number: 1
      directory: chapters
      source: /^chapter-\d+.markdown$/
      start: true
      page: 1
      pipeline: &pipelines
          - module: "@fedran/writing-hyphen"
          - module: "@mfgames-writing/hyphen"
    - element: appendix
      source: backmatter/about.markdown
    - element: appendix
      source: backmatter/fedran.markdown
    - element: appendix
      id: license
      source: backmatter/license.markdown
    - element: appendix
      source: backmatter/patrons.markdown
    - element: appendix
      source: backmatter/credits.markdown
    - element: colophon
      source: backmatter/colophon.markdown

EPUBCHECK

Once you get the basics in (cover, table of contents, and chapters), the resulting EPUB file will easily get through epubcheck which makes it easier to then convert that into a MOBI file using kindlegen.

First Steps Using MfGames Writing (GitLab, NPM)

2018-08-22T05:00:00Z

Getting started with MfGames Writing can be a bit overwhelming. This is the beginning of how to get started. These directions are mainly focused toward technical users who already have a basic skill in Git.

Series

I appear to be writing a short series of post about the tools I use for publication and writing.

Semantic Versions and Releases: Why semantic versioning helps with the writing process.
Evolution of MfGames Writing: A brief history and reasoning behind the tools.
First Steps Using MfGames Writing: How to get started with a project.
Adding Content to MfGames Writing: Adding front and back matter to novels.
Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.
Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.
Theming for MfGames Writing: A light introduction on how to customize the output.
Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.

Setting up Git

I always start by creating a new Git project on GitLab. You can do the same with GitHub, BitBucket, or just locally. Since I'm focusing on the tools, I'm going to assume you know how to create those projects.

$ git checkout https://oauth2:SecretPassword@gitlab.com/dmoonfire/test-project.git
... stuff happens
$ cd test-project

You could create a new folder, it doens't really matter.

$ mkdir test-project
$ cd test-project
$ git init

Setting up .gitignore

The way builds work, there are quite a few files that will be generated that wouldn't be checked into the repository. This is one thing that makes NPM so powerful. To avoid that, we create a .gitignore file that ignores those files.

# Emacs and VI creates these files and I'm too lazy to turn it off.
*~
\#*
.#*

# This is the output of the various build processes.
*.pdf
*.mobi
*.epub
*.docx

# Sometimes I use `build` for my output but most of the time I just put it in the root.
build/

# This is where Node/NPM puts its file.
node_modules/

Setting up .editorconfig

EditorConfig is a tool for providing some consistency in formatting. For novels, it doesn't make sense but I like to have it there to make sure the supporting files are consistent. This is an optional file, I just like having it.

# EditorConfig is awesome: http://EditorConfig.org

# top-most EditorConfig file
root = true

[*]
charset = utf-8
end_of_line = lf
indent_brace_style = K&R
indent_size = 4
indent_style = space
insert_final_newline = true
max_line_length = 80
tab_width = 4
trim_trailing_whitespace = true
curly_bracket_next_line = false

[*.{js,ts}]
quote_type = double

[*.yaml]
indent_size = 2
tab_width = 2

[package.json]
indent_size = 2
tab_width = 2

Setting up package.json

The versioning and management comes from package.json. This is the file that says which version of which utilities to use. Now, you can use npm init to figure it out but this is the basic file.

{
    "name": "test-project",
    "private": true,
    "version": "0.0.0"
}

There is other stuff, but these are the key parts. The name is the slug of the file. For me, it is always the name of the folder in the website and the Git repository. I also use this to build my website, so it is a pretty consistent format. You can call it whatever you want and change it as you need.

The private is to prevent you from accidently publishing your novel on NPM.

Finally the version is used for the semantic release and populates data on the legal page of the resulting book. In general, I start with “0.0.0” and then bump it to “1.0.0” when I finalize the book and publish it for the first time.

Write a Chapter

Now, we need somethig to put into the story or novel. Since I've started this, I organize even my short story into chapters. To do that, you can put it anywhere, but I always put my chapters in (unoriginally) the chapters/ folder. I also use two-digit numbers with zero padding so it sorts correctly (this is useful). You can see this at Sand and Blood's Git Repository.

For example, chapters/chapter-01.md:

---
title: Mistakes Were Made
---

It was a bright and terribly sunny day.

I didn't like it.

Setting up publication.yaml

Because of the flexibility (if you don't want to use chapters/, want to spread it out across deeper folders), nothing is automatic when it comes to formatting the books. To control it, we have a simple file called publication.yaml which tells the system how to create the output. This would go at the top-level file.

metadata:
    title: Test Project
    author: D. Moonfire
    language: en

    theme: "@mfgames-writing/clean"
    outputDirectory: .
    outputFilename: test-project-{{edition.version}}.{{edition.editionName}}

editions:
    epub: # This is the edition.editionName.
        format: "@mfgames-writing/epub2"

contents:
    - element: chapter
      number: 1
      directory: chapters
      source: /^chapter-\d+.md$/
      start: true
      page: 1

Now, since this file is the core of MfGames Writing, it needs a bunch of details.

Liquid

Many of the fields use Liquid templates. That is the stuff between {{ and }}. The main reason we use it is so we can simplify some of our logic. For example, instead of putting outputFilename in every edition, we can use variables to fill it in. Also, we can substitute {{ edition.version }}`` to have the version from the package.jsonfile. The{{edition.editionName}}will haveepubfor theepub:line. This means it wil produce a filetest-project-0.0.0.epub` when it runs.

Theme

The formatting is driven around the idea of a “theme”. Right now, there are only three themes. Two are private or require custom fonts, the other is @mfgames-writing/clean which is a utilitarian theme based on SASS and some templates. These themes are used to create a consistent style. I also have more specialized themes:

Fedran: For the books and stories at https://fedran.com/. This also is used for my print books.
Efferding: For Randy Roeder's Sins of Intent and the sequel coming out next month.

Both of these have custom fonts (Corda and Mr. Eaves) with a bit of logic. I could easily see this being expanded by anyone to create a gallery of formatting for books. Right now, I'm focusing on these three because they are functional enough.

See the later post on theme specifics.

Editions

Editions are basically versions of the output. For Sand and Blood, I have one for EPUB, HTML, and PDF. MOBI and DOCX are generated from those three sources. You can have any number of editions, different file names, different output directories, even different variables like ISBN numbers.

Content

Content is what goes into the file. Right now, we only have a single element, all the files in the chapter folder.

- element: chapter
  number: 1
  directory: chapters
  source: /^chapter-\d+.md$/
  start: true
  page: 1

The element tells the theme how to format it. Later examples will include more but you can have things like preface or appendix or titles and the like.

The number property basically is used if you want to have “Chapter 1” in the theme but with a custom title. If your chapter is just “One”, “Two”, “Three”, you wouldn't define this. My books do since I use chapter titles.

The directory and source work together to figure out which files are to be included.

start is used by EPUB and MOBI to figure out where to automatically navigate when the reader first goes into the page.

Finally page is used by the PDF system to restart page numbering. Usually this would be left blank.

Installing Tools

At this point, I have a minimum novel project. Time to make it actually produce something. Like in the previous, everything is self-contained in the directory. We use npm to install the packages needed to build the project.

$ npm install @mfgames-writing/format @mfgames-writing/clean @mfgames-writing/epub2

The @mfgames-writing/format provides the command-line tools and does the work. There is also the “clean” theme and a third for the output (EPUB2).

Once these install, running the output should work with the following command.

$ npx mfgames-writing-format build
... lots of colorful output
$ ls *.epub
test-project-0.0.0.epub

If all goes well, you'll see test-project-0.0.0.epub in the root directory. It won't pass epubcheck check (we are missing a few things), but this is the basics of the publishing system.

The Evolution of MfGames Writing

2018-08-21T05:00:00Z

If I wrote the reasons and methods I got to this point, I haven't either tagged properly or never got around to writing it. Regardless, here is a very brief history of these tools and how they evolved up to this point. Hopefully it will explain why I did certain things and which problems I was trying to solve.

Series

I appear to be writing a short series of post about the tools I use for publication and writing.

Semantic Versions and Releases: Why semantic versioning helps with the writing process.
Evolution of MfGames Writing: A brief history and reasoning behind the tools.
First Steps Using MfGames Writing: Starting a new project with MfGames Writing.
Adding Content to MfGames Writing: Adding front and back matter to novels.
Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.
Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.
Theming for MfGames Writing: A light introduction on how to customize the output.
Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.

Markdown and YAML

Let's start with the start of this: I write in Markdown. I've written a few posts about the process and reasons, so I won't go into those. If you want a short version: text files don't corrupt and source control means I can work across multiple machines without accidently deleting an entire chapter.

The problem with Markdown is that most people don't want to read a text file. It has no formatting. It doesn't look pretty. It also doesn't go into books very well. Normally, to get around this, many authors use tools like Microsoft Word or Scrivner (which I recommend to non-technical writers) which allow for writing a story/novel and then formatting it into something usable for readers. I don't use Scrivner myself because it uses binary files (doesn't play well with Git) and doesn't run nicely on Linux (my primary platform).

Most people aren't burdened by these self-inflicted limitations.

I also use YAML to keep track of useful information about the page.

Model-View-Controller as Applied to Writing

In development, there is the concept of the Model-View-Controller. This consists of three parts:

The model is the logic and data.
The view renders/formats the model into something useful.
The controller changes the model.

The nice thing is that when writing in Markdown, I don't worry about formatting. I have four basic things: bold, italic, blockquotes, and section breaks. I don't really need anything else because of my writing style. I also think this contributes to my productivity, if I can't play with it, I can't distract myself. This is basically the model of my novel; the chunky bits in a text file that constitute the contents of the novel.

The view is what takes that Markdown + YAML file and generates something useful or pretty out of it. That can include:

Online Reading: HTML pages for reading online
PDF: PDF for print-ready output.
EPUB: Ebook formats for those who like readers.

The controller is what I write with the novel. I basically use Atom or Notepad++. This is also where Author Intrusion will fit into the process.

The nice part about having the contents (the model) and the output (the view) is that they are completely separate in this step. If I decided to change the font of my books, I just update the view and reformat the books. All the words and everything will come over and I will just have a new appearance. The same if I want to change how epigraphs are formatted or want to support a new ebook format.

Automation

Having a separate view from the model also means I can automate the process of formatting and typesetting the book. This is useful because it is a tedious job formatting the book. Yes, I don't do it that often but I also like consistency. So as my “style” evolves, I also find that I go back and tweak the previous versions for consistency. For example, I moved from a 2 cm lead on the chapter pages to a 4 cm because I thought it looked better. I also made the epigraph a little larger because I can read 8 pt font easily but not everyone else.

Automation also takes out the little details that throw me. I know some books that start with “Chapter 1” and “Chapter 2”, but somewhere in the middle they switch to “Chapter Twenty-Two”. Or sometimes using typographical quotes and other time using ticks. These are things that will throw out a detail-oriented person (like me) but I also feel make an unprofessional looking book.

These are things that Microsoft Word and Scrivner will do, some better than others. I want the same thing because my goal is to create as technically correct and beautiful book as I can possibly create.

MfGames Writing sits in this place, to automate the formatting and handle all of the little details while giving me the ability to focus on one thing—the words—and let computers handle the rest.

Early Versions

When I started this epic quest of trying to automate formatting and typesetting, I started with a simple program: Make. It basically used a few tools in Linux to transform the Markdown into something usable. Shell scripts ended up not being enough, so I supplemented those with a combination of Perl, Python, and C# tools. Each language had its strengths and weaknesses (the biggest is how they handle Unicode since I use accents heavily).

Where most people write a little game or card program with new languages, I write a publication system.

At the time, those languages didn't have good support for installation. Both Perl and Python have a tendency to install packages for the entire machine or a user. C# didn't have NuGet at the time, so it was copying files around (plus it was always the weakest of the three implementations). This meant I could only have one version at a time, so I basically directed everything to my /home/mfgames-writing/ folder and used that.

Which… was good enough until I started changing my build process. I've been using this for about a hundred books and short stories now. The novels were the hardest because I would add features for fancy headers (like the PlayStation controller on top of Fangs for Nothing) or epigraphs in my page. As I went along, the process got better.

It also broke older versions. A template would change, or the flow. I started having to have v1, v2, and v6 hanging around because of the drastic changes as the tools got better at formatting and typesetting. Every time I would rewrite it, I would have slightly different input because of the language changes, and that would break older ones.

That's when I realized I needed to be able to “pin” a version of the tools for a specific book.

The latest attempt at C# was to take advantage of NuGet but then I realized the default NuGet implementation also created a machine-wide version.

I'm aware there are virtual environments for Python. I didn't know about them at the time. After I've played with them, I still find them a bit cumbersome compared to using the approach I went with.

TypeScript and NPM

I finally figured it out when I started playing with TypeScript. I like learning new languages and I was looking for a replacement for my scripting language (I'm very good at Perl and pretty good at Python). One of the tools with JavaScript based languages is NPM which is the package management (much like NuGet).

The beauty of NPM is that it is self-contained inside the development projects. All of the required files go into node_modules/ which means that each project is a folder that contains everything it needs to run and build. This also means that different projects can have different versions of the same library.

This ended up being the solution I went with. A self-contained, project-level packaging system that would let me evolve the tools for one version but still keep another novel at an earlier vesion so it doesn't break.

Fast-forward about a year of coding and tooling and we get to where we are, MfGames Writing.

State of writing with Markdown, YAML, and Git 2017

2017-05-24T05:00:00Z

A year ago, at one of my more successful panels at WisCon, I was on a panel with K. Tempest Bradford and Kristine Smith talking about writing processes. I got to see a lot of cool gadgets but I also got a chance to talk about my processes of writing with Markdown, YAML, and Git.

I'm not going to WisCon this year, but I thought this would be a good opportunity to write up where I am using all those technologies to write, both for my personal projects and as a publisher.

This is a long post in a single part. If you've read my blog, there is some rehashed information.

Markdown and YAML

Probably the one part that hasn't changed is my use of Markdown and YAML. I originally used Creole with a makeshift header but after playing with Jekyll for a while, I jumped whole-heartedly on Markdown with a YAML header.

Below is an example of one of the files. I'll be referencing it a bit in this post so please forgive the length.

---
availability: public
when: 1471/3/28 MTR
duration: 25 gm
date: 2012-02-18
title: Rutejìmo
locations:
  primary:
    - Shimusogo Valley
characters:
  primary:
    - Shimusogo Rutejìmo
  secondary:
    - Shimusogo Hyonèku
  referenced:
    - Funikogo Ganósho
    - Shimusogo Gemènyo
    - Shimusogo Chimípu
    - Shimusogo Yutsupazéso
concepts:
  referenced:
    - The Wait in the Valleys
purpose:
  - Introduce Rutejìmo
  - Introduce Hyonèku
  - Introduce naming conventions
  - Introduce formality rules
  - Introduce the basic rules of politeness
summary: >
  Rutejìmo was on top of the clan's shrine roof trying to sneak in and steal his grandfather's ashes. It was a teenage game, but also one to prove that he was capable of becoming an adult. He ended up falling off the roof.

  The shrine guard, Hyonèku, caught him before he hurt himself. After a few humiliating comments, he gave Rutejìmo a choice: tell the clan elder or tell his grandmother. Neither choice was good, but Rutejìmo decided to tell his grandmother.
---

> When a child is waiting to become an adult, they are subtly encouraged to prove themselves ready for the rites of passage. In public, however, they are to remain patient and respectful. --- Funikogo Ganóshyo, *The Wait in the Valleys*

Rutejìmo's heart slammed against his ribs as he held himself still. The cool desert wind blew across his face, teasing his short, dark hair. In the night, his brown skin was lost to the shadows, but he would be exposed if anyone shone a lantern toward the top of the small building. Fortunately, the shrine house was at the southern end of the Shimusogo Valley, the clan's ancestral home, and very few of the clan went there except for meetings and prayers.

I seem to be moving from the .markdown to .md extension. It's minor but I haven't quite jumped on it; I've only done it for the last two new projects.

Atom

My current editor of choice for novels is Atom with a backup of Emacs. Unlike Emacs, Atom has standard keyboards and looks good with the fonts I use. It is a little slow, but for the most part, works pretty well for belting out words.

I'm writing some extensions for it including a modified spell check (project-spell) that can handle project-specific dictionaries (language.json). This lets me tell the spell-checker not to mark character names as misspelled.

Later, I'll integrate my work into Author Intrusion with this but that isn't nearly ready for prime time even for my purposes.

Metadata

Over the last year I noticed I am putting lot more information in the header than before. Some of it is done long after the project but even while writing, I'll put in notes about the characters, the reason I'm writing the chapter, time of day, even the outline which I remove as I write.

I just redid the Fedran website, otherwise I'd show you how the characters and related fields show up in the sidebar. I'll get that worked in the next few months, but it let me cross-link things into my wiki. Having it in the header also means I can query it to get a summary of the page using a TypeScript tool I wrote called markdowny.

$ markdowny table *.markdown -f _basename title when
| _basename           | title                   | when          |
| :------------------ | :---------------------- | :------------ |
| chapter-01.markdown | Rutejìmo                | 1471/3/28 MTR |
| chapter-02.markdown | Confession              | 1471/3/28 MTR |
| chapter-03.markdown | Morning                 | 1471/3/29 MTR |
| chapter-04.markdown | Rivals                  | 1471/3/29 MTR |
| chapter-05.markdown | Decisions               | 1471/3/29 MTR |
$

Another nice thing about markdowny is that it also lets me show those YAML lists in a useful manner.

$ markdowny table -f _basename characters.secondary
| _basename           | characters.secondary
| :------------------ | :-------------------------------------------------
| chapter-01.markdown | Hyonèku
| chapter-02.markdown | Somiryòki, Tejíko, Gemènyo
| chapter-03.markdown | Desòchu, Gemènyo, Hyonèku, Mapábyo, Opōgyo, Panédo
| chapter-04.markdown | Desòchu, Karawàbi, Tsubàyo
$

I also has writing synopsis at the end, so what I can do is put each individual chapter in its header (in my case summary) and then use markdowny to pull them out. That way, I can write the synopsis as I go and not be overwhelemed at the end.

$ markdowny sections *.markdown | head -n 5
# Rutejìmo

Rutejìmo was on top of the clan's shrine roof trying to sneak in and steal his grandfather's ashes. It was a teenage game, but also one to prove that he was capable of becoming an adult. He ended up falling off the roof.

The shrine guard, Hyonèku, caught him before he hurt himself. After a few humiliating comments, he gave Rutejìmo a choice: tell the clan elder or tell his grandmother. Neither choice was good, but Rutejìmo decided to tell his grandmother.
$

The final bit is word counting. Because of the YAML metadata, I can't get good word counts using most tools becaues the header skews the number. So I use the tool to get me counts of the content minus the YAML header.

$ markdowny count *.markdown
chapter-01.markdown:   1520
chapter-02.markdown:   2144
chapter-03.markdown:   2905
chapter-04.markdown:   1173
chapter-05.markdown:   2570

I also have an alias for mdwc to the count since I use it pretty heavily.

Directory Structure

I'm leaning toward a semi-standard layout for my projects. Some of this was built up over the last few years but even my short stories have been migrating over to it.

* `README.md` contains a summary of the project and my tasks list. * `chapters/` contains all the chapters in `chapter-01.md`. I have never had a project with more than 89 chapters, so I stick with two digit zero pad so it remains alphabetical. * `frontmatter/` contains the frontmatter (dedication, legal) files. * `backmatter/` contains the backmatter (colophon, about, also by). * `characters/` has one Markdown file per character, for future use with [Author Intrusion](/tags/author-intrusion/). * `covers/v1`: The first variant of the cover. I have a couple and each one goes into a different `vX` folder so I can keep them apart.

Even with short stories, I have a chapters/ folder. I'm just picking up a “muscle memory” of going into chapters. I consider switching to putting things into a src/ like most of my programming projects I just haven't because I couldn't see the advantage other than having a neat folder.

Renumbering

One drawback of having chapter-01.md, chapter-02.md, etc is when I have to add a new chapter. I wrote a renumber script that lets me inject a chapter.

$ ls -l
total 8
-rw-r--r-- 1 dmoonfire dmoonfire  22 May 24 08:06 chapter-01.md
-rw-r--r-- 1 dmoonfire dmoonfire 127 May 24 08:06 chapter-02.md
$ touch chapter-01a.md
$ renumber *.md
$ ls -l
total 8
-rw-r--r-- 1 dmoonfire dmoonfire   0 May 24 08:07 chapter-01.md
-rw-r--r-- 1 dmoonfire dmoonfire  22 May 24 08:06 chapter-02.md
-rw-r--r-- 1 dmoonfire dmoonfire 127 May 24 08:06 chapter-03.md
$

As you can see, using 01a puts it before the first chapter. It works the same with deleting chapters. Sadly, this script isn't very clean but eventually I'll merge it into markdowny.

Git

One of the biggest advantages of using Markdown (text files in general) is how well is plays with source control, Git in specific. I can't stress how much Git has helped me over the years. I still remember accidentally losing a chapter because I copied the wrong file in the wrong direction. Or the painful way of tracking versions (chapter-01a.doc, chapter01b.doc, chapter-01b-final.doc, etc.).

At one point, I had a single master repository which all completed pieces in the master branch and branches off that for the works-in-progress. That actually worked out pretty well… until I started publishing. While working with text is great, the binary files of working through the covers ended up creating huge repositories which took forever to clone.

Last year, I hadn't quite split apart all of my repositories but that's pretty much done now. This also meant I can give access to editors and others to a single novel without exposing the other ones. I don't have to worry about Git modules or jumping through hoops to have the binaries.

I tried LFS (large file system) for a while but then dropped it. With it not baked in, it was a little difficult to coordinate with continuous integration servers. I think in another year or so, it could work out fairly well. I also suspect it has to do with my comfort with LFS more than technical issues.

Gitlab

Related to LFS is where I'm hosting Git. A year ago, I used Git over SSH with my ISP. However, Gitlab has been fantastic, both as their hosted environment and also on an instance running on Dreamhost (my provider). There are a lot of reasons to use Gitlab as a writer (and a publisher).

Probably the biggest is private repositories. Even if I didn't how it on my own site, being able to make each of the novels private is fantastic. Github, which is only slightly slicker, doesn't allow private repositories. Even if they did, I have over a hundred projects when you count works-in-progress, completed novels, and websites. Hosting that on Github would be expensive.

I also have a Gitlab instance. The Broken Typewriter Press business and my private novels on there. This gives me full control over the site (though I trust Gitlab) and I like it. The only people on the site are me, authors, and editors.

Like Github, Gitlab has some pretty nice features. When publishing the last few books, we've used the issue tab for authors to ask for me to order books or make corrections to their book. I can give them access to make their own typo corrections, which reduces the amount of work. At the same time, because of Git, I can be doing other changes and I don't have to worry about losing or screwing up their work. For the latest book, Sins of Intent, the author did a fantastic job of using the features.

I use the milestones for the various tasks of publishing a book. That way I can set up milestones for when the book has to be done, when the release it done, and various conventions. The issues are assigned to the milestones for both the author and myself and we can track what is missing or remaining to do. In effect, we can use a public project management to handle a book's release.

I normally turn off wiki and snippets though, they usually don't help.

One of the things with using Gitlab this way is that the author has full access to the raw files. Typically, I get a Word document and break it into individual Markdown files. Corrections are done against the Markdown as my baseline format for everything else. If they want to leave BTP or something catastropic happens, they can have exactly what I published. This is because I've been through cases where local edits were made but I couldn't get them back myself. In this case, they get and can see everything I do.

Which leads to the last feature of Gitlab which I use heavily, their Continious Integration service. This was difficult to set up at first, mainly because I had to educate myself. For most novels, the .gitlab-ci.yml looks like this:

image: dmoonfire/mfgames-writing-js:0.4.0

stages:
    - review
    - publish

review:
    stage: review
    only:
        - master
    tags:
        - docker
    script:
        - git lfs pull # Yeah, I'm using LFS with this project.
        - npm install
        - npm run build
    artifacts:
        expire_in: 1 week
        paths:
            - "*.pdf"
            - "*.epub"
            - "*.mobi"

publish:
    stage: publish
    script: "echo published"
    when: manual
    artifacts:
        paths:
            - "*.pdf"
            - "*.epub"
            - "*.mobi"
    dependencies:
        - review

Every time I check in, this rebuilds the project. For the little changes, the “staging” lets me and authors test the results of the file which show up underneath the build. They expire after a week but that's okay. “Releases” are done by moving a task to “publish”.

The nice part is I can grab the resulting PDF, MOBI, and PDF about 5-10 minutes after I check in. This is good to ensure that not only I have a reproducible build and also that I won't lost the ability to recover the output if my laptop catches on fire (it is almost seven years old, four majors cracks, and has a hernia).

This ability to see the final version is great because me or another author can make changes and see it, test it, and make sure it is exactly what they want. If I change formatting, I can see the results without overloading my computer.

mfgames-writing-js

Of course, using CI requires some way of actually formatting the books. Markdown is fantastic for some things but there are relatively few tools to format it into good-looking EPUB, MOBI, and PDF files.

Over the years, I have many variants of this. The original few were based on Makefiles and various Python or Perl tools. I've also integrated pandoc into the mix. Nothing ever worked quite the way I wanted for what I considered to be a “proper” book. Files were put in the wrong place, dedications don't need titles, making sure chapters start on the right. There were little things that pushed me closer to making something more specific.

I also had a problem that extending the features for a later book broke the generation of older books. I had to start over or rebuild (usually copy/paste/edit). Now this is a problem that has been addressed by NuGet and NPM, having the build process inside the project instead of using shared programs and libraries.

Between this and using Gitlab CI runner, I decided to create a new framework that specifically was geared toward having specific versions that could be used to reproduce the book even if the underlying libraries were updated. I ended up using NPM, mainly because I'm learning TypeScript for the last few months. It also had a better story for installing (npm install), wasn't whitespace-based, and specifically designed to be isolated to the project.

The result is mfgames-writing-js, a framework for creating EPUB and PDF files from Markdown files. This entire thing is controlled by a single file checked into the Git repository.

editions:
    epub:
        format:             mfgames-writing-epub
        theme:              ./lib/efferding
        outputDirectory:    .
        outputFilename:     sins-of-intent-{{edition.version}}.epub

    pdf:
        format:             mfgames-writing-weasyprint
        theme:              ./lib/efferding
        outputDirectory:    .
        outputFilename:     sins-of-intent-{{edition.version}}.pdf
        isbn:               978-1-940509-24-2

metadata:
    title:      Sins of Intent
    series:     Cletus Efferding
    author:     Randy Roeder
    language:   en

contents:
    - element: cover
      source: covers/v2/front.jpg
      linear: false
      exclude:
        editions: [pdf]
        toc: true
    - element: bastard
      source: frontmatter/bastard.html
      linear: false
      exclude:
        toc: true
    - element: title
      source: frontmatter/title.html
      linear: false
      exclude:
        toc: true
    - element: legal
      source: frontmatter/legal.md
      liquid: true
      linear: false
      exclude:
        toc: true
    - element: dedication
      source: frontmatter/dedication.md
      linear: false
      exclude:
        toc: true
    - element: toc
      linear: false
      exclude:
        editions: [pdf]
    - element: chapter
      number: 1
      directory: chapters
      source: /^chapter-\d+.md$/
      start: true
      page: 1
      pipeline: &pipelines
          - module: mfgames-writing-hyphen
            exceptions:
                # https://www.hyphenation24.com/word/driving/ says "driv-ing"
                - driving
                - drive
    - element: acknowledgement
      source: backmatter/acknowledgments.md
      pipeline: *pipelines

This actually has turned out better than I expected. Yeah, I had trouble with using WeasyPrint for PDF generation, but it has almost all the features I needed to do right-side chapter openings, first page headers differently, and stitching everything into a single PDF with properly embedded fonts.

The one thing I can't do is create Smashword's Microsoft Word. I'm still trying to decide if it is worth it.

Website

I also release chapters every week. I used to use WordPress but it got cumbersome to add new chapters so I switched to a static site generator over the years. I started with DocPad, then Jekyll, and then extended Jekyll with Python and Perl programs to handle my requirements.

Well… I decided to write my own. This uses the same input files as the rest of my system (Markdown + YAML). It also lets me change a header to release the chapter and include it into the build.

One of the complexities I had to figure out is the different repository for each novel. In the building process, I actually clone every project I'm publishing or have published, then use scripts to pull them into a single website before generating it. Jekyll couldn't handle it easily which is one reason I created my own.

The tool for making the website is CobblestoneJS but it has no documentation and I'm still fumbling through it. On the other hand, it handles the ten or so repositories needed to build fedran.com without my conlang, world data, and individual novels.

As part of the weekly release, I've gotten the tasks down to:

Update a header in the YAML (or use the scheduler to do it once)
Write a blog post about the chapter
Copy the chapter and the post over to Ello
Copy the chapter from fedran.com to Wattpad
Copy the post over to Patreon
Run the diaspora sync post

It takes about 1-2 hours which is much shorter than the 4 is used to be.

Summary

Well, there it is, my current state of writing with Git. You have everything from writing the chapters, supplying metadata, how to store it, various tools for getting through the publication process, and even formatting it for the various vendors.

I've gotten lost trying to automate and simplify a lot of this, but I think the current state is mostly usable by others and has been pretty solid for my own needs. I'm sure I'll improve and expand on it.

If you have question or comments, please ask. I love talking about processes, looking for improvements, or explaining in more detail why I do these things.

Thank you.

Writing with Markdown and YAML

2016-10-31T05:00:00Z

Four years ago, I wrote a Reddit post about why I switched to pure text files instead of using Microsoft Word, Libreoffice, or even Google Docs. The short answer is: text is forever and it is really hard to corrupt a text document.

There was a secondary reason for using text format and that was because I like keeping track of metadata while I write. A lot of authors do, actually, but I didn't care for the more common methods of keeping a spreadsheet in sync with the writing or getting Scrivner to behave. At the time, it wasn't so great, from what I've seen, a lot of what I do here can also be done in that program. That said, I still don't use it because of their file format doesn't play well with Git (I wrote a post about that last year).

Now, I have been doing this for a while. However, last year, someone at WisCon asked me about it and I thought I made the post but apparently I didn't. Today, someone asked me the same question ("how do you use Markdown and YAML to write?") and I went to point them to my article I didn't write last year and realized that I didn't write it.

So… here it is.

Format

When I write, the basic file looks like this:

---
title: Rutejìmo
---

> When a child is waiting to become an adult, they are subtly encouraged
> to prove themselves ready for the rites of passage. In public, however,
> they are to remain patient and respectful. --- Funikogo Ganósho, *The
> Wait in the Valleys*

Rutejìmo's heart slammed against his ribs as he held himself still. The
cool desert wind blew across his face, teasing his short, dark hair. In the
night, his brown skin was lost to the shadows, but he would be exposed if
anyone shone a lantern toward the top of the small building. Fortunately,
the shrine house was at the southern end of the Shimusogo Valley, the
clan's ancestral home, and very few of the clan went there except for
meetings and prayers.

The > is a block quote which I use for my epigraphs. There is a quote at the beginning of every chapter for world building. Everything after that is Markdown. I use *italics* and **bold** but otherwise not a lot.

Front Matter

The really important part is the YAML front matter. I picked YAML because it was really easy to read and change. It also parsed very well by computers. The only thing I don't like is that it doesn't like tabs… but can't be perfect.

There is a lot more in this header than I showed above. For the first chapter of Sand and Blood, the front matter looks like this:

---
availability: public
when: 1471/3/28 MTR
duration: 25 gm
title: Rutejìmo
locations:
  primary:
    - Shimusogo Valley
characters:
  primary:
    - Shimusogo Rutejìmo
  secondary:
    - Shimusogo Hyonèku
purpose:
  - Introduce Rutejìmo
  - Introduce Hyonèku
  - Introduce naming conventions
  - Introduce formality rules
summary: >
  Rutejìmo was on top of the clan's shrine roof trying to sneak in and
  steal his grandfather's ashes. It was a teenage game, but also one to
  prove that he was capable of becoming an adult. He ended up falling off
  the roof.

  The shrine guard, Hyonèku, caught him before he hurt himself. After a
  few humiliating comments, he gave Rutejìmo a choice: tell the clan elder
  or tell his grandmother. Neither choice was good, but Rutejìmo decided to
  tell his grandmother.
---

As you might be able to tell, I keep a fair amount of information in the top of my files. This basically contains references to characters, locations, why the chapter exists. I could also put tone (action scene, introspection), scene types (scene or sequel), or anything else I want.

The nice reason for this is when I rename or arrange the file, the header goes with it. If I need to list all chapters that have a character references, I can with a simple “find in files”.

I also have it tied into a future plan of creating a master book that has every novel I have with the chapters interspersed so the entire world is chronological. I think it will be cool once I start overlapping these stories. I also plan on having locations so I can show them on the map, if I ever write it down.

Tools

Now, “find in files” can only go so far. Over the years, I've written a number of tools to help me work with these type of files. I've tried it out in Python, Perl, C#. Most of them have bits and pieces that work, I actually use a combination of all of them.

Today, I realized I was tired of working with tools from three different languages so I decided to add a fourth language that works with some future plans I have: Typescript. So, over my lunch and while kids were showing up the door for treating, I got the first version of Markdowny written.

npm install --global markdowny

markdowny basically duplicates the most common functionality I use from the other languages and puts it into a fairly generic framework that can show off the usefulness of using YAML.

Counting

The major drawback of YAML headers is that it is hard to figure out how many words something is. The built-in word count features of most editors counts the large YAML. So, markdowny has a simple feature for giving me a chapter-by-chapter breakdown of word lengths without the headers.

markdowny count chapters/*.markdown --total

The output is pretty simple actually. I mostly use this to select the chapters I want to submit to the writing group; we have a four thousand word limit with submissions.

chapter-01.markdown:  1,520
chapter-02.markdown:  2,144
chapter-03.markdown:  2,908
chapter-04.markdown:  1,173
chapter-05.markdown:  2,570
Totals             : 10,315

If you don't include the --total parameter, you don't get the Totals line.

It is also useful for other submissions or finding out how big the novel is.

markdowny count chapters/*.markdown --total | tail -n 1

Synopsis

The summary key is a useful one. I write the chapter synopses as I go. It isn't nearly as overwhelming then trying to write the entire book in a single shot. That means when I have to come up with a chapter-by-chapter synopsis, I can do it fairly easily.

markdowny sections chapters/*.markdown > synopsis.markdown

The above command line goes through all of the chapter files, pulls out the title and summary block and creates a new Markdown file that contains all the summaries.

# Rutejìmo

Rutejìmo was on top of the clan's shrine roof trying to sneak in and
steal his grandfather's ashes. It was a teenage game, but also one to
prove that he was capable of becoming an adult. He ended up falling off
the roof.

The shrine guard, Hyonèku, caught him before he hurt himself. After a
few humiliating comments, he gave Rutejìmo a choice: tell the clan elder
or tell his grandmother. Neither choice was good, but Rutejìmo decided
to tell his grandmother.

# Confession

Rutejìmo returned to his grandmother's cave. He tried to sneak around,
but she caught him entering. When she asked what he had done, he told
her that he had been up at the shrine. His grandmother got upset and
began to berate and beat him, chasing him out of the cave.

Gemènyo interrupted her beating to ask if it was justified. When Tejíko
explained the reason, he agreed but she had lost her anger. As she went
back into the cave, Gemènyo sat down with Rutejìmo and asked
questions. The discussion came to the upcoming rite of passage and how
Rutejìmo would gain magical powers once he experienced it. Gemènyo
wouldn't give details about the rite, but he did tell Rutejìmo that
magic wouldn't change him.

Rutejìmo snapped back, saying that Gemènyo wasn't the greatest warrior
in the clan. Gemènyo didn't seem bothered, but suggested that if
Rutejìmo wanted to be the best, he needed to do something. Rutejìmo
agreed, even knowing he wouldn't.

# Morning

Rutejìmo was sent out to get breakfast for his grandparents. Along the
way to the valley floor, he got distracted by Mapábyo, Hyonèku's adopted
daughter, riding on top of Opōgyo, one of the clan's mechanical dogs
used for dragging heavy loads around. They talk for a short while before
she asked him to come with her to take the load up to her father, who is
at the lookout above the valley entrance.

Even though Hyonèku had caught him the night before, Rutejìmo
agreed. When they get there, Hyonèku started to read a naughty letter
from his wife before Gemènyo interrupted him. Embarrassed, Hyonèku
focused on his daughter while Gemènyo talked with Rutejìmo more.

A little bit of effort ahead of time and I have a chapter-by-chapter breakdown without getting overwhelmed.

Table

The final thing markdowny does is give me that summary table. I use this to compare chapters with each other and try to get an idea of what is going on with the “big picture”.

markdowny table chapters/*.markdown markdowny table -f _basename title _words characters.secondary -t File Title Words:rs "Secondary Characters"

The -f selects the fields inside the YAML. The -t lets me have a pretty name for them. The results happen to be a Markdown table, so it looks a bit prettier if I just show the end results.

Filename	Title	When	Duration	Location
chapter-01.markdown	Rutejìmo	1471/3/28 MTR	25 gm	Shimusogo Valley
chapter-02.markdown	Confession	1471/3/28 MTR	35 gm	Shimusogo Valley
chapter-03.markdown	Morning	1471/3/29 MTR	2 m	Shimusogo Valley
chapter-04.markdown	Rivals	1471/3/29 MTR	1 m	Shimusogo Valley
chapter-05.markdown	Decisions	1471/3/29 MTR	3 m	Shimusogo Valley

Okay, the format is based on the browser, but overall it is somewhat useful. Of course, the early chapters of that book is in the same location. Likewise, I could show the POV (which is useful for some writers) but it doesn't help much since I write single POV novels.

markdowny table chapters/*.markdown -f title _words:r characters.primary purpose

This gives me a decent summary of how long each chapter is, who the main character is, and a list of reasons for that chapter. From there, I have a decent idea of where I need to trim or expand the piece.

title	_words	characters.primary	purpose
Rutejìmo	1520	Shimusogo Rutejìmo	Introduce Rutejìmo, Introduce Hyonèku, Introduce naming conventions, Introduce formality rules
Confession	2144	Shimusogo Rutejìmo	Introduce Tejíko, Introduce Gemènyo, Introduce Rutejìmo lack of talent, Introduce style of punishment, Second chapter to get used to names

Other Tools

I have some other tools. There are ones that take these files and create EPUB, MOBI, and PDF files. I've been using those programs since Sand and Blood was first published. There are others that I've started wht convert the in-world dates (MTR in the above example) into a master timeline or identify every chapter of every novel that happen in a given location.

I also use these chapters to build up the various pages on my Fedran website so patrons can read ahead (that is the availability in the YAML).

Using Markdown

Using Markdown isn't fancy but I find it is easier. Since there is so little control over format, it is easier to focus on the words that need to be written instead of making it pretty.

Also, it is a text file. It plays very well with Git and source control. I don't have to worry about an editor corrupting the file and losing it, it takes up little space, and I can easily switch between Notepad++, Emacs, or Atom without a second thought. I can use the best tool for the job.

Messing with Markdown

2014-11-23T06:00:00Z

I have a lot of projects in my head, which is one reason why I try to avoid starting new things while working on others. It got too easy to start a project and then never finish it when I got bored or just moved on.

At the same time, many of my ideas don't “stick.” They seem like they will, but then it either doesn't work out, the ideas don't really gel, or simply I lose the passion. Some of them, like my old Exalted webcomic, haunt me for years with the nagging voice of “finish me” but I keep not doing it. Others end up being bigger projects, but suffer through constant revisions as I try to figure it out.

I do have some major projects that I haven continued to update and maintain quite a few years after I start them. One of the biggest is MfGames Writing Python which I started in 2010 when my father said I'd love Python. Even though I've long-since decided that I don't like Python, I've still been writing updates to the tools in support of my own writing efforts.

Like Glorious Saber, I've been thinking about converting my Python writing tools over to C#, but never really got to it. I had a couple stabs at it but nothing really “stuck.”

The Side Project

After ICON, something Jim C. Hines and Scott Lynch said during their “Beyond SF 101” panel was still echoing in my head.

Write more words and don't be a dick.

Now, Sand and Ash is still stuck in limbo, so I couldn't really do anything with that. So I had this idea for a writing project that would be complimentary to my Fedran world and let me get different ideas out. It involved smaller pieces, short stories and essays and lessons, so I was trying to figure out how to pull it all together.

For some reason, my Python tools just choked. They are optimized toward writing novels and single-file DocBook 5 files, but not a multitude of smaller DocBook XML files which would these individual pieces.

I started to look into my C# version, which had a different “gather” utility and, once again, realized that I probably wasn't that far off from getting the C# version working with its flaws and maybe move away from the Python.

One of my major difficulties with Python is that it doesn't handle UTF-8 characters natively. I seem to have a lot of non-ANSI characters in my desert world (macros are a killer) and it kept choking on them.

MfGames Writing CIL

So I shifted from my side project to MfGames Writing CIL which is the C# version of the Python tools “plus” additional functionality. The biggest is that I wrote the Python around Creole instead of Markdown, which is the markup language I've migrated my writing to.

While I was working with writing up a Markup conversion utility for the tools, I came upon CommonMark which is an attempt at a well-documented specification. I figured I could use that to help guide my effort on the conversion utility.

It didn't take long before I realized that I was duplicating my work with Author Intrusion for handling Markdown. Usually when that happens, I figure I should start up a new project to handle the common logic and write it once.

MfGames Text Markup CIL

And then I moved from MfGames Writing CIL to MfGames Text Markup CIL. This is an attempt to create a single, centralized reader (and eventually writer) of markup languages in general and Markdown in specific.

There were a couple reasons I went this project instead of another library:

Most Markdown libraries only convert to HTML, which meant I was parsing HTML to get into Author Intrusion or DocBook format. I wanted something that had an intermediate output that was ideal for converting to other formats.
Again, most libraries seem to load the entire Markdown file into memory at once (mainly because of the deferred links) and then writing it out. I haven't tested this completely, but I already know that I have 640k word series that have to be parsed; I do not want to have this loaded into memory. This implies a callback interface (SAX verses DOM).

I decided to write this in a similar style to C#'s XmlReader. Instead of loading everything into memory and then writing out the results, it just translates the Markdown file into element types.

// Loop through the Markdown and process each one.
while (markdown.Read())
{
    switch (markdown.ElementType)
    {
        case MarkupElementType.BeginDocument:
            this.WriteBeginDocument(xml);
            break;

        case MarkupElementType.EndDocument:
            xml.WriteEndElement();
            xml.WriteEndDocument();
            break;

        case MarkupElementType.BeginMetadata:
        case MarkupElementType.EndMetadata:
        case MarkupElementType.BeginContent:
        case MarkupElementType.EndContent:
            break;

        case MarkupElementType.BeginCodeSpan:
            this.WriteForeignPhrase(markdown, xml);
            break;

The system seemed to work out pretty well for my test cases, but when I started to throw “real” chapters at it, it started to crumble. Not from the foundation, but simply because I didn't write a good enough Markdown parser to translate them.

CommonMark

And here is where I started down the rabbit hole. The callback system worked great, but I needed to get my parser to be competent enough to handle what I wrote. Once I get that, converting to DocBook is trivial (as the above example probably shows).

A few days ago, I noticed that the CommonMark spec was a Markdown file with some magic for handling the input/output examples. Well, I could write a bunch of unit tests or… I could write a program that converted the 500+ examples into unit tests for me.

Why write things out by hand when I can write a program to do it for me?

Last night, I finished converting most of the unit tests over. Now, I just have to solve 508 unit tests, or at least 300 of them.

Rabbit Holes

I'm pretty sure this is going to be overwhelming but I think it will still further my goal of finishing up my writing tools and get back to my side project. We'll see how it ends up, but I still have a mountain to climb.

This is also the reason I haven't really posted for a few weeks to. I was lost in a rabbit hole.

Author Intrusion - Names

2014-08-07T05:00:00Z

I was slowly making my way through the Markdown and DocBook file processing when I hit an interesting snag: names.

Author names are a strange thing. It would be easy to just stick with the Western approach, which is what Amazon uses in their KDP program: first and last name. Smashwords does the same but they add suffix. No mention of title, which some people really hang up on (I almost went through a Ph.D. program just so I could be Dr. Moonfire).

Complexity of names

The problem is that I work with Japanese people who have family name first and given name second. There are others too, I just happen to know Japanese culture a little better than most others. Most websites I know don't actually handle this because they assume the first/last name approach.

You have some cultures where many people have two last names, one from the mother and one from the father.

The other interesting problem comes in with single name authors, let's go with Madonna or Teller for examples. Some websites allow you to go without both names, but there are a lot that don't. Google, for example, insists on something and NFN (No First Name) looks horrible when they say "Hello, NFN!".

An interesting one is Goodreads. Goodreads lets you have an account with only one name, but you can't sign up for a raffle without both names filled out.

There are also names that start with a lowercase character or have an apostrophe in the middle. Or a slew of other Unicode characters that most developers make assumptions that aren't true.

This bothers me, but that shouldn't be a surprise. One of the principals of Author Intrusion is “Author Know Best”. It means that the author knows what they are doing and we don't tell them how to work. This includes how format their own name. If they want a Unicode symbol, that's fine. I'll probably draw the line at images for authors, simply because I don't want to code that. I'm all for it, it's just low in my priority list.

Solving the problem

Normally, I use DocBook for inspiration on handling things. I figured after so many decades, they've solved most of the problems. And, in this case, I think they have.

<author>
  <personname>Madonna</personname>
</author>
<author>
  <personname>
	<honorific>Not-Dr</honorific>
	<firstname>Dylan</firstname>
	<othername role="middle">Robert Evans</othername>
	<surname>Moonfire</surname>
  </personname>
</author>

There are no specific conventions for othername, but it's better than most. It also lets me enter full names for the author and figure out how to format them later.

In my Markdown files, my author tag is usually a preferred name.

---
Title: Amazon Piece
Author: D. Moonfire
---

It was a dark and stormy tea cup...

This caused me some trouble with formatting ebooks because I occasionally need to know first name and last name to figure out sorting.

Reusing concepts

While I was in the process of figuring out names, I realized that I'm going to have the same problem with one of my future plans: characters. One of the reasons I'm writing this is to highlight character names. All of them, including possessives and nicknames. There is some other ideas around this, but I needed to handle a highly flexible approach there also.

From there, it was easy to figure out that I'm going to use the same class to represent an author name as well as character (location, objects, etc) names. And then I came up with NameInfo. This is a Dictionary<string, string> that uses interned key strings.

NameInfo name = new NameInfo();
name.PreferredName = "Shimusogo Rutejìmo";
// or name["Preferred Name"] = "Shimusogo Rutejìmo";
name["First Name"] = "Rutejìmo";
// or name.FirstName = "Rutejì";
name["Clan"] = "Shimusìgo;
name["Surname"] = "Shimusogo;
name["Nickname"] = "Jìmo";
name["Possessive"] = "Rutejìmo's";
name["Possessive Nickname"] = "Jìmo's";

NameInfo author = new NameInfo() { PreferredName = "D. Moonfire" };

When I write these out in DocBook, it will become a full XML block.

<personname>
	<firstname>Rutejìmo</firstname>
	<surname>Shimusogo</surname>
	<othername role="Clan">Shimusì/othername>
	<othername role="Preferred Name">Shimusogo Rutejìmo</othername>
	<othername role="Nickname">Jìmo</othername>
	<othername role="Possessive">Rutejìmo's</othername>
	<othername role="Possessive Nickname">Jìmo's</othername>
</personname>

<personname>D. Moonfire</personname>

In the Markdown formatter, I get this:

---
name:
  PreferredName: "Shimusogo Rutejìmo"
  FirstName: "Rutejìmo"
  Clan: "Shimusìgo"
  Surname: "Shimusogo"
  Nickname: "Jìmo"
  Possessive: "Rutejìmo's"
  PossessiveNickname: "Jìmo's"
author: D. Moonfire
---

Conclusion

Overall, this seems to work. If you have a simple name, as some authors do, it will create a neat output file and they just have to enter the name. For those who are a bit pickier about names, it will let them have a complex name or set of names. It also means that when I put in name highlighting and refactoring (I rename characters a lot, but I want to also change possessive and nicknames at the same time), I'll have the infrastructure to do so.

Author Intrusion - A black triangle CLI

2014-08-02T05:00:00Z

Over the last week, I've been working on the current version of Author Intrusion. It can get pretty discouraging since there is so much to do, so I'm focusing on a single task at a time and building out along the way so I will hopefully get a complete project by the end.

Formatting

One of the hardest things about coding is the GUI. There are so many things for getting events hooked up and the interface running smoothly. Being that GUIs are my weakest area of code, plus the state of the cross-platform .NET UI libraries is still a little weak, I'm going to work on other things.

I ended up deciding on project IO. For those who have read my blog in the post, you probably remember that I spend a lot of time working on automated the creation of ebooks and print from my source documents. Not to mention a lot of time working on creating a format that works the way I like to work.

That lead into my first goal: transforming a source project into another format. Eventually, this is going to lead into system used to create ODT, Microsoft Word, EPUB, MOBI, and everything else. Not to mention importing from other formats.

While I'm going to eventually have a GUI interface for it, I can also write a command-line interface (CLI) version which lets me get the functionality written without getting distracted by the GUI.

Progress

This weekend, I actually saw the beginning of my ideas show up. The initial framework for converting a Markdown file into DocBook 5 worked via the CLI.

$ aicli transform sample.markdown sample.xml

I went with aicli (Author Intrusion CLI) for the tool. The interface is inspired by Git with sub-commands and a somewhat flexible way of adding new tools.

The cool part is that the above command actually worked. It doesn't look like much, but there weeks of effort to get to that point. It's a black triangle moment in this development.

Formats

I arranged the file formats into, creatively enough, IBufferFormat objects. They are discovered via an IoC container (StructureMap) and represent a single type of input or output.

The initial ones to get the above code working were:

Markdown (input only)
DocBook (output only)

I'll get these to be both input and output, that way anyone can use either Markdown or DocBook files for their primary project format. I'll also use these for the output formats:

EPUB
MOBI
Libreoffice (ODT)
Microsoft Word (DOCX)

And it has options too!

Yeah, the other bit I like is the single configurable option.

$ aicli transform sample.markdown sample.xml -ORootElement=chapter

There may be more, but that puts me one step closer to replacing mfgames-creole docbook that I originally wrote for MfGames Writing Python.

Personal branching model

2013-04-03T05:00:00Z

Discipline is one of those things I struggle with. The biggest is how I use source control for both my programming and my writing projects. As we start to implement Git at works, some of my lazy habits at home are becoming more (painfully) obvious.

Noise

I commit a lot, mostly at the end of a thought. So, once I finish a chapter or a good scene, or write a series of methods and tests, I'll commit locally. I don't always give "good" messages for those commits, but enough to give me a hint of what is going on.

Checking in so I can move downstairs.

The problem with this is noise. There are a ton of check-ins that don't have useful comments. After a day or three, I won't care about those comments since they are basically just a progression to figure out something.

In the past, I usually worked on the master branch and cringed at the noise. Occasionally I would rebase to squash them down so I don't have quite as much noise, but overall I just suffered with them.

Seeing it differently

With the migration to Git at work, I realized that I need to actually use branches more effectively. Noise commits aren't so bad if they are put into a task branch and then later squashed once they get to the master root. So, I don't have to worry about going downstairs if I know those will be collapsed into a single useful commit later.

It also lets me work without too many restrictions during the creative process. Which is good since I find committing frequently to be critical to my process.

Where to put it

Now, the really big question is where to put it. I'm normally not fond of airing my "dirty laundry" as it were. I figured no one cares if I'm switching computers during the process and I don't blame them.

However, if I'm working between two machines, it seems silly to create an entirely different repository purely for the purpose of working between machines. This came up while working on the Fedran fonts.

At the moment, I decided to take a chance and just push it up to Github. There is going to be noise but it won't be on the master branch. I might prefix the branches with "drem-" to indicate they are task branches.

This lets me take advantage of Github's ability to manage branches but also to let other people join in (in the unlikely chance someone wants to help me work on something).

Novels

Since I also recently started my next novel (WisCon has a deadline for the writing group on the 15th), I decided to explore this process with the novel. I'm doing most of the work on the 'sal' branch because novel writing is pretty much independent of everything else I write. Same with the commissions and other WIPs (which I have a number of them).

Doing it this way means I can put novels in their proper place in my directory tree as a WIP instead of a dedicated folder. It's a minor thing, but I like the Fedran stuff underneath the "moonfire/fedran/" directory instead of in the "wip/" folder.

Markdown

I'm planning on switching my entire writing repository over to Markdown within a few (mumble). I need to finish a little bit of programming, but I already have the Perl script to do the conversion. If I go with this approach, then I can easily create a "markdown" branch and do the conversion until everything is ready to migrate.

The biggest hangup on the Markdown conversion is a good conversion to DocBook. I have the beginning of one in my writing tools, I just haven't finished it yet. It needs to be done so I can integrate the new code into my workflow.

New ideas hurt

I think the branching model will work out, though it feels strange. I just don't know if the strangeness is because I'm not used to it or because it is wrong. I think I'll have an answer in a few months.

Markdown

Capabilities

Fine-Grained

Approach 1 - Instance Information

Approach 2 - OpenAPI Support

References

Capabilities

Thoughts

Integrating Semantic Versioning into MfGames Writing

Series

NPM Packages

Configuration

Updating .gitlab-ci.yml

What Does This Give You

Theming for MfGames Writing

Series

HTML

Components

Recursive Templates

Styling

Stylesheet Variables

Creating New Themes

Additional Formats for MfGames Writing

Series

Using Formats

EPUB

HTML

PDF

MOBI

DOCX

Working with MfGames Writing, CI, and Docker

Series

Adding Scripts

CI Configuration

Docker

Adding Content to MfGames Writing

Series

More Than Chapters

Covers

Titles

Legal Page

Table of Contents

Pipelines

Sand and Blood example

EPUBCHECK

First Steps Using MfGames Writing (GitLab, NPM)

Series

Setting up Git

Setting up .gitignore

Setting up .editorconfig

Setting up package.json

Write a Chapter

Setting up publication.yaml

Liquid

Theme

Editions

Content

Installing Tools

The Evolution of MfGames Writing

Series

Markdown and YAML

Model-View-Controller as Applied to Writing

Automation

Early Versions

TypeScript and NPM

State of writing with Markdown, YAML, and Git 2017

Markdown and YAML

Atom

Metadata

Directory Structure

Renumbering

Git

Gitlab

mfgames-writing-js

Website

Summary

Writing with Markdown and YAML

Format

Front Matter

Tools