asdf

generator-mfgames-nix-project v1.4.1 and generator-mfgames-writing 1.0.0

2021-12-19T06:00:00Z

I create a lot of projects. I think between Github and Gitlab, I have a few hundred now. Much if it is because I started pushing toward strictly being one story per repository (OSPR). This means I have a lot of common patterns that not only do I have to set up but also maintain as my skills and knowledge evolve. The problem comes when I figure out a new pattern, I have to update a couple hundred repositories if I want to keep them consistent. Needless to say, this is one drawback of a monorepo but there are a lot of advantages of OSPR.

A lot of my inspiration comes from Larry Wall's Three Virtues of a Great Programmer:

Laziness: The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful and document what you wrote so you don't have to answer so many questions about it.
Impatience: The anger you feel when the computer is being lazy. This makes you write programs that don't just react to your needs, but actually anticipate them. Or at least pretend to.
Hubris: The quality that makes you write (and maintain) programs that other people won't want to say bad things about.

This means, instead of manually doing it, I want to find a nice automated way (virtue one) that I can script and have it run across everything at once (virtue two).

Previously, I had used Yeoman but I was wondering if I needed to break out of that to work on a “me-specific” CLI tool. That line of thought came down because so many of the tools I'm using all package their own custom CLIs. While I do enjoy writing a good CLI, I don't want to write one if I can't.

In the process of finding an alternative, I finally understood how to compose Yeoman generators together. That means I could create a generic project generator and then layer the writing one on top of it.

At the same time, I could migrate away from asdf and onto my preferred infrastructure du jour, Nix.

Side Note: Let me be honest, I'm constantly evolving my structures and they change as I learn new tools and my processes change with me. I won't be on Nix forever, but right now, Nix works out very nicely for what I want it to do.

This weekend I pulled something together and actually got a working version by tonight. This involves three projects:

mfgames-writing: My publishing framework for Markdown and YAML files.
generator-mfgames-nix-project: A Yeoman generator for creating a basic project layout.
generator-mfgames-writing: A Yeoman generator for setting up a writing project.

generator-mfgames-nix-project v1.4.1

I pulled out the basic project stuff from generator-mfgames-writing and made it a dedicated generator while improving it. It still asks a bunch of questions, but mainly it does the following:

Set up semantic-release to handle non-romantic releases. “Non-romantic” meaning automated and every time I push up to main.
Set up Conventional Commits to drive the semantic release process. This is why my check-ins start with feat:, fix:, or something other tag. That way, I can just focus on a given commit being a new feature, fixing something, or breaking everything.
Set up commitlint to make sure the commits are properly handled for the semantic releasing.
Set up Husky to make sure commitlint is automatically handled.
Configure Prettier to make everything pretty and normalized. This mostly helps if someone ever contributes, but I like things being consistent and neat. While I disagree with Prettier on many things, it does a fantastic job.
Set up a Nix flake to represent all the build tools needed to build the project.
Set up direnv to automatically ‘enter’ the environment when you go into the directory.
Write out the files needed to run Gitlab's CI/CD on push and generate all the files and automatically release.
Set up EditorConfig for normalized formatting.

That's a lot of work and it was getting tedious as I migrated/created new projects over the last month or so as I figured out the pattern. I like it, though it is opinionated. Previously, I hated that word, but this is honestly how I work and I think it's a reasonable pattern for any writing or coding project.

Running it is pretty simple:

$ npm install -g generator-mfgames-nix-project yo
$ mkdir name-of-project
$ cd name-of-project
$ yo mfgames-nix-project
... lots of prompting

Overall, I like the resulting project and I'm going to be testing it fairly heavily in the new few weeks.

generator-mfgames-writing v1.0.0

Most of the intial work for generator-mfgames-nix-project came from generator-mfgames-writing, but I ended up also expanding that over the weekend to ask better questions about theme, author information, and the release process. It works the same as the project generator.

$ npm install -g generator-mfgames-writing yo
$ mkdir name-of-project
$ cd name-of-project
$ yo mfgames-writing
... a different set of questions
$ npm run build
... lots of noise
$ ls *.epub *.pdf
name-of-project-0.0.1.epub
name-of-project-0.0.1.pdf

This one bypasses some questions that the project one does (build, test, and release questions) and adds some other ones (theme, configuration file format, title of the work), but otherwise works the same as the other. The end result is something you should be able to build and get the output immediately.

I honestly think that is really cool.

mfgames-writing

There were a bunch of versions with this, but I needed to tweak some processes because I found some bugs in these tools while writing the generators. Sadly, a “simple” fix of moving rimraf from a developer dependency to a normal dependency required me to upgrade TypeScript to the last version and accept a wall of warnings because some of the linting tools don't like the latest version.

Using Nix

2021-11-30T06:00:00Z

Apparently, it's been a few months since I've blogged about anything. That isn't to say I wasn't thinking about it. I would find myself working on a little bit of code and think “this would be something interesting to talk about” or come up on a post about elitism that I want to respond. But then… it didn't happen.

The last four month have been probably one of the hardest I've had in a long time, even over the struggles of 2020's derecho and pandemic. It started with just a bit of stress at work, but then something got dumped on top of it. And then something on top of that, then on that, and then on that. At one point, for every time I took off my task list, two more were added.

A good example is while I was fixing Partner's toilet that wasn't flushing (just a simple stalk replacement) and our youngest comes up the stairs because the downstairs toilet basically exploded (water pouring out of the bottom, ruptured wax seal, no longer flushing, soaking the carpet). I've been trying to get around to it for almost two months and just removed the carpet today to find out how much drywall damage was done.

There was a lot more, but going through them just turns into a story like Willard (2003) where the main character starts off in the dump, for a brief moment sees hope, and then plunges even further into the hole. Since I try to keep my blog posts positive, let's just go with that.

One of those bright points was getting Merger of Evil published. Shannon has always been a fantastic author and I'm glad I got to see the sequel to Minion of Evil in print. Plus, the cover is fantastic (even if I didn't do it).

My brother says my life is frequently a “that's good, that's bad” game. while I was able to do my part in getting Merger of Evil out, I was unable to finish Second-Hand Dresses or Flight of the Scions out this year, which I wanted to. The reasons are worth of a different post, but basically I had to scoop things off my pile to avoid drowning.

Working on Merger of Evil hit a couple of technical annoyances. A few months ago, I started using asdf to let me switch environments more fluidly. The project for Merger seemed like a great place to use that, but somewhere in that time, there were some updates to Node and various packages that turned a minor maintenance task into something far more complicated simply because I couldn't get into the right head space.

As a side note, when I was forced from my apartment in 2008 because of the flood in Cedar Rapids, I had a few months of high stress that also made things difficult to function. Much like the the last few months, minor problems became complicated when there were dozens of other things competition for my time and attention. When I got back to my apartment, I found that my old life didn't quite fit anymore. My daily ritual of going through 20-30 web comics and sites then seemed pointless so I purged my RSS feed, I stopped playing Kingdom of Loathing, and basically had to review what “sparked joy” in my life.

Those technical annoyances got me to thinking about improvements. Yeah, I had a bunch of other things demanding my attention, but optimizing a process is one of the ways I reduce stress (along with rewriting problems) and I desperately needed that distraction. As things were, someone on my social feed started talking about migrating from asdf to Nix. Nix is a lot like the old GNU Stow but had a much better package management and, more importantly, allowed me to have seamless transitions to different versions. Nix also is more of a holistic approach since one could use flakes to wrap everything including specific Node, Python, and C# packages into a single unified whole.

After a few weeks of using Nix, I can say that it definitely solves the little nits and scratches that asdf had plus handles my need for reproducible/consistent builds. I'm sure there are other problems with it, starting with the complexity of the setup files and the fact the featuers I want are still in beta, but I think I'm going to stay with it and see if it works out.

Things are still in flux and life is still heaping up on my shoulders, which means there is only one thing I can do: break apart the problems as much as I can and then focus on one thing at a time.

generator-mfgames-writing v0.3.2

2021-07-31T05:00:00Z

Related to process of writing from a few days ago, I've been working on another tool that I find myself using fairly often: Yeoman. This is a scaffolding tool, which basically means it asks a few questions and then sets up a project. In my case, it does a lot of the drudge work of creating a writing project and getting it ready for Gitlab's CI which I also use heavily (and why I wrote commitlint-gitlab-ci).

Like many of the writing tools, this is based on Javascript (but not Typescript this time).

$ sudo npm install -g yo generator-mfgames-writing

The first thing it does is ask a number of questions about the project. I'll use Nor Curse Be Found, my Beauty and the Beast sequel, as an example.

$ mkdir nor-curse-be-found
$ cd nor-curse-be-found
$ yo mfgames-writing
? The title for your novel or story: (Nor Curse Be Found)

All of my writing projects have a unique “slug” to describe them. This is used for the website's URL, such as https://fedran.com/nor-curse-be-found, and also the name of the output (dmoonfire-nor-curse-be-found-0.0.1.epub). I use a library to pull out the current directory's name and try to make a reasonable title out of it. I can change the title to something more accurate, maybe with apostrophes or possessives, but generally I keep the slug pretty even. In this case, the parenthetical item is the suggested output, so I can just hit return.

? The title for your novel or story: Nor Curse Be Found
? The slug based on the title: (nor-curse-be-found)

Just in case the directory doesn't match the directory, I use a library to take that given title and create a slug out of it. One of the biggest things, my Fast Trip rule, is that I don't want to tell someone how to work. That is why I called this semi-opinionated in that there are a lot of options but the default is how I work.

There is also a question about bylines, but then we start to get into the technical bits.

? The title for your novel or story: Nor Curse Be Found
? The slug based on the title: nor-curse-be-found
? Author's name or byline: D. Moonfire
? Do you want to create the Git repository? (Y/n)

This came from some years ago, but I do one story per repository, more so if I think the story has a possibility of being a bigger piece (which many of mine are), I want to track the versions independently, or I might be sending to an editor or submitting it somewhere (not likely these days, I self-censor a lot there). Since this is so common, I put it in as part of the questions.

I have a slowly evolving set of practices when it comes to writing processes. One of the first ones is setting up a Git repository. This always starts with configuring husky, semantic-release, conventional-commits, and commitlint. (I'm working on updating my Yeoman generator to make this a lot easier, but that will be later this week.)

? Do you want to create the Git repository? Yes
? Any command to run after `git init`? set-moonfire-git-user

I also write for a number of different projects (and bylines) which means I don't always use the same Gitlab login or even email. To faciliate this, I do not have a global user.name and user.email set up in Git so it blows up if I try to make a commit without setting a local one. Since that is tedious, I write a bunch of scripts that set the user information and SSH keys (set-moonfire-git-user for writing, set-mfgames-git-user for programming for example) and I want to call them before I do that first commit. If I just hit return to have a blank, then nothing extra will happen.

? Any command to run after `git init`? set-moonfire-git-user
? Do you want to create the initial commit? Yes
? Do you want to create the initial Git tag? Yes
? What version do you want to start? 0.0.1

Now, the initial commit is a messy one and a philosophical difference between me and the Semantic Release folks. They have their reasons to hard-code the value to “1.0.0” but I'm specifically start this process before a release, so I feel it should be “0.0.1” because I want that tracking as soon as possible.

Since I don't have the ability to set it via a configuration file, I get around it by creating the initial commit and then allowing it to be tagged so it can be pushed up and get the starting version I want.

This is also because I tag versions I give to editors, beta readers, and even my writing group (when I went to it). The tagged version also shows up on the various Fedran pages and mainly it shows progression as I write.

So, Fast Trip so I jump through a few hoops. It should also be clear why I needed that command to run after git init to set my user, otherwise this part would blow up with a “user not set” message.

? What version do you want to start? 0.0.1
? Which package manager do you use? (Use arrow keys)
❯ NPM
  Yarn

Moving away from Git and into the build process, the first question is package management. I have a love/hate relationships with npm. The command works great, but I've been struggling with it (more on than off lately) so I've been using yarn fairly heavily in the last year or so. But, with Yeoman I can give both so the user can use up or down keys to select and hit return to choose. Once they do, the menu goes away and is replaced with the choice.

? Which package manager do you use? Yarn
? Do you want to use semantic releases? Yes
? Do you want to use conventional commits? Yes
? Do you want to use Husky? Yes

The next three are to set up the common functionality I use with releases. They set up semantic-release, conventional commits, commitlint, and Husky which I use to make sure all the commit messages are clean. These four program have been a stable in most of my development lately, for coding and writing, for Typescript and even C#.

? Do you want to use Husky? Yes
? Do you want to use CI/CD? Yes
? Do you use Gitlab for your CI/CD? Yes

I can't say enough for Gitlab's CI. Whenever I push up code to the server, it runs the entire publication process. Assuming it works, it then produces a zip archive with EPUB, MOBI, PDF, and other formats without me doing anything else. Every single time. With sematic releases and the rest, each one has a unique version so I don't have “final”, “final-2”, “final-2a”, etc. Just a clean version number that shows up everywhere, starting at 0.0.1.

? Do you use Gitlab for your CI/CD? Yes
? Do you use asdf? No

This is another tool I use, asdf. This was also something that pretty much started last year, but one of the goals of mfgames-writing was to be able to produce consistent output of novels even years later. Well, what I didn't realize is how much the Node ecosystem would change and Node 15 doesn't exactly run Node 8 packages very well due to deprecations and changing libraries.

This is where asdf comes in. It puts a small file (.tool-versions) which says which version of Node, Yarn, C#, Python, whatever and will automatically change to those versions when I cd into the directory. It basically rolls in all the functionality of nvm, virtualenv, and a bunch of other localized version libraries into one. Since I write and work across many languages, having one tool to manage that is easier for me. So, saying “yes” to the question copies that .tool-versions into the resulting directory.

? Do you use asdf? Yes
? Which formats do you want to use? (Press <space> to select, <a> to toggle all, <i> to invert selection)
❯◉ PDF
 ◉ EPUB
 ◯ DOCX
 ◯ HTML

With this next question, we are out of the build section and into the formats. Basically, this just queries which types of files will be written out. The directions are pretty clear, but basically this will determine which of the format libraries are included.

? Which formats do you want to use? PDF, EPUB, DOCX, HTML
? The name of the theme package to use: (@mfgames-writing/clean-theme)

I only have three themes right now, two are “generic” and one I prefer just for Fedran stories (but obviously don't prevent anyone from using it). Later, I hope to have more themes, but basically the choices are @mfgames-writing/clean-theme and @mfgames-writing/greekil-theme (a Gentium-based theme).

? The name of the theme package to use: @mfgames-writing/clean-theme
Creating initial Git repository
Initialized empty Git repository in fedran/nor-curse-be-found/.git/
Running custom Git command


I'm all done. Running npm install for you to install the required dependencies. If this fails, try running the command yourself.


   create package.json
    force ../../../.yo-rc-global.json
    force .yo-rc.json
   create release.config.js
   create commitlint.config.js
   create .husky/commit-msg
   create .gitlab-ci.yml
   create .tool-versions
   create publication.yaml
   create yarn.lock
   create README.md
   create .gitignore
   create chapters/chapter-01.md

Changes to package.json were detected.

Running yarn install for you to install the required dependencies.

With that, it just producing a wall of tell as it creates all the files, hooks up everything, and basically gets my entire project ready, including a sample first chapter, a really basic README.md, and a Git repository ready to push.

$ ls -a
.                     .git            node_modules       README.md
..                    .gitignore      package.json       release.config.js
chapters              .gitlab-ci.yml  package-lock.json  .yo-rc.json
commitlint.config.js  .husky          publication.yaml

Right off the bat, I can build the files and see the output.

$ yarn build:pdf
... lots of output
$ ls *.pdf
nor-curse-be-found-0.0.1.pdf
$ yarn build
... huge amounts of output
$ ls nor-curse-be-found*
nor-curse-be-found-0.0.1.docx  nor-curse-be-found-0.0.1.epub
nor-curse-be-found-0.0.1.html  nor-curse-be-found-0.0.1.pdf

I'm also ready to push up to Gitlab.

$ git remote add origin git@gitlab.com:fedran/nor-curse-be-found.git
$ git push --tags
$ git push -u origin main

Hopping over to Gitlab, I can see it building immediately.

As soon as it is done, it uploads the output as an artifact.

I can then click on the browse (or download) button on the right.

Browsing lets me see a list of all the outputs that I've added.

And I can even click on the PDF to preview it.