Development

Leicmin

2026-04-04T05:00:00Z

I haven't posted in three months. That isn't to nothing has been happening, only that I was so focused on the “now” that I didn't really have a chance for any retrospection (which is effectively what a lot of my posts are) or idle thoughts (which cover most of the rest). Part of that is how my thoughts work, I withdraw from everything when I'm struggling with problems.

There is always the usual family crises (we lost a family member last week and there is a good chance we're going to lose a close one “soon”), work pressure, and the difficulties of being a parent.

Age Verification

On top of that, the various news about age verification laws, the casual tossing around of felonies and crippling monetary fines, and the rise of privacy-busting legislation were hitting my communities. Too many legislation were salivating at the openings that the Texas got with the Supreme Court ruling, the idea of forcing all cases to be dragged into the state with the most restrictions, and generally finding Machiavellian ways to force creators from closing up shop without explicitly saying that porn was illegal (because it isn't).

I know Fedran is “mostly” excluded from most of those age verification laws. There are a million words on the site and less than 10% of them are what I would consider “adult” or graphic. I could easily say it isn't pornographic, but then Tennessee's Protect Tennessee Minors Act got put into effect and it appears to apply to my fiction.

It was that law that triggered a “fight or flight” response. I seriously considered just closing down the shop and giving up writing. I was already in the middle of the burnout from my 300 consecutive week challenge and my struggles after Covid. It felt like an easy escape, just cut off that part of my life.

But then I didn't. I'm terrible at giving up, I mean that was pretty much the major thread throughout Sand and Blood, Sand and Ash, and Sand and Bone. That part is part of me. I might take a while to get there, but I don't give up easily. I also saw other creators who were also struggling with the same problems, but they were clearly in the arenas the laws were targeting. Creators who were writing things that were more erotic (it's never about violence, which I universally consider more disgusting than sexuality).

So the flight (giving up) turned into fight (doing something). I already knew that I wasn't going to be great about anything I wrote, but it was something I could do. I have to be honest, I didn't come out of Covid unscathed and I'm painfully aware of how much I struggle with things that I found pleasurable less than ten years ago.

Also, quite a few years before all this, I had an idea of creating a self-hosted site that could coordinate various subscription/patron services. Basically, taking the supporter badges from forums, the ability to have Patreon or Ko-Fi supporters (or for a while, cryptocurrency), and be able to control access to sites. Ideally, in a manner that would not require exposing personal information (because no one needs to know those details). And no gamification or social interactions, just something between a creator and someone who likes their work.

That also would give me the ability to integrate age verification systems or even geo-blocking or VPN-blocking (there were a few bills that required periodic verification of age or the need to ban all VPN usage) in a central place.

Would it be helpful? Maybe. But if anything, I could implement something to protect my ass and maybe it would help others.

Leicmin

And that ended up being Leicmin (more detail on the development site and the forge).

Well, multiple iterations of Leicmin. I started with trying to implement a C# version since I thought I was really good at the language and it would be “easy” to bang up something.

I didn't do it because it was easy. I did it because I thought it was easy.

After a few months, I realized that I had too many things rattling in my head, and the ideas that I toyed with for years thinking they would make a great system (event sourcing, automated auditing, API-first) ended up crumbling once I started to do significant things. I spent more time fighting libraries and tools than I was moving forward.

Somewhere in December, I realized it wasn't going to work out so I decided to burn it down and start from scratch. This time, I jettisoned most of the fancy ideas and went with something simple: a website that required no Javascript, had no API, and was about as plain and simple as I could make it. I also decided to go with Rust, but avoided the fancier front ends and went with straight, old-school templating.

There were some things that I wasn't going to get rid of: there are type-safe identifiers everywhere, almost everything is CLI first, it is based on Postgres, and there are almost no general solutions anywhere.

And I got progress. I could see things moving forward and I didn't feel like I was writing myself into a corner. True, the C# attempt taught me a lot, but it was really nice seeing slow but steady progress to something that works. I felt bad that it wasn't “fancy” or more capable but I was happy that I was creating something I thought could benefits others.

My goal was the end of February, but it ended up being the end of March, that I got something with a very simple functionality.

Folks could log in and change passwords
They could connect Patreon and Subscribe Star (a feature requested online)
They could set up a password that could be used basic authentication
The creator could export a .htpasswd file, hook it up to a static site, and be able to authenticate

Through that, I could get a “basic” age verification since Patreon and Subscribe Star both do age verification on their sides. That means I could mark my patreon as “mature” (I haven't done that yet), wire up my site in MfGames.Nitride to use it, and have what I need to quell that “fight” response in the back of my head.

And someone (the person who asked for Subscribe Star) is already using it. And reporting bugs and defects as they onboard their subscribers onto it. Which is exciting, because I rarely see someone actually using my tools (Nitride and MfGames Writing mostly).

Going Slow

Some of the painful parts is that progress was slow. I've been getting a lot of pressure from work to embrace large language models but I didn't want to do that with Leicmin. I want to lay down the code, to come up with the ideas, and work through them because I cherish those little insights when I find some pattern I never thought about before.

Protecting Minors

I'm not against the idea of limiting minors from accessing content. What I don't like is how law makers was going about it, since little of what they are producing is about protecting children, but very clearly attacking something they consider immoral.

(I also think it should be something parents should decide on, not law makers making decisions for everyone.)

Even listening to them talk about it, it was just another attack on people they don't like, they just use minors as their shield to defend their own biases.

That also means as I have the bandwidth, I'll find alternatives to handing over personal information to for-profit companies. Or find a way not to need a subscription service. I read about and bookmarked a couple promising alternatives, but they aren't quite there yet.

Going Forward

There are a ton of things that still need to be done before I consider it “done” in terms of user experience and usability.

Light theme was the first thing I was asked about
Documentation
Better authentication options
Emails
More age verification options
More geo blocking options
Auditing compliance (okay, that one is me)
Many more

On my personal side:

Set up an instance of my own
Go through my Fedran stories and mark the “adult” ones
Set up the instance to put a password on the adult stories

And, keep slowly moving forward. It was almost immediately after I was able to lean back and say “I got the MVP” that I was able to relax. Even not setting it up myself (which I will do), the knowledge that I had a clear path forward to handle that fight or flight was a major relief.

Just keep swimming.

—Dory, Finding Nemo

Allegro

Speaking of writing, I'm slowly submitting Allegro through the writing group. Of the alpha readers, about half of them came back with some good feedback and half of them never finished. It probably means that it will end up as well as Flight of the Scions, but I'm still going to try publishing the best book I can.

Other Smarter People

2026-01-04T06:00:00Z

I don't like reinventing the wheel. At the same time, I'll look for something and not find it, go through the effort to create something of my own, only to find out that my search skills weren't high enough and someone has already figured that out.

In a matter of twenty-four hours, I found that two such situations had happened. Which means, I can close the topic on identifiers and simply point to other folk's work and build the little quirk I want on to of those.

Series

2024-05-30 Generating and Using Identifiers - A little discussion on unique identifiers, including formatting and presentation. A small database lesson on page splitting.
2024-06-29 Generating and Using Identifiers (Part 2) - A few expanded ideas about identifiers after discussing it with others.
2026-01-04 Other Smarter People - I found another specification for representing identifiers that is better supported, has more attention, and covers most of the basics I need to.

Identifiers

Let's start with the last six months (it's been that last since the last of this series… actually, of most of my posts): I actually used my ideas to see if they were viable in practice as they were in my head.

I found that when I elided the identifiers, which means only showing the last group of the identifier along with the prefix, it didn't give the benefit I thought it would. For example, group__01_hz613s22_k8nr6w6g_rv1y5c52 being elided to group__rv1y5c52. When seen on screen, the group__ was still the bulk of the screen estate and the content was sufficient to make it unimportant.

Then while I was working on Leicmin, I stumbled onto a series of Rust libraries that implemented a standard that was almost completely the same as mine, including using Crockford32 encoding: TypeID (or my local tag).

TypeID uses the underscore separator, so double-click and selection works. It defaults to UUIDv7 but can also be used with other UUIDs. It only has one between the prefix and identifier block and no grouping. Which means the above example would be group_01hz613s22k8nr6w6grv1y5c52.

I lose the grouping and there no rules for eliding, but those are easy to handle. Just take the left X characters and not worry about anything else. So, in C#, eliding would simply be identifier.ToString()[^8..] and getting rv1y5c52.

Good enough.

Fortunately, there are a lot of implementations of TypeID. None of them work the way I want them too, or they are quite there, but it's a much better start and works with a standard that someone else has already established.

Package URLs

Speaking of finding standard, when I was giving my thoughts on Bakfu, I was trying to come up with a useful approach for referencing packages across different ecosystems.

While learning about TypeID, I noticed that the crates.io packages has a PURL link on the sidebar. I had never seen that before, but it basically lead me down to learning about a “mostly universal” URL format for packages so… other people were trying to solve the same thing and they had some interest behind their own (as opposed to none behind mine).

Some examples (from their website):

pkg:deb/debian/curl@7.50.3-1?arch=i386&distro=jessie
pkg:docker/cassandra@sha256:244fd47e07d1004f0aed9c
pkg:gem/jruby-launcher@1.1.2?platform=java
pkg:golang/google.golang.org/genproto#googleapis/api/annotations
pkg:maven/org.apache.xmlgraphics/batik-anim@1.9.1?repository_url=repo.spring.io%2Frelease&packaging=sources
pkg:npm/%40angular/animation@12.3.1
pkg:nuget/EnterpriseLibrary.Common@6.0.1304
pkg:pypi/django@1.11.1
pkg:rpm/fedora/curl@7.50.3-1.fc25?arch=i386&distro=fedora-25
pkg:rpm/opensuse/curl@7.56.1-1.1?arch=i386&distro=opensuse-tumbleweed

Well, that covers pretty much the same scope I was trying to implement, so another thing I can move to an accepted standard and stop working on my own.

Thoughts

A lot of my personal work is in isolation. I've always struggled to join communities and I have a relatively small “footprint” when it comes to social networks. And, because of my fragmented nature, I'm rarely in touch of people who are in the “know” who can tell me that someone else is already working on, or has solved, a problem I'm working on.

That all said, I'm glad I still went down the rabbit hole of trying to solve it. More so when my solution is only trivially different from the more popular one. It tells me I'm in the right direction.

I also think as a thought exercise, it gives me more appreciation for the work that goes behind these efforts and frequently makes me a fan or even advocate because I'm emotional behind the reasons.

So, time not wasted but also relief that someone else figured it out instead.

Git For Authors (v0.1.1)

2025-08-08T23:41:54Z

🛈 This isn't done. Not even close. As you can see from the front page, a lot of the bullet items aren't leading to links, which I full intend to flesh out into more detailed sections. Plus I'm still in the process of dumping information and the organization isn't there. If this was the 90s, I would have a “Under Construction” GIF.

I enjoy writing as much as I like coding. While I used entirely separate tools to do either, over the years, those two processes have converge to my writing projects look like my coding projects. They use pipelines to generate the final output, source control, automated versioning, and the same suite of tools. In many cases, these tools are designed to automate tedious parts of publishing or coordinate work with others, something that can be difficult with existing tools.

The inspiration of this is Fast Trip, a story by James White.

Organization

I don't like just saying “this is how I do things” because this isn't, nor will it ever be, the One True Way™ of writing with Git. Instead, I've attempted to organize the topics from high-level ones, to global concepts, and then into advanced topics.

There is a lot of cross-linking of topics within the individual pages for relevance, but you can easily choose to ignore those links since if you follow this index page, you'll get to them in “discovery” order.

Meta Topics

Target Audience - Who this plot is aimed for and what skills are useful to take advantage of it
Conventions describes the notations used
Model-View-Controller is the high-level concepts of separating the content (model) from its appearance (view)

File Formats

While the goal of this plot is to use a program—Git in this case—to manage writing, the first part is talking about “what” is being managed. In this case, it is the words you want to put into a novel or story. There are other things that you might want tracked, such as character notes, research, various pin boards and themes. These are all going to be “tracked” as part of the project.

Text Files is a discussion of why writing should be done in plain text files
Top Matter are the details about a given chapter or story
Binary Files talks about why Git doesn't play well with binary files

Because of limitations in text files for doing extra formatting, I use markup to tell when to make something italic, bold, or write an epigraph.

Markup is how to decorate a text file with some of the formatting rules that writers need
Markdown is the most popular form of markup

Organization

Organization also becomes important with bigger projects. While you can throw everything into the project directory in a glorious mess, I have established some conventions that work well for me.

Project Layout is how I organize my files and directories for writing projects.

Source Control

The entire reason for this plot is to use a source control program to manage writing. These programs keep a historical record as you write, allow to review changes, but also avoid accidentally overwriting changes. Programs like Git also allow you to handle feedback from editors and reviewers even when they come in over a period of time and after you've already made changes.

Source Control discusses the systems in general and my history with them
Git is the source control system I mainly use while writing
Repositories are copies of the writing project
Commits are how changes are saved to a repository (and a branch)
Tags can be used to identify points in time where changes are made
Branches are used to coordinate with readers and editor, or to redo chapters
Merging talks about how to integrate changes from a branch, editor, or normal writing

Text Editors

One advantage of using text files is that almost any editor or IDE can be used to write. They have different capabilities or plugins that help.

Visual Studio Code is a lightweight and flexible text editor

Transforming

A critical component of using text formats is how to transform them into other formats that editors and readers need.

Transforming talks about transforming source files into the desired output
MfGames Writing is a publishing tool for making EPUB and PDF files

Source Forges

Building on top of source control, Git in specific, are code forges which give a web view of your projects, limit others from seeing them, but also gathering feedback and edits.

Code Forges is about online systems for managing Git repositories, pipelines, and issues
Pipelines are automated tasks that run
Issues are a task tracking for a project
Pull Requests are for getting changes from others

Each forge is different from the others in almost every way. That includes setting them up, how to run the pipelines, or even how issues are tracked.

Forgejo
SourceHut
GitLab
GitHub

Versioning

One way of dealing with the “final23” files

Versioning describes while creating a version for a story is important and how to use semantic versioning
Conventional Commits describes a common formatting for Git commit messages to automate versioning

Other Tools

Beyond the format of the files, how a project is organized is important to keeping everything separate and also to make some of the tools more effective.

Querying talks about using tools to work with the source files
Markdowny is a tool used for querying the top matter

Setting Up the Environment

I've been writing novels and stories for many decades. While most of the time, I just write it and then produce the resulting PDF or EPUB, occasionally I want to go back only to find that the libraries have “bit rotted” or I based on them on a global package I forgot to migrate to my new machine or that I had modified to handle a newer book that broke the old one.

Inevitability, this means days or even weeks of bringing the project up to date just to get a PDF out of it.

Along the way, I got frustrated with that and have come up with various tools or frameworks for producing a consistent environment years or even decades later.

Using NixOS describes setting up NixOS to make it easier to write

About This

Finally, this is a living document. Writing it as a digital plot and versioning it will hopefully allow me to evolve as we go.

License

This book is distributed under a Creative Commons Attribution-ShareAlike 4.0 International license. More info can be found at https://creativecommons.org/licenses/by-sa/4.0/

The preferred attribution for this novel is:

“Git For Authors” by D. Moonfire is licensed under CC BY-SA 4.0

In the above attribution, use the following links:

Git For Authors: https://d.moonfire.us/garden/project-layout/
D. Moonfire: https://d.moonfire.us/
CC BY-SA 4.0: https://creativecommons.org/licenses/by-sa/4.0/

Forges

The cannon source of this project can be found on the Moonfire Games forge. However, due to AI scrapers and automated scammers, the site gets occasionally locked down to avoid overwhelming resources. However, this project is also mirrors to the following locations, which are also monitored for new issues or discussions.

You are also free to contact Dylan directly.

Project

This project has a unique identifier to make it easier to distingiush from other queries. All blog posts and pages will hopefully include this if it becomes difficult to find.

01a09c8d-b936-7318-8763-4b8e64cf39ec

Using Nitride - Front Matter

2025-06-12T05:00:00Z

Being able to associated metadata for a page is a very useful thing. It can be used to group pages into categories, control how it is styled, or to simply provide internal notes. To do that, we use something called a YAML front matter to describe the details.

---
title: Name of the Page
summary: Summary of page
image: /path/to/image.png
---

This is the front page.

This is a major functionality of the Markdown + YAML pages that I use in my technical, fantasy, and even blog posts. I use the categories and tags heavily on this page, not to mention giving a summary details for links. It can also be used to provide OpenGraph schemas.

Series

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

Defining a Model

Because C# is a static language, we want to take advantage of a schema so we can have type-safe. Historically, I've called this PageModel because I wrap that in a TemplateModel when I get to the styling.

// In //src/dotnet/Models/PageModel.cs
namespace Generator.Models;

public class PageModel
{
    /// <summary>
    ///     Gets or sets the optional list of categories associated with the page.
    /// </summary>
    public List<string>? Categories { get; set; }

    /// <summary>
    ///     Gets or sets the URL of the image associated with the page.
    /// </summary>
    public string? Image { get; set; }

    /// <summary>
    ///     Gets or sets the summary for the page.
    /// </summary>
    public string? Summary { get; set; }

    /// <summary>
    ///     Gets or sets the title of the page.
    /// </summary>
    public string? Title { get; set; }
}

Naturally, this can and will get a lot more complicated. Because we are using YAML, we can have nested objects, tags, and references that are appropriate for our page. In the above example, we are defining an optional title, summary, an image, and a list of categories.

It may come to a surprise to you, but this will eventually become a component in the page Entity class that is passed through the pipelines.

Including the YAML module.

Like the others MfGames.Nitride modules, we have to add it as a NuGet reference and tell the system to use it.

cd src/dotnet
dotnet add package MfGames.Nitride.Yaml

Adding it to the system is just a matter of modifying Program.cs to include it.

// In //src/dotnet/Program.cs
var builder = new NitrideBuilder(args)
    .UseIO(rootDirectory)
    .UseMarkdown()
    .UseHtml()
    .UseYaml()
    .UseModule<WebsiteModule>();

Parsing Front Matter

Parsing the front matter uses an operation, MfGames.Nitride.Yaml.ParseYamlHeader to parse the text content, pull off the front matter, wrap it in the model call, and then replace ITextContent with the page without the YAML header.

In effect, this:

---
title: Name of the Page
summary: Summary of page
image: /path/to/image.png
---

This is the front page.

... becomes “This is the front page.” in the text content with a PageModel component.

Adding the operation is relatively simple but this operation uses a generic parameter to identify the model to parse the YAML as.

// In //src/dotnet/Pipelines/Inputs/PagesModel.cs
public PagesPipeline(
    ILogger<PagesPipeline> logger,
    ReadFiles readFiles,
    IdentifyMarkdownFromPath identifyMarkdownFromPath,
    MoveToIndexPath moveToIndexPath,
    ParseYamlHeader<PageModel> parseYamlHeader)
{
    _logger = logger;
    _identifyMarkdownFromPath = identifyMarkdownFromPath;
    _moveToIndexPath = moveToIndexPath;
    _parseYamlHeader = parseYamlHeader;

And, like the others, it's just a matter of adding the operation into the pipeline.

// In //src/dotnet/Pipelines/Inputs/PagesModel.cs
var list = _readFiles
    .Run(cancellationToken)
    .Run(_identifyMarkdownFromPath)
    .Run(_parseYamlHeader)
    .Run(_moveToIndexPath)
    .ToList();

Once we run it, we get the following output:

[01:44:05 INF] <PagesPipeline> Entity: Path /contact/index.md, Components ["Zio.UPath","Generator.Models.PageModel","MfGames.Nitride.Contents.ITextContent","MfGames.Nitride.Markdown.IsMarkdown"]
[01:44:05 INF] <PagesPipeline> Entity: Path /index.md, Components ["Zio.UPath","Generator.Models.PageModel","MfGames.Nitride.Contents.ITextContent","MfGames.Nitride.Markdown.IsMarkdown"]

When we look at the HTML output, you'll notice it has a simplified version.

$ cat build/typewriter/html/index.html 
<h1>Typewriter Press</h1>

Retrieving the component is code just requires the Get<PageModel>() method or to use the various LINQ-based calls from MfGames.Gallium.

var page = entity.Get<PageModel>();

var list = entities
    .SelectEntity<PageModel>(OnlyRunOnEntitiesWithPageModel)
    .ToList();

public Entity OnlyRunOnEntitiesWithPageModel(Entity entity, PageModel page)
{
    return entity.Set(somePageSpecificVariable);
}

Gallium Methods

In the above case, the SelectEntity call will skip calling the lambda for entities that don't have a PageModel, but will still return it into the list. This is the “systems” part of the ECS system. The following will strip out the entities that don't have the appropriate models.

.SelectEntity<PageModel>((entity, page) => entity, includeEntitiesWithoutComponents: false)

So far, Gallium is set up to allow up to four components. I haven't had a need to do more, but those are easy to add. This makes it useful when doing some model processing on files that have a future date.

.WhereEntity<PageModel, UPage, Instant>(FilterOutFuturePages)

The SelectEntity is rather powerful because it takes an Entity and returns one. It can be the same entity, or it can be one that has zero or more components added or changed inside it. I found this really useful when I'm parsing out categories or I'm trying to build up a list of some specialized functionality that the styling will need.

public Entity OnSelectEntity(
    Entity entity,
    PageModel page,
    UPath path,
    ITextContent textContent)
{
    return entity
        .Set(nextPreviousModel)
        .SetAll(parentPage, parentUrl)
        .Set(modifiedPageModel);
}

What's Next

We have the minimum number of components and systems needed to setup styling. Next time, I'll use Handlebars to style the page and put a little chrome around the output.

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

Using Nitride - Markdown

2025-06-10T05:00:00Z

Yesterday, I created an over-engineered program to copy a single file from one directory to another. Now, time to make it less overkill by transforming that Markdown file into simple HTML.

Series

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

MarkDig

I don't like reinventing the wheel. I mean, I seem to keep doing it but I don't enjoy it. That was one reason why I tried another static site generators before working on MfGames.Nitride. However, when it comes to redoing a Markdown parser, even I'm not that foolish when there is already MarkDig, an excellent library for turning Markdown into HTML and extendable enough that I could also turn Markdown into Gemini (a later post).

In these cases, we need to tell Nitride how to do anything with Markdown since I didn't make it part of the core library. To do that, we need to pull in the NuGet package. While we're at it, we're also going to add the HTML processing library.

$ cd src/dotnet
$ dotnet add package MfGames.Nitride.Markdown
$ dotnet add package MfGames.Nitride.Html

Once we have the packages installed, we need to add those modules into the system. This is where Autofac came in helpful since I just have to add a module for the package it will handle the registration of any operations, components, and systems that we need to use.

// In //src/dotnet/Program.cs
var builder = new NitrideBuilder(args)
    .UseIO(rootDirectory)
    .UseMarkdown()
    .UseHtml()
    .UseModule<WebsiteModule>();

As you can see, I'm trying to follow the generic host pattern for the setup.

Identifying Markdown

While it may be obvious to convert any entity class that ends in .md into .html, we break this apart into separate steps. First, is that we identify a file as a Markdown file. This does two things, it adds the MfGames.Nitride.Markdown.IsMarkdown as a component, and then treats the contents as text instead of binary.

If you remember previously, we had this output:

[00:41:57 INF] <PagesPipeline> Read in 1 files from /src/pages
[00:41:57 INF] <PagesPipeline> Entity: Path /index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]
[00:41:57 INF] <SimplifiedMarkdownPipeline> Reading 1 entities
[00:41:57 INF] <StyleHtmlPipeline> Reading 1 entities
[00:41:57 INF] <OutputHtmlPipeline> Writing out 1 files
[00:41:57 INF] <OutputHtmlPipeline> Entity: Path /build/typewriter/html/index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]

Now, we're going to use a new operation, MfGames.Nitride.Markdown.IdentifyMarkdownFromPath. This could be put in a central place, such as SimplifiedMarkdownPipeline, but I found it is better to do this earlier than later so I usually put the identify process in the input methods. In this case, PagesPipeline:

// In //src/dotnet/Pipelines/Inputs/PagesPipeline.cs
public PagesPipeline(
    ILogger<PagesPipeline> logger,
    ReadFiles readFiles,
    IdentifyMarkdownFromPath identifyMarkdownFromPath)
{
    _logger = logger;
    _identifyMarkdownFromPath = identifyMarkdownFromPath;

    _readFiles = readFiles
        .WithPattern("/src/pages/typewriter/**/*.md")
        .WithRemovePathPrefix("/src/pages/typewriter");
}

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    var list = _readFiles
        .Run(cancellationToken)
        .Run(_identifyMarkdownFromPath)
        .ToList();

This operation doesn't take any parameters because it attempts to “do the right thing” with a minimal amount of effort. Running the code now produces this:

[00:46:43 INF] <PagesPipeline> Entity: Path /index.md, Components ["MfGames.Nitride.Markdown.IsMarkdown","Zio.UPath","MfGames.Nitride.Contents.ITextContent"]

The big things is that we have a new component, IsMarkdown, and the IBinaryContent changed to ITextComponent which gives us some additional extension methods but also indicates that the file is a text file instead of treating it as a simple binary. It still hasn't loaded the file into memory, it just switched how it is handled.

This is a separate step is because sometimes I want to keep a specific file as Markdown and not convert it to HTML. Also, there are times when I construct an Entity directly without having a valid path and I just have to add the IsHtml and ITextContent and it then acts like every other file without having to have any special rules.

Content

Entities with ITextComponent have a number of useful extension methods associated with them. (Technically, you can use these methods against any Entity class and it will try to convert a binary to text if there is a IBinaryContent and we want a ITextContent).

bool hasText = entity.HasTextContent();
string text = entity.GetTextContent();

entity.SetTextContent(stringValue);
entity.SetTextContent(stringBufferValue);

These are also stored as a ITextContent instead of string or StringBuffer. This is because the default interface is to leave the content on the disk and use Zio to retrieve it. However, as soon as SetTextContent is used, then it keeps that value in memory for the rest of the execution. This is the point where memory pressure begins to increase.

In the future, we could easily create a ITextContent implementation that writes large text files to the disk to get them out of memory. However, even my largest chapter of twenty-five thousand words doesn't create too much of a problem so I haven't bothered trying to implement that at this point (but if I did, it would go into //.cache in some manner).

Converting Markdown to HTML

Just identifying a file as Markdown doesn't do anything in itself. To convert it, we use another operation, MfGames.Nitride.Markdown.ConvertMarkdownToHtml. This easily goes into the StyleHtmlPipeline to handle the conversion and styling. It allows any MarkDig extension or plugin to be called as part of the setup, allow one to customize exactly how the output is generated.

We also want to change the extension from .md to .html. I realize I should have baked that logic into the ConvertMarkdownToHtml since it is a common operation, which I will do, but for now, we also need to use the MfGames.Nitride.IO.Paths.ChangePathExtension to do that.

public StyleHtmlPipeline(
    ILogger<StyleHtmlPipeline> logger,
    SimplifiedMarkdownPipeline simplifiedMarkdownPipeline,
    ConvertMarkdownToHtml convertMarkdownToHtml,
    ChangePathExtension changePathExtension)
{
    _logger = logger;
    _changePathExtension = changePathExtension.WithExtension(".html");

    _convertMarkdownToHtml = convertMarkdownToHtml
        .WithConfigureMarkdown(builder =>
        {
            SmartyPantOptions smartyPantOptions = new();
            SmartyPantsExtension smartyPants = new(smartyPantOptions);

            builder
                .Use<GenericAttributesExtension>()
                .Use(smartyPants);
        });

    AddDependency(simplifiedMarkdownPipeline);
}

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    var list = entities
        .Run(_convertMarkdownToHtml, cancellationToken)
        .Run(_changePathExtension, cancellationToken)
        .ToList();

    _logger.LogInformation("Reading {Count:N0} entities", list.Count);

    return list.ToAsyncEnumerable();
}

Running this gets us:

$ just build
[01:00:43 INF] <PagesPipeline> Read in 1 files from /src/pages
[01:00:43 INF] <PagesPipeline> Entity: Path /index.md, Components ["MfGames.Nitride.Markdown.IsMarkdown","Zio.UPath","MfGames.Nitride.Contents.ITextContent"]
[01:00:43 INF] <SimplifiedMarkdownPipeline> Reading 1 entities
[01:00:44 INF] <StyleHtmlPipeline> Reading 1 entities
[01:00:44 INF] <OutputHtmlPipeline> Writing out 1 files
[01:00:44 INF] <OutputHtmlPipeline> Entity: Path /build/typewriter/html/index.html, Components ["MfGames.Nitride.Html.IsHtml","Zio.UPath","MfGames.Nitride.Contents.ITextContent"]
$ find build -type f
build/typewriter/html/index.html
$ cat src/pages/typewriter/index.md 
# Typewriter Press
$ cat build/typewriter/html/index.html
cat build/typewriter/html/index.html
<h1>Typewriter Press</h1>

And now our over-engineered copy method has duplicated markdown2html for a single file.

Components

You may notice that IsMarkdown has been removed and IsHtml was added. This is part of where I struggled with Statiq and lead me down the path of using components. I don't have to pre-define the different data types, purposes, or even formats of a file. Enums are great, but they don't allow easy extension but with an ECS, it's just a matter of adding and removing components based on the use.

I've used components for a lot of things including identifying pages that should be in the blog archives, special notices, or pages that I want to ignore because they are aliases. I also embed indexes and lists into the pages to allow things like the “next” or “previous” links. If I was doing a web comic, I could have a per-character next/previous system easily implemented via those components.

My other static site generators didn't even have the content type tagging Statiq did, which was a novel concept for me and one that I'm glad I had a chance to use. It simplified a lot of my logic and lead nicely into where I am today.

Planning Ahead

As I'm planning ahead, I'm going to do the following change:

Rename SimplifiedMarkdownPipeline to ContentPipeline
Move the logic I just added into a new pipeline called BareHtmlPipeline and insert it into between ContentPipeline and StyleHtmlPipeline

The reason for this is because RSS/Atom feeds use bare HTML to generate their content, so it makes sense to have that bare pipeline feed both of them while having the ContentPipeline handle a lot of the linking and references that we'll need.

Directory Paths

One last thing for this post: I prefer paths that end in directory slashes instead of files. So, if create a contact page at //src/pages/content.md, we want the HTML to be at https://typewriter.press/content/. This is, creatively enough, another operation: MfGames.Nitride.IO.Paths.MoveToIndexPath.

// In //src/dotnet/Pipelines/Inputs/PagesPipeline.cs
public PagesPipeline(
    ILogger<PagesPipeline> logger,
    ReadFiles readFiles,
    IdentifyMarkdownFromPath identifyMarkdownFromPath,
    MoveToIndexPath moveToIndexPath)
{
    _logger = logger;
    _identifyMarkdownFromPath = identifyMarkdownFromPath;
    _moveToIndexPath = moveToIndexPath;

    _readFiles = readFiles
        .WithPattern("/src/pages/typewriter/**/*.md")
        .WithRemovePathPrefix("/src/pages/typewriter");
}

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    var list = _readFiles
        .Run(cancellationToken)
        .Run(_identifyMarkdownFromPath)
        .Run(_moveToIndexPath)
        .ToList();

And we add a contact.md page which a run gives us this:

$ just build
[01:12:26 INF] <PagesPipeline> Entity: Path /contact/index.md, Components ["MfGames.Nitride.Contents.ITextContent","MfGames.Nitride.Markdown.IsMarkdown","Zio.UPath"]
[01:12:26 INF] <PagesPipeline> Entity: Path /index.md, Components ["MfGames.Nitride.Contents.ITextContent","MfGames.Nitride.Markdown.IsMarkdown","Zio.UPath"]
[01:12:26 INF] <ContentPipeline> Reading 2 entities
[01:12:26 INF] <BareHtmlPipeline> Reading 2 entities
[01:12:26 INF] <StyleHtmlPipeline> Reading 2 entities
[01:12:26 INF] <OutputHtmlPipeline> Writing out 2 files
[01:12:26 INF] <OutputHtmlPipeline> Entity: Path /build/typewriter/html/contact/index.html, Components ["MfGames.Nitride.Contents.ITextContent","MfGames.Nitride.Html.IsHtml","Zio.UPath"]
[01:12:26 INF] <OutputHtmlPipeline> Entity: Path /build/typewriter/html/index.html, Components ["MfGames.Nitride.Contents.ITextContent","MfGames.Nitride.Html.IsHtml","Zio.UPath"]
$ cat src/pages/typewriter/contact.md 
# Contact Us

Our emails is [contact@typewriter.press](mailto:contact@typewriter.press).
$ cat build/typewriter/html/contact/index.html 
<h1>Contact Us</h1>
<p>Our emails is <a href="mailto:contact@typewriter.press">contact@typewriter.press</a>.</p>

As you can see, the input is //src/pages/contact.md, but through the pipeline, it is written out as //build/typewriter/html/contact/index.html.

What's Next

Next up, handling front matter. We need that for many reasons, but one of the biggest reason is to provide metadata to generate styled HTML output.

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

Using Nitride - Entities

2025-06-09T05:00:00Z

From yesterday's post, there was an dangling topic, the Entity class. Unlike most of the static site generators I've worked with, MfGames.Nitride uses an ECS (Entity Component System) to generate its files.

Series

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

History

I've worked on a number of static site generators over the years, including more than a few adhoc ones until I finally got something stable with CobblestoneJS which carried me for almost a decade of heavy use. A lot of the patterns that Nitride was based on were on how I found I had to build my pages in Cobblestone, including having a separate “gather” step which wrote out temporary files and then the “process” step which formatted them and made everything pretty. I had to do that because I wanted to have cross-linking and references work across multiple Git projects.

A good example is the Fedran website. At the bottom of a chapter, I have a list of characters and locations in the chapter. The links are built from data from the wiki project but the chapter information comes from the story Git repository. The gather step let me combine both of those together to generate those links.

That also meant that I needed to have some supplementary information stored in the gather step (which was written to a //build folder) to help with those linking.

Near the end, once Cobblestone began to break under its abstraction and complexity, I wanted to get away from writing my own and switched to GatsbyJS which made a very pretty website, but it was painful to debug and develop. It only lasted a year or so before I wanted to move (though the Gatsby version of Fedran was probably the prettiest I had ever made). If anything, Gatsby ended up being an ecosystem that needed me to pay attention to update things and my cycle of coming around to update a project was too far for it so I spent days trying to get it working again to build it.

So I tried out Statiq. I never successfully made a website out of it, but it introduced me to some really interesting concepts including pulling in data from other sources (Gatsby also did this, just never really used it). But with the C#, it had some really interesting problems. However, I quickly found that I was trying to do something that was outside of the planned usage, which involved me trying to extend an enumeration which involved basically rewriting large hunks of the system just to handle it. Plus, I really like Autofac and I could never get it to play well with that.

Which lead me into creating Nitride.

Now, I want to say, both Gatsby and Statiq are both good systems. They just aren't good systems for me. These don't do what I want them to do and they are designed around patterns that I'm not comfortable with. Even with the desire not to write my own, it wasn't enough.

Entity Component Systems

Nitride is build on MfGames.Gallium which basically means the formal name of the system Gallium Nitride:

Gallium nitride (GaN) is a binary III/V direct bandgap semiconductor commonly used in blue light-emitting diodes since the 1990s. The compound is a very hard material that has a Wurtzite crystal structure. Its wide band gap of 3.4 eV affords it special properties for applications in optoelectronics, high-power and high-frequency devices. For example, GaN is the substrate that makes violet (405 nm) laser diodes possible, without requiring nonlinear optical frequency doubling.

Gallium is a small ECS library. It doesn't intend to be fast or efficient, but it was written to match the idiomatic patterns of System.Linq. I wanted something that wasn't going to reinvent the wheel, which also meant not making custom collection classes or lookup tables. Instead, I wanted to work with BCL primitives which is why much of the system is based on IEnumerable<Entity>.

There is some high-level documentation for Gallium on its project page and its test, but I'll give the basics here.

The core object is MfGames.Gallium.Entity. This is a small class that has an integer identifier and basically a Dictionary<Type, object> called Components. It has some methods to make it easier, but basically you can set or get components from it:

// This creates an empty entity.
Entity entity1 = new();

// This creates an entity with four components:
//   "System.Int32": 123
//   "System.String": "bob"
//   "System.IO.FileInfo": ...
//   "System.IO.DirectoryInfo": ...
Entity entity2 = new()
    .Set(123)
    .Set("bob")
    .SetAll(
        new FileInfo("/tmp/bob.txt"),
        new DirectoryInfo("/tmp")
    );

Assert.True(entity2.HasComponent<FileInfo>);
Assert.False(entity2.HasComponent<decimal>);

Assert.True(123, entity2.Get<int>());

In effect, the class of the component is the key. There are a number of other functions, the test class is probably the best documentation at this point.

The systems part of the system is the LINQ-inspired collection classes. Most of them work off IEnumerable<Entity> with the test class being the bulk of the documentation:

// Set up a list of entities.
List<Entity> list = ...;

// This gets all the entities that have a FileInfo.
var fileOnlyList = list.WhereEntityHas<FileInfo>().ToList();

// This gets all entities that have both a FileInfo and an int.
var fileAndIntList = list.WhereEntityHasAll<FileInfo, int>().ToList();

// Get a modified list based on components. This looks for all
// entities that have a FileInfo and then adds a marker component,
// IsHtml, into it.
IsHtml isHtml = IsHtml.Instance;

var modifiedList = list
    .SelectEntity<FileInfo>((entity, file) => entity.Set(isHtml))
    .ToList();

As a note, Entity is “mostly” immutable. The Set command will return a new Entity instance with the same ID but with a different set of components. The “mostly” bit comes that the objects that it points to can be mutated.

🛈 The lack of formal documentation for Gallium and Nitride is one of the reasons this is still alpha or beta level code.

Operations

The next concept needed for today's post is an operation which extends MfGames.Nitride.IOperation. These are just easy ways of creating some functionality that works on an IEnumerable<Entity> that can be combined together.

Naturally, there are some extension methods that make it easier to work with operations with an IEnumerable<Entity>:

IEnumerable<Entity> entityList = ...;

IEnumerable<Entity> otherList = someOperation1.Run(entityList);

IEnumerable<Entity> resultList = otherList
    .Run(someOperation2)
    .Run(someOperation3)
    .Run(someOperation4);

This makes it relatively easy to create easy processing through composition of one or more operations. Alternatively, we could easily just work directly off the enumerable.

Zio

There is no question, I'm fond of the Zio library. In this case, Nitride uses two big features with it: to create an abstract view of the project that is independent of the working directory and the UPath class which normalizes paths.

If you remember from the previous posts, we set a root directory in the Program.cs:

public static async Task<int> Main(string[] args)
{
    var rootDirectory = new DirectoryInfo(Environment.CurrentDirectory);

    var builder = new NitrideBuilder(args)
        .UseIO(rootDirectory)
        .UseModule<WebsiteModule>();

    return await builder.RunAsync();
}

This caused an Zio.IFileSystem to be injected in the system which treats that directory as the /. So, in our case where we had our index file in //src/pages/index.md, it has a UPath of /src/pages/index.md. It doesn't seem like a lot, but it means that all paths in the system are always relative to the root of the project and there is no need to pass a root directory throughout the system which means path operations are considerably simplified.

And the UPath ensures a consistent pattern for accessing so this works on Windows, Mac, and Linux without a problem. Not to mention, any issues with .. are handled automatically.

It also simplifies writing tests.

Copy File in Six Files

Put all that together, we can use two operations to create a massively over-engineered copy file operation using Nitride. Admittedly, the complexity will make sense later, but at the moment, we're going to read the file in the PagesPipeline, through SimplifiedMarkdownPipeline, through StyleHtmlPipeline, and finally write it out in OutputHtmlPipeline.

Making Some Noise

To start with, let's put some logging to show how everything flows from pipeline to pipeline. To do that, we're going to inject an ILogger<> into the constructor into all four pipelines and then put a logging message into the last three (the PagesPipeline is going to do something different).

// In //src/dotnet/Pipelines/Content/SimplifiedMarkdownPipeline.cs
public class SimplifiedMarkdownPipeline : PipelineBase
{
    private readonly ILogger _logger;

    public SimplifiedMarkdownPipeline(
        ILogger<SimplifiedMarkdownPipeline> logger,
        PagesPipeline pagesPipeline)
    {
        _logger = logger;

        AddDependency(pagesPipeline);
    }

    public override IAsyncEnumerable<Entity> RunAsync(
        IEnumerable<Entity> entities,
        CancellationToken cancellationToken = default)
    {
        var list = entities.ToList();

        _logger.LogInformation("Reading {Count:N0} entities", list.Count);

        return list.ToAsyncEnumerable();
    }
}

Internally, Nitride uses Serilog, so using {Count:N0} does the right thing even thought we are using a logging abstraction for the actual call. When we run this, we get the following output (I'm not going to show the just build call for these today):

[14:00:46 DBG] <DirectoryService> Setting root directory and filesystem to /home/dmoonfire/src/typewriter-press/typewriter-press-website
[14:00:46 DBG] <BuildCommand> Processing 0 pipeline options
[14:00:46 INF] <BuildCommand> Running pipelines
[14:00:46 INF] <SimplifiedMarkdownPipeline> Reading 0 entities
[14:00:46 INF] <StyleHtmlPipeline> Reading 0 entities
[14:00:46 INF] <OutputHtmlPipeline> Reading 0 entities
[14:00:46 INF] <PipelineManager> Completed in 00:00:00.1079413
[14:00:46 INF] <BuildCommand> Command took 00:00:00.1097142

🛈 I moved NuGet.config to the root level because there is where the Website.sln file was located and it felt right.

Reading Files

Now, to read a file, we are going to use the MfGames.Nitride.IO.ReadFiles operation which reads file from the Zio file system and passes them through. We've already included that NuGet package in a previous step, so nothing to do there.

// In //src/dotnet/Inputs/PagesPipeline.cs
using MfGames.Gallium;
using MfGames.Nitride.IO.Contents;
using MfGames.Nitride.Pipelines;
using Microsoft.Extensions.Logging;

namespace Generator.Pipelines.Inputs;

public class PagesPipeline : PipelineBase
{
    private readonly ILogger _logger;
    private readonly ReadFiles _readFiles;

    public PagesPipeline(
        ILogger<PagesPipeline> logger,
        ReadFiles readFiles)
    {
        _logger = logger;
        _readFiles = readFiles.WithPattern("/src/pages/**/*.md");
    }

    public override IAsyncEnumerable<Entity> RunAsync(
        IEnumerable<Entity> entities,
        CancellationToken cancellationToken = default)
    {
        var list = _readFiles
            .Run(cancellationToken)
            .ToList();

        _logger.LogInformation(
            "Read in {Count:N0} files from /src/pages",
            list.Count);

        foreach (var entity in list)
        {
            _logger.LogInformation("Entity: Path {Path}, Components {ComponentList}",
                entity.Get<UPath>(),
                entity.GetComponentTypes().Select(x => x.FullName));
        }

        return list.ToAsyncEnumerable();
    }
}

This requires a bit of explanation. We are using dependency injection to insert the ReadFiles operation into the constructor. This is not a singleton, so you can have multiple ReadFiles if you need to read from multiple locations (or use a globbing operator to do it in one).

_readFiles = readFiles.WithPattern("/src/pages/**/*.md");

In the constructor, I configure it. I could do this in the RunAsync function but I found it made it easier to parse the files when I configure in the constructor. Like most things in Nitride, this uses a fluent coding style where each object sets the properties and returns itself to make a nice chained list of calls.

There are two properties being set, Pattern and RemovePathPrefix. I use a source generator to automatically create the With... pattern for fluent coding. The first one tells which files to read, relative to the root directory. This happen to use a globbing that allows me to grab all Markdown files in the source directory.

var list = _readFiles
    .Run(cancellationToken)
    .ToList();

This run the _readFiles operation and gets a list of the files it read in. (I'm also going to put this in the OutputHtmlPipeline to show how everything writes out.)

foreach (var entity in list)
{
    _logger.LogInformation(
        "Entity: Path {Path}, Components {ComponentList}",
        entity.Get<UPath>(),
        entity.GetComponentTypes().Select(x => x.FullName));
}

This is just some debugging to show how things are working and how Gallium works. In this case, the read files sets two components: the UPath for the path of the file and a IBinaryContent which is an abstraction to the binary content. Since we haven't done anything to the file, this just points to the file system itself without loading it into memory. That way, we can easily deal with gigabyte-sized files without overloading the system.

return list.ToAsyncEnumerable();

With this, we return the list of files that we just read in. With this change, running the build gets us this:

[14:20:03 DBG] <DirectoryService> Setting root directory and filesystem to /home/dmoonfire/src/typewriter-press/typewriter-press-website
[14:20:03 DBG] <BuildCommand> Processing 0 pipeline options
[14:20:03 INF] <BuildCommand> Running pipelines
[14:20:04 INF] <PagesPipeline> Read in 1 files from /src/pages
[14:20:04 INF] <PagesPipeline> Entity: Path /src/pages/typewriter/index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]
[14:20:04 INF] <SimplifiedMarkdownPipeline> Reading 1 entities
[14:20:04 INF] <StyleHtmlPipeline> Reading 1 entities
[14:20:04 INF] <OutputHtmlPipeline> Writing out 1 files
[14:20:04 INF] <OutputHtmlPipeline> Entity: Path /src/pages/typewriter/index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]
[14:20:04 INF] <PipelineManager> Completed in 00:00:00.9543639
[14:20:04 INF] <BuildCommand> Command took 00:00:00.9559424

As you can see, that single file read in is then passed through the other pipelines because each of them return their input list and then the final pipeline gets those same entities and paths.

Changing Paths

Now the problem with the above example is that we are writing our output to the same location as we are reading. That isn't optional. What we want to do is strip off the /src/pages/typewriter from the beginning of the read in path and add /build/typewriter/html in front of it. This is easily done with two additional operators, RemovePathPrefix and AddPathPrefix.

To start with, let's remove the prefix from the read methods to normalize the paths.

// In //src/dotnet/Inputs/PagesPipeline.cs
public PagesPipeline(
    ILogger<PagesPipeline> logger,
    ReadFiles readFiles,
    RemovePathPrefix removePathPrefix)
{
    _logger = logger;
    _removePathPrefix = removePathPrefix.WithPathPrefix("/src/pages/typewriter");
    _readFiles = readFiles.WithPattern("/src/pages/**/*.md");
}

And then we chain the operation using an extension method on operations:

// In //src/dotnet/Inputs/PagesPipeline.cs
public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    var list = _readFiles
        .Run(cancellationToken)
        .Run(_removePathPrefix)
        .ToList();

Since each operation also takes an IEnumerable<Entity> and removes the same type, they can flow from one operation to another. When we run the build, the output line looks like this:

[14:23:35 INF] <PagesPipeline> Read in 1 files from /src/pages
[14:23:35 INF] <PagesPipeline> Entity: Path /index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]

Adding the output prefix is done on the output step:

// In //src/dotnet/Html/OutputHtmlPipeline.cs
public OutputHtmlPipeline(
    ILogger<OutputHtmlPipeline> logger,
    AddPathPrefix addPathPrefix,
    StyleHtmlPipeline styleHtmlPipeline)
{
    _logger = logger;
    _addPathPrefix = addPathPrefix.WithPathPrefix("/build/typewriter/html");

    AddDependency(styleHtmlPipeline);
}

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    var list = entities
        .Run(_addPathPrefix)
        .ToList();

Because we already have a list of entities, we can just use the extension method to pipe it through the operation and produce the following output:

[14:28:35 INF] <PagesPipeline> Read in 1 files from /src/pages
[14:28:35 INF] <PagesPipeline> Entity: Path /index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]
[14:28:35 INF] <SimplifiedMarkdownPipeline> Reading 1 entities
[14:28:35 INF] <StyleHtmlPipeline> Reading 1 entities
[14:28:35 INF] <OutputHtmlPipeline> Writing out 1 files
[14:28:35 INF] <OutputHtmlPipeline> Entity: Path /build/typewriter/html/index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]

The reason PagesPipeline doesn't use entities.Run(...) is because that is the first pipeline and the entities is going to be an empty list. Instead, we need to make entities, which is why we call _readFiles.Run() instead.

Writing Files

And the last bit for today's post is to write out files. This, creatively enough, uses the WriteFiles operation.

// In //src/dotnet/Html/OutputHtmlPipeline.cs
public OutputHtmlPipeline(
    ILogger<OutputHtmlPipeline> logger,
    AddPathPrefix addPathPrefix,
    WriteFiles writeFiles,
    StyleHtmlPipeline styleHtmlPipeline)
{
    _logger = logger;
    _writeFiles = writeFiles;
    _addPathPrefix = addPathPrefix.WithPathPrefix("/build/typewriter/html");

    AddDependency(styleHtmlPipeline);
}

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    var list = entities
        .Run(_addPathPrefix)
        .Run(_writeFiles)
        .ToList();

In this case, we've already done all the fancy operations to adjust the path so it writes out the files in the proper location:

$ find build -type f
build/typewriter/html/index.md

Convenience Methods

Now, adding and removing path prefixes is pretty common, so both the ReadFiles and WriteFiles have methods to do those in a single call.

// In //src/dotnet/Inputs/PagesPipeline.cs
public PagesPipeline(
    ILogger<PagesPipeline> logger,
    ReadFiles readFiles)
{
    _logger = logger;
    _readFiles = readFiles
        .WithPattern("/src/pages/typewriter/**/*.md")
        .WithRemovePathPrefix("/src/pages/typewriter");
}

// In //src/dotnet/Html/OutputHtmlPipeline.cs
public OutputHtmlPipeline(
    ILogger<OutputHtmlPipeline> logger,
    WriteFiles writeFiles,
    StyleHtmlPipeline styleHtmlPipeline)
{
    _logger = logger;
    _writeFiles = writeFiles.WithAddPathPrefix("/build/typewriter/html");

    AddDependency(styleHtmlPipeline);
}

What's Next

And there you go, an over-engineered system for copying a single file from one directory to another. In the next post of this series, we'll turn that Markdown file into an HTML page.

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

Using Nitride - Pipelines

2025-06-08T05:00:00Z

I found a good way of working on a project is being able to see the end product. That means forging a minimal path from the start to end, and then going back to flesh it out. Interestingly, this is the opposite of how I write novels because, in those, the journey is the important part. Not so much with development projects.

In this case, the goal is to get a website generated and the ability to see it locally.

Series

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

Project Setup

Since this mainly a .NET project, we start by creating a new project and solution. This is a fairly simple generator and I don't plan on needing more than one assembly, so I'm going to throw everything into //src/dotnet/ for the website.

$ dotnet new console --name Generator --output src/dotnet
$ dotnet new solution --name Website
$ dotnet sln add src/dotnet/Generator.csproj

And since we want to have the project run, we also update the Justfile:

# //Justfile

# Builds the typewriter.press website
build-typewriter:
    dotnet run --project src/dotnet/Generator.csproj --

And that gives us an easy command to run that works anywhere underneath the Git repository (thanks to Just).

$ just build-typewriter 
dotnet run --project src/dotnet/Generator.csproj --
Hello, World!

Setting up NuGet

At the moment, the MfGames.* assemblies are not on nuget.org. There are a couple of reasons, mostly self-doubt that anyone would find this useful and the dread of setting up a SSL certificate which appeared to be required. My current intent is to do it once I have a number of sites using Nitride or I get a request to do so. Until then, I use NuGet.config file to hook up my libraries to the website.

<!-- //src/dotnet/NuGet.config >
<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="mfgames.com" value="https://src.mfgames.com/api/packages/mfgames-cil/nuget/index.json" protocolVersion="3" />
  </packageSources>
  <packageSourceMapping>
    <packageSource key="nuget.org">
      <package pattern="*" />
    </packageSource>
    <packageSource key="mfgames.com">
      <package pattern="MfGames.*" />
    </packageSource>
  </packageSourceMapping>
</configuration>

Setting up the Generator

While MfGames.Nitride could use configuration files to do the bulk of the setup, I went with a configuration-as-code approach to it since I almost always end up writing custom code. That way, everything is done in the same manner. This also uses a model roughly based on the generic host for command line.

🛈 I'm in the process of reworking the initial setup to use the generic host more completely. The current implementation was my best approach to write a dependency-injected console application that is configured via the CLI instead of JSON files.

The first step is to include the required package. Nitride is organized in a lot of small, focused packages. This is it avoid pulling in too much for a given project but also to help keep everything organized. For the initial work, we can add the first two required assemblies.

$ cd src/dotnet
$ dotnet add package MfGames.Nitride
$ dotnet add package MfGames.Nitride.IO
$ dotnet add package Autofac

Nitride using the excellent Zio library for abstracting file system access, plus making it easier to write test to make sure everything work. We also use Autofac for a lot of the module registration and processing.

🛈 I'm working on removing the Autofac dependency and using Microsoft.Extensions.DependencyInjection instead but it will take a while. There are a lot of useful features in Autofac that the extensions don't have so it will take some time.

First, we create the website module with a very generic, naive implementation (unless otherwise mentioned, all of these files are in //src/dotnet/):

// WebsiteModule.cs
using Autofac;

namespace Generator;

public class WebsiteModule : Module
{
    /// <inheritdoc />
    protected override void Load(ContainerBuilder builder)
    {
        builder
            .RegisterAssemblyTypes(GetType().Assembly)
            .AsSelf()
            .AsImplementedInterfaces()
            .SingleInstance();
    }
}

With that, we change the initial program to use Nitride:

// Program.cs
using MfGames.Nitride;
using MfGames.Nitride.IO.Setup;

namespace Generator;

public static class Program
{
    public static async Task<int> Main(string[] args)
    {
        var rootDirectory = new DirectoryInfo(Environment.CurrentDirectory);

        var builder = new NitrideBuilder(args)
            .UseIO(rootDirectory)
            .UseModule<WebsiteModule>();

        return await builder.RunAsync();
    }
}

If we run this, we'll get an error:

$ just build-typewriter 
dotnet run --project src/dotnet/Generator.csproj --
Required command was not provided.

Description:

Usage:
  MfGames.Nitride [command] [options]

Options:
  --log-level <log-level>                    Controls the verbosity of the output, not case-sensitive and prefixes allowed: 
                                             Verbose, Debug, Information, Warning, Error, Fatal [default: Warning]
  --log-context-format <log-context-format>  Controls the format of the source context for log items, not case-sensitive and 
                                             prefixes allowed: None, Class, Full [default: Class]
  --log-time-format <log-time-format>        Controls the format of the time in the log messages, such as 'HH:mm:ss' (default) or 
                                             'yyyy-MM-ddTHH:mm:ss'. Blank means exclude entirely. [default: HH:mm:ss]
  --log-level-override <log-level-override>  Overrides log levels for certain contexts in the format of either 'Context' or 
                                             'Context=Level' (repeat for multiple) [default: Microsoft=Warning]
  --version                                  Show version information
  -?, -h, --help                             Show help and usage information

Commands:
  build  Generate the website
  watch  Generates the site and then watches for changes

error: Recipe `build-typewriter` failed on line 15 with exit code 1

The watch command doesn't work very well right now, but we want build, so we'll update our Justfile:

# //Justfile
build-typewriter:
    dotnet run --project src/dotnet/Generator.csproj -- build

And then try again:

$ just build-typewriter
dotnet run --project src/dotnet/Generator.csproj -- build
[12:27:17 ERR] <PipelineManager> There are no registered pipelines run, use ConfigureContainer to include IPipeline instances
error: Recipe `build-typewriter` failed on line 15 with exit code 1

Pipelines

Now we get to the first big concept of Nitride: pipelines. Some of the ideas are built up from Statiq and also the patterns I've worked out with CobblestoneJS. They basically are larger “units” of processing for the site that are intended to gather a bunch of files and then feed them into another pipeline which then splits out into different ones for output.

It also allows a group of inputs to be handled differently, such as linking the “next” and “previous” to posts which doesn't make for pages. Another case where I've used this heavily is to copy binary assets directly to the styled output, generate CSS through webpack or esbuild, or generate project-specific pages from the input.

        +-------+ +-------+
        | Pages | | Posts |
        +-------+ +-------+
            |         |
            +----+----+
                 | 
                 v
          +-------------+
          |  Simplified |
          |   Markdown  |
          +-------------+
                 |
     +-----------+-----------+
     |           |           |
     v           v           v
+---------+ +---------+ +---------+
|  Style  | |  Atom   | | Style   |
|  HTML   | |  Feeds  | | Gemtext |
+---------+ +---------+ +---------+
     |                       |
     v                       v
+---------+             +---------+
|  Output |             | Output  |
|  HTML   |             | Gemtext |
+---------+             +---------+

Pipelines attempt to run in parallel to reduce the amount of time on the developer's machine. They are also asynchronous at their core since a pipeline could be used to query another system, require long-running processes, or perform some other action that would require using that functionality.

Obviously, you could use a single pipeline. I use to start that way, but the pattern of gathering the information, combining them together to generate tags and categories, and then styling fits better as separate steps for me. Since this is my website, we're going to create a minimum from the pages, generate simplified Markdown, style it, and then write it out.

Since we use dependency injection throughout Nitride, we pass in the dependencies into constructor of a pipeline and then add it to that project. Internally, Nitride runs every pipeline in a separate task when it is building the site.

// ./Pipelines/Inputs/PagesPipeline.cs
using MfGames.Gallium;
using MfGames.Nitride.Pipelines;

namespace Generator.Pipelines.Inputs;

public class PagesPipeline : PipelineBase
{
    public override IAsyncEnumerable<Entity> RunAsync(
        IEnumerable<Entity> entities,
        CancellationToken cancellationToken = default)
    {
        throw new NotImplementedException();
    }
}

// ./Pipelines/Content/SimplifiedMarkdownPipeline.cs
using Generator.Pipelines.Inputs;
using MfGames.Gallium;
using MfGames.Nitride.Pipelines;

namespace Generator.Pipelines.Content;

public class SimplifiedMarkdownPipeline : PipelineBase
{
    public SimplifiedMarkdownPipeline(
        PagesPipeline pagesPipeline)
    {
        AddDependency(pagesPipeline);
    }

    public override IAsyncEnumerable<Entity> RunAsync(
        IEnumerable<Entity> entities,
        CancellationToken cancellationToken = default)
    {
        throw new NotImplementedException();
    }
}

// ./Pipelines/Html/StyleHtmlPipeline.cs
using Generator.Pipelines.Content;
using MfGames.Gallium;
using MfGames.Nitride.Pipelines;

namespace Generator.Pipelines.Html;

public class StyleHtmlPipeline : PipelineBase
{
    public StyleHtmlPipeline(
        SimplifiedMarkdownPipeline simplifiedMarkdownPipeline)
    {
        AddDependency(simplifiedMarkdownPipeline);
    }

    public override IAsyncEnumerable<Entity> RunAsync(
        IEnumerable<Entity> entities,
        CancellationToken cancellationToken = default)
    {
        throw new NotImplementedException();
    }
}

// ./Pipelines/Html/OutputHtmlPipeline.cs
using MfGames.Gallium;
using MfGames.Nitride.Pipelines;

namespace Generator.Pipelines.Html;

public class OutputHtmlPipeline : PipelineBase
{
    public OutputHtmlPipeline(
        StyleHtmlPipeline styleHtmlPipeline)
    {
        AddDependency(styleHtmlPipeline);
    }

    public override IAsyncEnumerable<Entity> RunAsync(
        IEnumerable<Entity> entities,
        CancellationToken cancellationToken = default)
    {
        throw new NotImplementedException();
    }
}

And since we want to avoid exceptions, we'll ignore the entire Entity bit and do a couple of changes that will be explained in the next post.

In PagesPipeline.cs, we want to return an empty list:

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    return Array.Empty<Entity>().ToAsyncEnumerable();
}

For all the others, we're just going to pass whatever we got in back out:

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    return entities.ToAsyncEnumerable();
}

I'll admit, I stumbled into the usage of IAsyncEnumerable. It lets me use await calls inside a function without needing to load an entire list into memory just to copy it out again into another list to make the pipelines to work. Given that one of my baseline websites was to process my entire fiction website without killing my machine, I want to reduce the number of copies in memory and reduce the pressure there.

Using IAsyncEnumerable is cumbersome though, so might I might create a convenience method in PipelineBase that allows for a Task<IEnumerable<Entity>> and change it an IAsyncEnumerable but that is a future idea to work out.

When we run this, we get this (after adding ``–log-level debuginto the//Justfile`):

$ just build-typewriter 
dotnet run --project src/dotnet/Generator.csproj -- build --log-level debug
[13:22:48 DBG] <BuildCommand> Processing 0 pipeline options
[13:22:48 INF] <BuildCommand> Running pipelines
[13:22:48 INF] <PipelineManager> Completed in 00:00:00.1051617
[13:22:48 INF] <BuildCommand> Command took 00:00:00.1075694

It doesn't really do anything in the 0.1 seconds that it takes to run, but it basically gets the pipeline chain working so things will be generating with the next in this series.

Observations

I'm also aware that this appears to be overkill for most websites. If I was doing an HTML-only website, I would have started with only a single pipeline but I already know that I'm going to generate Atom feeds out of that and they work better with simple HTML instead of fancy chrome. Being able to style the Atom's output separately gives me some flexibility to do different things there. Also there are going to be multiple inputs eventually so experience tells me that this is the “best” approach for generating output given the circumstances of this specific website.

You'll also notice that I'm not doing a site-specific pipelines. I've tried a couple of approached and my current one is to run the entire generation process once per website using some configuration settings, which I'll get after I finish the main website.

Future Plans

The pipelines also are the units that are triggered on the watch command (which doesn't really work). If a pipeline identifies that it needs to be run, it will run but then trigger a re-run of every pipeline that depends on that.

The advantage of that if I'm working on styling CSS, I can have a pipeline feeding into the StyleHtmlPipeline that generates that. When the hypothetical CssHtmlPipeline triggers a rebuild, it runs itself, StyleHtmlPipeline, and OutputHtmlPipeline only because of those dependencies. But changing a page would trigger everything but the CssHtmlPipeline which should keep watches fairly performant.

What's Next

In the next post, I'll explain what Entity is and start make this actually generate something useful.

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

Using Nitride - Introduction

2025-06-07T05:00:00Z

One drawback of having a lot of irons in the fire is occasionally projects that should not have been forgotten are and they bitrow. A good example is the Typewriter Press and it's associated reflected websites:

Sassy - Romance novels
Dusty - Historical fiction
Broken - Fantasy including contemporary

Eventually, when I start publishing my sci-fi stories, they would go under “Electric”. I call them “reflected” sites because they are just filtered versions of the main Typewriter site with some slightly different branding. That way, if I wrote more romance, I could put them there and folks could go to the romance-only site or jump over to the main one to see that I have no single genre (much like I don't have a single genre for my writing).

However, it's been years since I've updated those sites and they are beginning to creak underneath their age. Not to mention, I haven't included the last two books I published on them and that is humiliating for me.

So, I decided to switch the sites over to MfGames.Nitride and document the process.

Series

This is going to be broken into multiple posts, mostly because I know I'm going to get distracted around the fifteenth of the month when I switch to writing, but also to keep these topic-focused.

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

Conventions

My usual conventions when giving file names is to use // for the Git repository root. So, //flake.nix means the flake.nix in the root level of the project. Also, I'll add a trailing / when I'm talking specifically about a directory, such as //node_modules/.

NixOS Flakes

Since I'm going to be creating a new site instead of converting the existing, I can easily start from the beginning. That usually means starting with a NixOS flake which is my preferred way of writing.

# //flake.nix
{
  inputs = {
    nixpkgs.url = "nixpkgs/nixos-25.05";
    flake-utils.url = "github:numtide/flake-utils";
    mfgames-project-setup.url = "git+https://src.mfgames.com/nixos-contrib/mfgames-project-setup-flake.git";
  };

  outputs = inputs @ { self, nixpkgs, flake-utils, mfgames-project-setup }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        project-config = mfgames-project-setup.lib.mkConfig {
          inherit system;
          pkgs = nixpkgs.legacyPackages.${system};
          dotnet.enable = true;
          prettier.proseWrap = "never";
        };
      in
      {
        devShell = pkgs.mkShell {
          packages = [
            # Development
            pkgs.dotnet-sdk
            pkgs.nodejs_20

            # Local Serving
            pkgs.miniserve
            pkgs.agate
          ]
          ++ project-config.packages;

          shellHook = project-config.shellHook;
        };
      });
}

There really isn't much to this other than my use of mfgames-project-setup-flake. This is a flake that sets up a lot of the common elements I use: a consistent .editorconfig, treefmt for formatting various parts of the system, lefthook for setting up conventional commits.

And naturally, I have to set up direnv because I want to set up my environment as soon as I change into the directory.

# //.envrc
use flake || use nix

dotenv_if_exists .env

PATH_add $PWD/node_modules/.bin

I'm adding the node_modules path because I know I'm going to be using webpack as part of this site.

Once those two files are in, then I can get my basic setup.

$ git add flake.nix .envrc
$ direnv allow
direnv: nix-direnv: Renewed cache
nixago: updating repository files
nixago: '.conform.yaml' link updated
nixago: '/.conform.yaml' added to .gitignore
nixago: '.editorconfig' copy created
nixago: 'lefthook.yml' link updated
Error: unknown shorthand flag: 'a' in -a
Error: unknown shorthand flag: 'a' in -a
nixago: '/lefthook.yml' added to .gitignore
nixago: '.prettierrc.json' link updated
nixago: '/.prettierrc.json' added to .gitignore
nixago: 'treefmt.toml' link updated
nixago: '/treefmt.toml' added to .gitignore
mfgames-project-setup: '/.direnv/' added to .gitignore
sync hooks: ✔️ (commit-msg, pre-commit)
direnv: export +AR +AS +AWS_ACCESS_KEY_ID +AWS_BUCKET +AWS_ENDPOINT +AWS_REGION +AWS_SECRET_ACCESS_KEY +CC +CONFIG_SHELL +CXX +DOTNET_CLI_TELEMETRY_OPTOUT +DOTNET_NOLOGO +DOTNET_SKIP_FIRST_TIME_EXPERIENCE +DOTNET_SKIP_WORKLOAD_INTEGRITY_CHECK +HOST_PATH +IN_NIX_SHELL +LD +MSBUILDALWAYSOVERWRITEREADONLYFILES +MSBUILDTERMINALLOGGER +NIX_BINTOOLS +NIX_BINTOOLS_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu +NIX_BUILD_CORES +NIX_CC +NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu +NIX_CFLAGS_COMPILE +NIX_ENFORCE_NO_NATIVE +NIX_HARDENING_ENABLE +NIX_LDFLAGS +NIX_STORE +NM +NODE_PATH +OBJCOPY +OBJDUMP +RANLIB +READELF +SIZE +SOURCE_DATE_EPOCH +STRINGS +STRIP +__structuredAttrs +buildInputs +buildPhase +builder +cmakeFlags +configureFlags +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +doCheck +doInstallCheck +dontAddDisableDepTrack +mesonFlags +name +nativeBuildInputs +out +outputs +patches +phases +preferLocalBuild +propagatedBuildInputs +propagatedNativeBuildInputs +shell +shellHook +stdenv +strictDeps +system ~PATH ~XDG_DATA_DIRS
$ l
total 64K
drwxr-xr-x  5 dmoonfire users 4.0K Jun  7 11:10 .
drwxrwxr-x 18 dmoonfire users 4.0K Dec 26 08:08 ..
lrwxrwxrwx  1 dmoonfire users   56 Jun  7 11:10 .conform.yaml -> /nix/store/pckycri07q2ah90swk6hf21ilsi59a4s-conform.yaml
drwxr-xr-x  4 dmoonfire users 4.0K Jun  7 11:10 .direnv
-rw-r--r--  1 dmoonfire users 5.2K Jun  7 11:10 .editorconfig
-rw-------  1 dmoonfire users  259 Jun  7 10:40 .env
-rw-r--r--  1 dmoonfire users  13K Jun  7 11:10 flake.lock
-rw-r--r--  1 dmoonfire users  983 Jun  7 11:03 flake.nix
drwxrwxr-x  8 dmoonfire users 4.0K Jun  7 11:10 .git
-rw-r--r--  1 dmoonfire users  139 Jun  7 11:10 .gitignore
lrwxrwxrwx  1 dmoonfire users   56 Jun  7 11:10 lefthook.yml -> /nix/store/ihdb9vnc6jp535m43ndas79s4p9viyjn-lefthook.yml
lrwxrwxrwx  1 dmoonfire users   59 Jun  7 11:10 .prettierrc.json -> /nix/store/mjdm21r27r4n43aq92qkvmdcvy0l2qrc-prettierrc.json
-rw-r--r--  1 dmoonfire users 1.2K Jun  7 10:40 README.md
lrwxrwxrwx  1 dmoonfire users   56 Jun  7 11:10 treefmt.toml -> /nix/store/4lk6rmb2khs5l3yn91rah9y4c6zmxybf-treefmt.toml

Probably one of my favorite things is that it also sets up the initial .gitignore file:

$ cat .gitignore
# nixago: ignore-linked-files
/treefmt.toml
/.prettierrc.json
/lefthook.yml
/.conform.yaml
# mfgames-project-setup: ignore-files
/.direnv/

Overall, using that just cuts out that first hour of starting up a new project. And it lets me evolve my changes to style and different tools by simply running nix flake update.

Directory Layout

Related to using a flake to setup a lot of the boilerplate, I have written my thoughts on setting up polyglot projects such as this in a consistent manner. I think it is a relatively short digital garden plot, but there are a couple applicable things.

Source code goes into //src/ and the output is going into //build (which has an entry in .gitignore). One of the artifacts of my WordPress days is that I have static pages, which I put into //src/pages/, and blog posts which go into //src/posts/. There is some magic that goes on with posts, but that's a different post.

$ find src
src
src/pages
src/pages/typewriter
src/pages/typewriter/index.md
src/posts
src/posts/2025-07-01-website-redesign.md

I put an extra layer of directories under //src/pages/ for a site “slug” to identify which site the page will go on. In this case, I'm going to have typewriter for the main site, sassy, dusty, broken, and electric. There will also be a common for pages that are shared across all pages such as the contact page.

For output, those same slugs are going to be used. I'm intending to have this output both HTML and Gemtext pages for HTTPS and Gemini respectively. I know not a lot of people use Gemini these days, but I still think it is useful and there is a certain minimalism to it that appeals to me.

find build
build
build/typewriter
build/typewriter/html
build/typewriter/gemtext

Just

Another part of my standard project layout is using Just to handle the build system. There are a number of reasons, but I'm going to be dealing with Node programs, such as webpack, and also .NET projects for the build. There is also CLI executables for hosting locally (miniserv and agate).

I made a conscious decision not to write wrappers for Node and the other CLIs into Nitride. There is the ability to execute, but I don't want to tightly tie the site generator to a specific tool. Not to mention, most of the time, I want to set them up in different tabs to watch them so I want a more “generic” task runner than the ones built into Node or .NET which have some biases I've struggled with.

Just is something that is self-contained, doesn't make a lot of assumptions, and doesn't drag an entire ecosystem in with it.

# //Justfile
set dotenv-load

_default:
    just --choose

# Cleans up the project without removing the local .env
clean:
    git clean -xfd --exclude .env

# Builds all the websites
build: build-typewriter

# Builds the typewriter.press website
build-typewriter:
    echo I did something

# Serves the Typewriter website
serve-typewriter-html:
    miniserve --index index.html build/typewriter/html

# Serves the Typewriter Gemini pod
serve-typewriter-gemtext:
    mkdir -p .cache/agate
    agate --content build/typewriter/gemtext --certs .cache/agate --hostname localhost

Naturally, .cache/ has to be added to the .gitignore file. I use .cache from the project layout plot.

What's Next

Now that we have the basic boilerplate for a new project, time to start adding .NET into the mix.

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

Why SeaweedFS?

2024-11-08T06:00:00Z

I got an email last week asking if I could explain why I set up a Seaweed server. It's been a while since I talked about it, mostly in hints in this post from 2022 and this post from 2024. The request gives me an opportunity to expand on it, and to remind myself why I do it.

Stories

In some regards, there is a bit of trauma in me when it comes to losing out on things. This includes hearing stories that my father told me and realizing that when he died, they were gone forever. These same thoughts came up in my novel Sand and Bone:

The little girl's confessions echoed in Rutejìmo's head in a quiet symphony as he wrote in his book. The hand-bound collections of pages creaked under his hand, the leather thong strained to hold the almost fifty pages of tightly-spaced writing. Over the years, he had added a dozen pages to the collection. It wouldn't be too long before the binding couldn't handle the additional pages, but he thought he had a few more years left before that happened.

Even with his additional pages, he didn't have room to write down all of the stories he had heard over the years. He wanted to detail the joys of the little girl's death, such as the choked story about how she had stolen her brother's toy when he wasn't looking. He had also wanted to write the horrors, like one man's confession for killing his sister. Each one was precious and important. Time would erase their stories and a part of Rutejìmo died every time he forgot one.

— Sand and Bone 8: Alone

This fear is more than just one person, losing stories of things that happened decades ago as told by an old man. This includes my own stories, which will also be lost when I die. There is little chance I will be “done” with Fedran. There is little chance I will get all of the stories out of my head and onto paper. There isn't enough time, I don't have the self-esteem to do it, and I don't have the drive. When I'm gone, so are those stories. But, even going beyond that, there are so many stories of horror and joy that I feel we are losing with the passage of time. Happy stories of people falling in love, horror stories of Auschwitz, and everything in between. It is that breadth of stories, lessons, and happiness that we lose.

But the big question is, will anyone care? Does anyone want to know the first time I got caught shoplifting. Or the first time I finally said “I love you” to Partner?

Probably not.

But, you never know.

I'm never going to record them, but when I was given my father's artwork and decades of effort, I don't want to lose them yet. I don't want to just chuck it aside. So, I need to keep them.

Hoarding

I come from a long line of hoarders. I realized that as I look at my DVD collection or the boxes of books I have. My LEGO collection isn't huge, but it does fill five or six boxes. I once got rid of it, and I felt guilty for years for doing that; which is why I built it up again. Sometimes, I just play.

My mother did it. She had a massive display case of dragons and cats. She has dozens of sets of china in the basement. Entire room with the bones of a bankrupt lumber mill. A massive library (that I inherited).

My father did it. I spent days going through pieces of paper where he kept every calendar, shred of note, and personal letter he sent to his children. I saw research when I asked for help to get through college, mainly to prove that I was going the wrong path and I really should just drive two hours one way to go the college he thought was a better choice for me. I ended up saying “screw you” and went into debt for a couple decades instead.

My father has terabytes of images that he drew. Decades of him struggling to be a “good” artist, self-doubt and pain. But I'm so happy to watch him to do it. Also, he did the artwork for the nuclear reactor project he was on, and the particle accelerator. And birthday cards for all of his children and grandchildren.

For me, there is a good size of digital hoarding going on. I mean, a couple thousand ebooks is one things, but I have PDFs, for game systems that go back years. The original PDFs of HERO System 5 and 6. Every scan of the Dragon magazine. GURPS. Legal documents, funny stories, and the like. And then there are my DVDs. Back when I didn't have children, I was buying 3-5 a week. When Suncoast Video went out of business, I had a huge refund from the IRS. Walked into the store and said “I'm gunna buy everything I can.” Now, there are DVDs I can't buy anymore, I can't find. But they are slowly rotting in my file cabinet (did you know they only have a 20-30 year shelf life?). I don't want to lose them, even if I don't use them every day.

And then there is Partner's photography business. They do large photo shoots but they also need to keep them around for years “just in case” someone loses their wedding photos. She doesn't “have to”, but I don't want to ever have to say “sorry, I don't have them” for a senior photos or someone's puppies. Or family members are lost.

So I want to keep them.

Storage

All this takes space. The nice thing about digital space is that I can make backups. I can take them with me with a few kilograms of hard drives or upload them into the cloud. I can make copies to make it harder to lose (like when I was hit with a ransomware that took out my media server). The goal is 3-2-1 Backup Strategy: three copies, two locations, and at least one offline copy.

Now, dealing with storage is something I've done for quite a long time. I remember being pulled out of sixth grade school for a day to help my mother recover her RAID 5 box. At the time, drives were in the low 100s of megabytes, but I had to learn a lot about how RAID worked to figure it out. Then watching the slow recovery that took an entire day… then having another drive fail about a day after we recovered the first time and doing it all again. I used and watched hardware and software RAID controllers fail.

At the time, the data wasn't static. We were processing millions of records to do analysis. The drives would get corrupted by poor power, heavy usage, and the eventual failure of mechanical drives. We had customers lose data, watched our data centers blow a disk. Like Partner's photos, we had to keep our analysis for years after because of re-evaluation or the occasional lawsuit.

Over time, it became apparent that there was a maximum size where RAID was useful. After a while, it wasn't a matter “if” a drive goes back but “when” a drive goes bad. Backblaze (where I keep my backups, I recommend them) had a quarterly blog post about drive stats. They track with my own experiences. Yeah, 1.7% failure rate doesn't seem like much but when I'm dealing with stuff that needs to be kept for years, it hits a lot faster than you'd think it would. And offline backups fail too.

When I left my mother's company, I went back a little and just threw some drives into a big machine. Didn't bother with RAID because I knew it would fail since I was looking at a decade of hard drive use and started with a manual mirroring between a couple Windows partitions. But as the DVDs got ripped, music got purchased from Amazon, and photos kept being taken, I started to run out. Soon, it wasn't just Q:\, R:\, and S:\ being copies but each one having a portion of everything.

When I got hit with the ransomware attack, I lost my DVD collection (years of ripping) but not most of Partner's photos and the other things. I started to recover them but one of the drives didn't make it.

Then I lost another drive a few months later.

Then we didn't have money to handle the failure indicators, so I was just watching the SMART notices popping up knowing that there was nothing to do but watch the slow-moving train accident.

Then the media server decided to go down for a week.

Not having access to anything was a stark reminder that I had to do “something” if I didn't want to lose everything. And it wasn't just tossing in another drive every once in a while. I had hit the threshold where I needed to spread it out across more than just drives, I needed to spread it across servers.

Ceph

Enter Ceph. I've read about Ceph for many years before that point and it always appealed to me. It seemed to solve a lot of the problems I've experienced over the years. And it didn't have to require same-sized disks to pull of a RAID. And I could add machines if I needed to add more storage.

The biggest thing with Ceph is that it balanced across multiple servers to reduce failure, but also let you mark files as needing one, two, or more copies. Partner's photo library? Two copies. Videos? One is probably good enough. The servers didn't have to be that powerful either, so I could use my older machines to keep the disk when I had to upgrade to a newer one. In theory, I could use a Raspberry Pi to act as a cluster.

The incident when this happened was when the media server died again. I had saved some money from my commissions (earmarked to get a book edited) and used it to buy fresh hard drives. If anything, replacing the 10+ year old drives with something new and bigger would give me some. I also took the opportunity to switch to NixOS instead of Windows, which I dislike anyways. I mean, NixOS had options for Ceph, how hard could it be?

Apparently, it really wasn't ready for prime time. Eventually the wiki popped up to say that.

Took me a week to figure it out. There was a lot missing (and still is) from the core, but I managed to write up some notes for myself on how to make it work. But, after days of trial and errors, of “almost got it” joys only to watch it crash, I finally got something working. And it was glorious. At least until I realized was swapping like mad and Ceph really needs 1 GB per TB of storage. A few frantic purchases later, and I had a couple more cheap Dell machines (one of the ones that actually died this week) and I ended up with a fairly balanced Ceph cluster.

SeaweedFS

I liked Ceph, but I'm not on it anymore. Partially it was selfish reasons, I want to fix the struggles I had bringing on a new drive into the cluster but the community decided to go a different way. But there was also no one who was able to to do that different way while I was down. To test a fix myself, I had to build for twelve hours just to see if something worked (I also work with 10+ year old computers a lot). I was on unstable, so I went weeks being unable to build because I couldn't downgrade to stable. I also learned that nixpkgs had some patterns that were really hard to get into, like redefining lua at the package level to mean a specific version of lua and no one on the Matrix or Discourse forum was able to tell me that until I happened to find one person who explained it and said I should have just “known” about that mapping.

And then, the same day I realized my efforts to get Ceph working were linked on the NixOS wiki for Ceph, I also saw a little line:

Another distributed filesystem alternative you may evaluate is SeaweedFS.

I vaguely remembered looking at SeaweedFS before, but I was focused on getting Ceph working. And in that moment, when Ceph was not building and I didn't have the resources to fix the problem myself, I decided to try it out.

SeaweedFS does 80% of Ceph. It didn't have all the fancy features, but it had the features I wanted:

Distributed across multiple machines
Ability to add or remove storage on the fly
Variable replication copies
Currently built and could be run
Can be mounted on Linux
It uses 30 GB volumes on standard ext4 partitions instead of a custom one I cannot debug
Single Go executable (don't code Go but it only takes twenty minutes build, not twelve hours)
Could create S3/cloud tier backup (Ceph couldn't do that)

Yeah, it didn't have NixOS options but Google and GitHub gave me a starting point for me to puzzle it out on my own. I learned a new library and started with a little partition to see if it worked. Like Ceph, I had a lots of fits and starts, trials and puzzling through it, but eventually got it working.

And it was more scattered in terms of information but it worked.

It didn't do Ceph's “deep cleaning” to detect bit rot on failing drives.

It blew up when you tried to create too many encoding shards without the space.

It blew up when you tried to replicate without having enough nodes.

Around that time, Ceph started building on NixOS again but I was already enamored by SeaweedFS. It was “good enough” for me. It took me about a week to migrate hunks of the Ceph data to the Seaweed, decommission a Ceph drive and make it a SeaweedFS drive. Then repeat until everything was moved over.

A few weeks ago, I tried to consolidate my dad's hard drives into the cluster and ran out of space. I ended up buying a new minicomputer and throwing 11 TB worth of drives into it to give me room (two copies of everything, even the media files). This week, I lost one of those old Dells so I had to shuffle around the volumes with a few commands and it “just worked”. I'm going to replace the dead computer and I'm confident that it will “just work” then too. And, more importantly, I don't have to spend a week to figure out the commands to make it happen since I can just copy/paste a bunch of Nix code and redeploy my servers.

Thoughts

I could hoard less. It takes time and energy to keep the computers running but it makes sure Partner has their Golden Girls, the kids have their videos, and I have my dad's artwork. Also, there is something peaceful about looking at a 29.9 TB partition and having it working smoothly.

+---------------------------------------------------------------------------------------------------------------+
| 1 fuse device                                                                                                 |
+-------------+-------+-------+-------+-------------------------------+----------------+------------------------+
| MOUNTED ON  |  SIZE |  USED | AVAIL |              USE%             | TYPE           | FILESYSTEM             |
+-------------+-------+-------+-------+-------------------------------+----------------+------------------------+
| /mnt/home   | 29.9T | 20.9T |  9.0T | [#############.......]  69.8% | fuse.seaweedfs | fs.home:8888:/         |
+-------------+-------+-------+-------+-------------------------------+----------------+------------------------+

Eventually this is all going to go away. When I die, my family isn't going to be able to keep it going. Like Rutejìmo's stories and my dad's artwork, all this will fade. But I'm going to keep it going as long as I can. And try to find more pages for my book.

Recovering SeaweedFS

2024-10-25T05:00:00Z

Lately, I've been quite fond of SeaweedFS. It isn't as powerful as Ceph but it considerably easier to maintain and manage. There are some tradeoffs, such as finding bit rotting (when the disks start to fail), but I find it not quite as “fragile” when it comes to using a random collection of Linux machines.

One of the features I want to play with SeaweedFS is the ability to upload a directory transparently to a S3 bucket (not AWS though, they are too big). I'm thinking about that for later, when I want to make an extra, off-site back up critical files including Partner's photo shoots.

Overfilling

Last week, I worked on one of the tasks I've been stalling on: archiving my dad's artwork. He had a lot of copies of nearly identical files and I didn't have the working storage on my laptop. I figured I had this huge (22 TB, though mostly full) cluster, I could use that.

Yeah… not the best of ideas.

I didn't realize I had made a mistake until everything started to fail because all of the nodes were at 98% or more full and the system couldn't replicate even the replication logs. I didn't even realize that until Partner said Plex was down.

Well, with replication down, I couldn't even use the weed shell to remove a file. When I did that, it just hung for hours.

$ weed-shell
> rm -rf in/dad-pictures

Nix Shell Scripts

Above, I use weed-shell. This is a custom script I generate with NixOS that is installed in any server that can talk to my SeaweedFS.

inputs:
let
  shellScript = (
    pkgs.writeShellScriptBin "weed-shell" ''
      weed shell -filer fs.local:8888 -master fs.local:9333 "$@"
    ''
  );
in
{
  environment.systemPackages = [ shellScript ];
}

This lets me handle common functions I use when maintaining things. In this case, I don't have to enter the common parameters needed to talk to my SeaweedFS cluster.

Cleaning Up

I tried a bunch of things, such as trying to force a more extreme of vacuuming (cleaning deleted files):

> volume.vacuum --help
Usage of volume.vacuum:
  -collection string
    	vacuum this collection
  -garbageThreshold float
    	vacuum when garbage is more than this limit (default 0.3)
  -volumeId uint
    	the volume id
> volume.vacuum -garbageThreshold 0.1

This didn't help as much as I hoped, but it did allow some replication and some commands to go through. I needed to clear up a lot more space so I could remove files properly and do a wholesale rm -rf to blow away father's files and try again later once I get some more space.

Replication

I have my volumes set to 010 replication. These are three numbers as data center, rack, and host.

Data Center: Make X copies to replicate across multiple data centers. I don't have multiple data centers, so this is always 0 for me.
Rack: This is to replicate across multiple racks. My setup is that each computer is a “rack” so my 1 replication means make an extra copy on a different machine. I treat this as a rack because I also have a DeskPi Super6C which is 6 Raspberry Pi CM4 (compute modules) in a single case, so I treat all six as a “rack” but with separate hosts.
Host: The same machine. I don't have much use for two copies on the same machine, so I always set this to 0.

If I ever got a friend where I could set up a local server, I would consider setting up a second “data center” to have an off-site backup. That probably would require Tailscale but that's beyond my current scope.

Volumes

SeaweedFS basically creates multiple 30 GB files which act as a blob with multiple files inside it. That way, the problems with thousands of small files aren't an issue since everything is done on the 30 GB files called “volumes”.

Replication is done at the volume level, which means I was able to turn off replication for a series of volumes.

> lock
> volume.configure.replication -replication 000 -volumeId 1
> volume.configure.replication -replication 000 -volumeId 2
> volume.fix.replication
> volume.balance -force
> unlock

The lock and unlock are important when making changes like this, they prevent some critical operations from corrupting the cluster. The commands will tell you when it is needed.

volume.configure.replication basically changed those volumes to no replication (risky). Once that is done, volume.fix.replication and volume.balance -force deletes the excessive copies and shuffle things around, giving me some breathing room to get replication running again so I can mass delete files.

When I'm done, I just go and change all the nodes back to -replication 010 to give me the second backup.

Data Hoarding

The problem ultimate is data hoarding. Both my father and I both have multiple copies of files running around. It isn't great, but when you don't have time to clean out a copy of a laptop dying, it is sometimes easier to rsync the entire laptop into a directory of the new machine and then move on.

In this case, I needed to do some trimming of the duplicates from his files. The script is based on the one from a StackOverflow answer:

find . -not -empty -type f -printf "%s\n" \
    | sort -rn \
    | uniq -d \
    | xargs -I{} -n1 find . -type f -size {}c -print0 \
    | xargs -0 sha256sum \
    | sort \
    | uniq -w32 --all-repeated=separate

The output is pretty simple because it only lists duplicates and the paths to find them.

$ echo one > a.txt
$ echo one > b.txt
$ echo two > c.txt
$ echo two > d.txt
$ echo three > e.txt
$ find . -not -empty -type f -printf "%s\n" \
    | sort -rn \
    | uniq -d \
    | xargs -I{} -n1 find . -type f -size {}c -print0 \
    | xargs -0 sha256sum \
    | sort \
    | uniq -w32 --all-repeated=separate
27dd8ed44a83ff94d557f9fd0412ed5a8cbca69ea04922d88c01184a07300a5a  ./c.txt
27dd8ed44a83ff94d557f9fd0412ed5a8cbca69ea04922d88c01184a07300a5a  ./d.txt

2c8b08da5ce60398e1f19af0e5dccc744df274b826abe585eaba68c525434806  ./a.txt
2c8b08da5ce60398e1f19af0e5dccc744df274b826abe585eaba68c525434806  ./b.txt
$

With this, I can find the duplicates in my own system and delete them to clear out a few terabytes worth of data. It takes time, but I haven't done it before so I pointed it at /mnt/seaweed and let it run.

Once that is done, I can turn replication back on, fix replication, rebalance, and I should be good to go.

Forward Steps

I knew I was running out of storage for a while now, so I blew my monthly budget and got the fourth server ordered. This one has three 4 TB NVMe sticks (about 11 TB which will be added to the cluster) and should give me enough room to get my dad's files collected, deduplicated, and then look into uploading them to a cheap S3 storage (Backblaze) for later.

Bakfu and OCI Containers

2024-09-27T05:00:00Z

I'm giving myself a week to focus on something that I've been trying to do for a while, come up with a generic package management library. This of course brings me to the first thought: don't. Actually someone else said it better and with a lot more detail why this is a terrible idea. There are many other blog posts and warnings out there.

And yet I'm still going forward with writing a package management library: Bakfu.

Purpose

The main reason is that I see a need to have package management in a number of projects that are on my plate or I would like to be on my plate. While I could write a dedicated one for each one, I'm loathe to spend that much effort when I can spent far more effort coming up with something generic to separate the various concerns. Ultimate, this is the XKCD's General Problem.

The projects I have in mind for needing a plugin infrastructure are:

MfGames.Culture: My library for handling arbitrary calendars
Author Intrusion: My CLI-based tools for working with writing
Some vague game ideas in my head for mod support

To be honest, Author Intrusion has stalled out years ago, but I keep wanting to write it. The culture library, on the other hand, is on my mind because it is the largest identifiable “rock” causing my writer's block. I haven't been able to edit Second-Hand Dresses because I got stuck on the calendar of events. It's a minor thing, but I really liked it when Fedran had the master timeline and per-character timelines also.

MfGames Culture

My current thoughts for the culture library is to implement the calendars as WASM plugins that let me take the calendar components (e.g., Gregorian) and merge it with the cultural elements (English, French, etc) and allow the conversion between them. If I can, I'd implement each of my calendars as a separate plugin and let me evolve them. (There are a number of more, plus I use the Japanese “small seasons” in my world and a rough time of day also.)

Many of my chapters have a date associated with them. This was to help me identify the dates when certain events happened, but also to make sure a character has enough time to get from one place to the other (given that I don't have a map of the world yet).

---
when:
  start: 1471/3/28 MTR 4::22
  duration: 0::30
---

My original use was to parse the date (1471/3/28 MTR 4::22) into a common format (Julian Days or JD), then use that value to calculate the date in all the calendars I use. This worked out nicely since I could create a tooltip that showed the date on the website.

The original culture library was written in Typescript, which worked fine when I was generating my website with CobblestoneJS, but now that I'm using MfGames.Nitride, which is in C#, that is less useful, hence the rewrite since I dislike opening a pipeline to a CLI to generate websites (so slow…).

But the main part is that I don't want to package all the calendars and cultures in a single file. I want to be able to make one just for Fedran, one that covers US information, or allow someone to pick the minimum set they need for a given project. That leans toward a plugin infrastructure where I can have a file like package.json that lists the calendars needed for that project.

Reevaluation

The eighteen months I've been working on the ideas, mostly trying out a few things and failing a lot. I started a Rust implementation, switch to C# to prototype faster (as a more comfortable language for me). But I want to finish this and move on, so that is why I'm spending this time to hopefully get a working package manager library.

I hadn't reviewed my goals in a while and I was suffering with scope creep.

I realized that this morning when I read the article Distributing WebAssembly components using OCI registries. In specific, I was reading it hoping that the OCI Containers would handle the problem and I could abandon my project entirely (the best goal, to be honest). To be honest, WASM OCI Containers are perfect for my plans for MfGames.Culture, but there is something missing.

That is when I realized I was trying to figure out a container format and also figure out an abstracted dependency system.

Containers

The container format isn't really the important part even though I thought itw as. There are already good formats out there: OCI Containers, zip archives, NuGet packages.

With OCI Containers (specifically Docker images), a developer would curate a set of containers and compose something, but they don't worry too much about updating them or having them move around. Yeah, they could update their tag to :latest, but that is fraught with troubles if my experience updating with the Docker version of Nextcloud was any indication. Too many times having to walk through the updates, one major version at a time with occasionally manually updating.

The composition is also part of the problem. Composing services is awesome for packages, but I would consider a rather technical task. As opposed to someone going into a list of game mods, checking a bunch, and then expecting it to run. Getting that list of mods (which could be OCI containers) isn't provided in a generic manner, including the base libraries (such as UI injection library or a parsing framework) as needed, and then making it work isn't in the scope of OCI Containers. As far as I can tell, it isn't in the scope of anything at a generic level. Everyone seems to reinvent the wheel, or retrofits an existing wheel like JetBrains did with their module system (from my understanding).

Abstraction

Abstraction is something important to me. Libraries evolve, not just from refactoring code or being built on a different pattern, but how they are managed. Maintainers come and go. Sites come and go, though the development world would be in chaos if GitHub ever went away.

But, at one point, SourceForge was thought to be the same. And now, it is barely there as nothing more than a download repository with excessive advertising.

Also, the project owners change. Sometimes they just throw up their hands and delete everything. Sometimes, they just retire. A good example is me. A long time ago, I used to maintain csharp-mode, the C# mode for Emacs. After a while, I got burned out trying to implement generics and wandered off; someone else offered to take the reins and I happily gave it to them.

Another example is the friction that comes when one part of the organization is moving slower than the other. I think BouncyCastle is an example of that. There are three libraries that do roughly the same thing, but each one is different based on their platform (.NET Framework verses .NET Core), package name (BouncyCastle verses Portable.BouncyCastle), and evolution (BouncyCastle.Cryptography is the latest). But in this case, I can usually manually change all references from BouncyCastle or Portable.BouncyCastle to BouncyCastle.Cryptography and have it work.

In the above case, it would be a lot easier if I could say all references to BouncyCastle or Portable.BouncyCastle could be addressed to BouncyCastle.Cryptography and not have to manually edit older packages that don't know about the new library.

There is also remixes and forks. In Minetest, there are numerous “beds” plugins that don't play with each other but are dependencies of others. In many cases, I just have to pick one of them.

It isn't only just the package names that need to be replaced. Many modern languages allow you to pull in a package from a Git repository (if I'm lucky, it doesn't have to be GitHub). NPM allows it, Rust does, I'm pretty sure Go also does. This is great because a developer can pin a specific hash to the version.

But that is also fragile. What happens if the developer deletes the repository? Or changes their name. Suddendly dmoonfire/some-project doesn't work but the person using that library isn't maintaining their project either, so I have to figure out how to get a second- or three-degree reference to point to somewhere new like my local copy. This is basically the same problem as the BouncyCastle example above.

There is no abstraction, which is what I think is important. I want something like this:

PackageA depends on PackageB
PackageC provides/substitutes as PackageB

That way, if “PackageB” in the example goes away, then “PackageC” can takes it place without having to rebuild everything. In a more concrete example:

MyProject depends on BouncyCastle
Portable.BouncyCastle substitutes BouncyCastle
BouncyCastle.Cryptography substitutes as BouncyCastle

For locations, the same thing, I want to be able to say that a specific package can be found “here” or “here” or “here”.

Annotation

Security vulnerabilities are a still a big thing for me. I noticed NuGet is starting to download a list of vulnerabilities and report them in Visual Studio.

I want to expand that concept into something more generic, like Bluesky has with their distributed moderation. This is also something I'd love to see with Fediverse servers.

Being able to subscribe to a service that validates libraries and looks for hacks or does code auditing (and gets paid for it) is something I think this world needs. I could also see it as a local project that prevents certain packages from being viewed (like our problem being unable to move past netstandard2.0 but a minor version jumped to netstandard2.1). Or a curated list of assemblies for a project that is checked at the project root.

I use EasyQuery as an example. I love the library and what it does for my day job, but they use the minor version for breaking changes but there is nothing in NPM that tells me that. So I have a wiki page to developers to tell them, but an annotation service could say “breaking changes on minor versions” (they aren't the only one who doesn't honor sematic versioning).

Or also “here is a local cache of the package.”

Final Thoughts

This is just a rehash of what I've said previously, but for my needs. I need avoid scope creep and focus on what I need if I have any hope of finishing.

Obviously, finding that someone else has solved these problems would be awesome. Then I could abandon this project. Until then, I'm going to plug at it and see if I can get something minimal that works for getting me my calendars back.

Generating and Using Identifiers (Part 2)

2024-06-29T05:00:00Z

Last month, I did a post about snowflake and UUIDv7 identifiers. I was pretty happy with it, but then I was playing around with Clew which is a recent, smaller web search engine and decided to look up my identifers post just to check it out.

Right underneath mine was a post by Unkey on the same topic. It has some good observations that I didn't think about when I wrote mine, so decided to expand on mine to include the ideas I like.

Series

2024-05-30 Generating and Using Identifiers - A little discussion on unique identifiers, including formatting and presentation. A small database lesson on page splitting.
2024-06-29 Generating and Using Identifiers (Part 2) - A few expanded ideas about identifiers after discussing it with others.
2026-01-04 Other Smarter People - I found another specification for representing identifiers that is better supported, has more attention, and covers most of the basics I need to.

“Copying UUIDs is annoying”

Yes, but I didn't think about it. Both UUIDs (including my current crush on v7) and my preference for the Crockford encoding both have the same problem with double-clicking to copy, because of the dashes.

01-hz61-3s22-k8nr-6w6g-rv1y-5c52

I still feel that removing the separators turns it into a long series of numbers that is difficult to parse and elide in a consistent manner, so just removing the separators isn't an option. However, underscores don't have the same behavior (and the Unkey uses them later for the prefix discussion).

01_hz61_3s22_k8nr_6w6g_rv1y_5c52

I don't normally think about underscores since they conflict with underlines for links, but in this case, the identifier should be treated as a whole, so favoring the double-click and single word behavior is a major benefit.

Groups of Four

This isn't from the Unkey post, but with underscores, I think the groups of four is excessive. I don't know why, but the lower bar somehow makes it more obvious. Plus, how often do you need to hand transcribe or read out loud the numbers.

I think splitting the identifiers into groups of eight would be sufficient, as long as they are consistently eight instead of UUIDs default formatting of 8-4-4-4-12 which is difficult for me.

01_hz613s22_k8nr6w6g_rv1y5c52

I still think groups of eight (from the right) is still workable, has a natural break point for eliding (“rv1y5c52” in the above example) and doesn't need quite as much space.

“Prefixing”

I can't believe I forgot about prefixes. When Github and Gitlab both started adding prefixes to their API keys, I thought it was a great idea. Then, when I went to Gitea and later to Foregjo, seeing just a bunch of hex characters for an API key felt subtly “wrong” to me.

The Unkey post pointed out, purpose. A bunch of numbers is one thing, but knowing the purpose of the key helps identify it as a secret (e.g., something not to check into code) and also possibly give a way of preventing usage.

Naturally, since we want to treat an identifier, it should be separated by an underscore from the rest of the key. Also it solves a problem when the first character of the number is a number, it is an invalid code identifier. Not that someone is going to say:

const 01_hz613s22_k8nr6w6g_rv1y5c52 = getIdentifier();

However, it fits better into the “idea” of code to treat it as one. So, starting the prefix with a non-number would make this more useful:

player_01_hz613s22_k8nr6w6g_rv1y5c52

Separation

While thinking about it, I considered making a distinction between the identifier and the code with a double underscore.

bedor_player__01_hz613s22_k8nr6w6g_rv1y5c52

Originally, I didn't think it made sense because we don't really want to parse the identifier to pull out the prefix from the code itself. That said, there is one situation where we need to be able to parse: eliding.

rv1y5c52

Just breaking off the last group would result in stripping off the contextual prefix. We want to keep that information even when shortened, which means we have to have a mechanical way of identifying the prefix to determine how to break it apart.

bedor_player__01_hz613s22_k8nr6w6g_rv1y5c52
bedor_player__rv1y5c52

The double underscore also means that it remains a single word for purposes of selection.

Global Prefixes

Having multiple words does make sense to me, so “bedor_player” seems reasonable to have since it can be arbitrarily long. That said, the identifiers already produce a “globally unique” value, so the prefix also doesn't have to be global. That means we just need enough scope information for the producer of the identifier, but not a full scope like:

com_mfgames_bedor_player__01_hz613s22_k8nr6w6g_rv1y5c52

Good examples would be:

forgejo__01_hz613s22_k8nr6w6g_rv1y5c52
f__01_hz613s22_k8nr6w6g_rv1y5c52
api_prod__01_hz613s22_k8nr6w6g_rv1y5c52
ap__01_hz613s22_k8nr6w6g_rv1y5c52

In this case, shorter is better.

Suffixes

That led into the question about suffixes. For example:

api_prod__01_hz613s22_k8nr6w6g_rv1y5c52
api__01_hz613s22_k8nr6w6g_rv1y5c52__prod

The main reason I don't think suffixes make sense is that I believe most people look at the beginning and the end of variables in most cases. And, having a textual prefix and suffix adds more complexity since both have to be looked at to understand the scope. Having only a prefix means the contextual information is always on the same side of the code.

Eliding

Eliding (abbreviating) is important when dealing with large identifiers. Git allows you to use an arbitrary number for reducing the hashes to identify a Git, but I remember trying to figure out if six or eight was the best for some purpose. With the grouping above, there is a natural break at eight and sixteen.

I realized this really is only a concern if there are smaller numbers were most of the identifier is zero such as 00000000-0000-0000-0000-000000000000 which has a Crockford encoding of 0.

api__0

However, for most UUIDv7 identifiers, this isn't possible because the first 48-bits are a timestamp and won't ever be zero (again) after that epoch millisecond.

But, there is a known problem with eliding: there is a higher level of collisions:

01_hz613s22_k8nr6w6g_rv1y5c52 is “rv1y5c52”
01_k8nr6w6g_hz613s22_rv1y5c52 is “rv1y5c52”

In these cases, there needs to be a check for duplicates as part of the code. That is the cost of being able to elide, additional complexity when selecting.

I will have to mention, eliding is a “human” thing, not a protocol thing. We wouldn't be sending up an elided identifier as part of a HTTP header or an authorization key. We need eliding for showing a table of all known identifiers so someone can see additional information or delete the key. That said, in a grid, you probably don't need the prefix, but the double underscores makes it possible to have it just in case you mix your keys in the same grid.

“Specification”

There isn't a formal specification, but a current summary of my wandering thoughts.

Generate a 128-bit UUIDv7
Encode the 128-bit number into lowercase Crockford32
Put underscores every eight characters starting at the right
If eliding, remove the left-most groups until the desired length
Add an optional prefix that does not start with a number with a “__” separator

Conclusion

I'm still happy having a framework for this, though I could see extending some of my favorite C# strong typing library: StronglyTypedId to support the format and prefixes. I mean, it would be nice to be able to say:

using System;
using StronglyTypedIds;

[StronglyTypedId(Template.Crockford, Prefix="a")]
public partial struct ApiId { }

var id = new ApiId(Guid7.NewGuid());

Console.WriteLine(id);
// a__01_hz613s22_k8nr6w6g_rv1y5c52

Console.WriteLine(id.Elide(1));
// a__rv1y5c52

Hopefully, I've worked out most of the kinks I've found. I might come back, or I might formalize this into libraries. Either way, I think it is a workable pattern for the difficulties I've experience in the past.

Generating and Using Identifiers

2024-05-30T05:00:00Z

Over the last few months, I've had a number of conversations about unique identifiers. For starters, I get pulled into a periodic meeting to justify decisions we made about six years ago for my day job. Then there was a conversation on fedi (not a link to the conversation, it was too long ago and I auto-delete my posts).

Series

2024-05-30 Generating and Using Identifiers - A little discussion on unique identifiers, including formatting and presentation. A small database lesson on page splitting.
2024-06-29 Generating and Using Identifiers (Part 2) - A few expanded ideas about identifiers after discussing it with others.
2026-01-04 Other Smarter People - I found another specification for representing identifiers that is better supported, has more attention, and covers most of the basics I need to.

Introduction

So, identifiers. Almost everything in the database needs a way of unique identifying a record. Composite keys are nice and precise but a PITA to manage, which suggests a single field is much better.

For a long time, an int (32-bit) was “good enough” because it easily handled ~2e9 (2,147,483,647) records (most database I've worked with don't give you unsigned integers and negatives are horrible to work with). Slap your favorite database's way of creating a unique number with every record (sometimes called IDENTITY) and then hope everything works until you leave the project.

That doesn't always happen.

At many of my jobs, I've had to deal with millions (and occasionally billions) of rows of data. My current one, which is a multi-tenant database, we have customers with 10-20 million records each for the primary tables, easily a 3x or 10x that many on the secondary tables, and adding 100k records every day. That makes hitting the 2e9 limit uncomfortably probable since we're expected to be functional for at least three decades (imagine every gift card that is created and how each one has to be tracked for 3-5 years before it is filed with the state).

To handle that, we took the effort to bump up the key fields from int to bigint (64-bit, long for C# folks). This gave us plenty of breathing room since now we can handle more records than we will ever have inserted into the database.

Horizontal Scaling

Automatically generating identifiers works great with a single database, but what happens when you have two, three, or even a hundred? Some multi-tenant applications get around this by having one database per customer, but that increases the maintenance load to manage a thousand or more databases. That increases update times and requires our DBA to watch everything for failures.

When the identifiers have to be shared, they need to be unique otherwise they aren't very good identifiers. In our case, we have a number of servers that are all capable of inserting rows and we don't want to funnel every row creation through a single database server to make sure they are unique (this called “single point of failure”). Instead, we want to make sure each one can create a unique identifier with some authority and be confident that another server isn't going to steal the same identifier.

I'm glossing over a lot of the specifics because this blog is focusing on a specific methodology of creating unique identifiers.

There are many techniques for having database servers create non-overlapping identifiers. We have sequences, identities in certain ranges, and lookup tables. The problem I find with most of these is that they require a certain amount of manual configuration and tweaking. Adding a new server causes everything to rebalance the possible generated IDs, or smaller “blocks” of identifiers are assigned and there is a risk of starving out one active server because it generates identifiers at eighteen times the rate of the others (this happened to me, the east coast server was answering 80% of the the load).

Ideally, we want a setup that lets us toss together a new server, start it up, and then have it insert itself into the process smoothly without disrupting the others.

Conventions

For the rest of this post, I'm going to use the following conventions:

r means a random number, ideally cryptographically strong random
t for bits related to a timestamp
d for device or worker
s serial number, starting at 0 and incrementing by one
n the negative bit
4 the number 4

Enter UUIDs v4 (related GUIDs)

UUIDs are awesome and horrible in the same way. On one hand, they are fantastic for creating non-overlapping identity keys that are highly unlikely to have a collision. They are also 128-bit numbers which gives them a “huge” amount of space to work with.

The most common UUID I've deal with is the v4 which is entirely random except for a “4” jammed into the beginning of the third block.

In hex:

rrrrrrrr-rrrr-4rrr-rrrr-rrrrrrrrrrrr

Look at all those randoms. For a v4, there are 122 bits of random which means 5.3e36 possible numbers which makes it difficult to get a collision. I have never seen one in the wild, but that doesn't mean it is impossible. Just highly unlikely.

A GUID is the same as UUID, but with slightly different byte order.

Up until this point, most of the code I've dealt with has been creating v4 UUIDs. Microsoft recommended them, there is almost no chance to have a conflict, and they ensure a nice random coverage of identifiers.

...

Database hate them. The problem is that they are “random” which means when the database builds up an index of the rows, it wants to put everything in a nice neat order so it is easier to find things. When the database can find something quickly, then the user doesn't complain that it takes two seconds to open up a page.

The reason is that most relational databases (Postgresql, MariaDB, SQL Server) arranges data into “pages”, which are blocks of data. There are data pages (where the row data is located) and index pages (where the index is located). When the DB server inserts a row, it will arrange the data pages in primary key order (you can change with a “clustered” index).

Let's use this as an example, assuming that we can get four rows on a single page.

Page	ProjectId	ProjectTitle
1	0d5a76af-8a9f-460b-8e00-bcb23d8e4d57	Flight of the Scions
1	148dc206-d9bd-464a-a5b9-8ba49856fd46	Sand and Bone
1	4aaab767-4571-42f3-9b2f-4d1e69c73e23	Sand and Blood
1	69b46634-46dc-452b-a4d8-468403310009	Sand and Ash
2	a2fbaa43-52da-484c-9dc2-3352b4b0aa8f	Second-Hand Dresses
2	ce0db656-97a5-4b06-85e1-173074b96950	Allegro
2	e8af805b-bb8c-4adb-98a6-7fa9c4a86bb8	Nor Curse Be Found
2	f991f021-8e83-4f23-8c75-ed4ba0953915	Pack Daughter

If we insert a new row, such as “5e187e66-25bb-4e5b-ae4c-07e986cb92fa” for Son of Vo, we have to insert it between Sand and Blood and Sand and Ash, but since we already have two filled pages, we have to split the first page by creating a new page, inserting it between pages one and two, and then linking them together (basically turn the old page two into three, pages are a linked list). Then, then new row is added to one of them (in this case, the least full one).

Page	ProjectId	ProjectTitle
1	0d5a76af-8a9f-460b-8e00-bcb23d8e4d57	Flight of the Scions
1	148dc206-d9bd-464a-a5b9-8ba49856fd46	Sand and Bone
1	4aaab767-4571-42f3-9b2f-4d1e69c73e23	Sand and Blood
2	5e187e66-25bb-4e5b-ae4c-07e986cb92fa	Son of Vo
2	69b46634-46dc-452b-a4d8-468403310009	Sand and Ash
3	a2fbaa43-52da-484c-9dc2-3352b4b0aa8f	Second-Hand Dresses
3	ce0db656-97a5-4b06-85e1-173074b96950	Allegro
3	e8af805b-bb8c-4adb-98a6-7fa9c4a86bb8	Nor Curse Be Found
3	f991f021-8e83-4f23-8c75-ed4ba0953915	Pack Daughter

Now imagine doing this a million times. Millions of jamming rows into gaps and splitting when there isn't room. By the end, you'll have all these empty spaces because re-indexing (rebuilding the data pages so they are nice and packed) is an expensive operation so it isn't done automatically. Deleting rows will also create spaces but if there is a space while creating a row, then the server will use it and fill it back up.

All of these empty spaces are called fragmentation. When it gets back, the database server spends a lot of time jumping over the empty spaces. Every time there is a page split, the server has to stop and do the split. Rebuilding the index requires it to stop longer to pack everything in.

When setting up the table, one could set the “fill factor” which is how much space to reserve but insert enough GUIDs into the table and the page splitting starts to happen.

Also, UUIDs are 128-bits which means they aren't a “native” data type for most database servers and servers (64-bit). This means they are usually stored as an array of bytes or strings.

Snowflakes

This where I was about seven or eight years ago. We knew that we needed to move toward horizontal scaling, UUIDs were not an option if we wanted performance, and we needed something that wouldn't require a lot of work.

Enter snowflake IDs. Originally from Twitter, the 64-bit snowflake has four parts:

1 bit to always make it positive
41 bits of timestamp
10 bits device identifier
12 bits of serial

In a bit layout because for some reason, they didn't make these numbers fit on a hex boundary:

ntttttttttttttttttttttttttttttttttttttttttddddddddddssssssssssss

The biggest part was that the timestamp part was in the front. Since time was always increasing, this meant that the generated identifiers were “mostly” increasing. This meant that there was significantly less random inserts in the middle of the index and everything was appended at the end (or nearly at the end depending on frequency).

It also had the device ID. This meant that a server could “reserve” a number (0-1023) and there would be no chance of a collision. Of course, that also meant that the devices (which could be a server or process) had to coordinate the device assignment.

The last part was a serial number that started with 0 and went up to 4096. Since the timestamp was frequently at millisecond scale, that meant we could create 4096 rows per millisecond with no chance of collision.

For my day job, this is what we went with since it was the smallest data type that fit our expected key spaces (billions but not quadrillions), and we were able to coordinate the device identifiers to ensure no chance of collisions.

Also, snowflakes are 64-bit which means most languages can handle them natively (not Javascript though, which chokes on them). The need to coordinate the device, however, makes it not a great option for truly distributed services (such as Fediverse servers or Matrix rooms).

There are also 128-bit versions of snowflake IDs in some libraries.

UUID v7

Recently, I learned about UUIDv7. Like the v4, this is a long list of hash with a “7” in the third group. The primary difference is that the first part is a millisecond-based timestamp instead of random number but the latter half is still random.

In hex:

tttttttt-tttt-7rrr-rrrr-rrrrrrrrrrrr

This solves the page split problem nicely by taking that leading 48-bits of timestamp so the value are always increasing and uses a smaller random space (74 bits or only 1.89e22 possible numbers). This means there is a greater chance of having a collision but only if the numbers were generated in the same millisecond (that's the big part).

UUIDv7 also doesn't need the coordination of snowflakes, which lowers the complexity of generating a new identifier. At the moment, languages like C# doesn't have a built-in version for v7 and databases still struggle with the larger data type (compared to a 64-bit number which is native to most processors).

For most of what I do, those negatives are relatively trivial with v7, so any new project would be based on UUIDv7 and I'd feel pretty good about it.

ULIDs

Notable, there are also ULID (specification) which also try to solve the problem. These are a lot like UUIDv7 except:

They don't have the version in the third group
They have the same 48 bits of timestamp
They have 80 bits of random instead of 74 for an increase of randomness (1.21e24 instead of 1.89e22)

They are also formatted with capital Crockford Base32 encoding instead of hex numbers. This creates a considerably more compact format such as “01HZ613S22K8NR6W6GRV1Y5C52” instead of “022f4166-c274-43b1-a69d-a9c3c2153ca2” (26 characters verses 36).

While the ULID page says it is compatible with UUIDs, they aren't because of that missing version field. There are systems that validate those entries so I find that claim to be not valid when dealing with very strict people (like my DBA and customers). Still, they are “close” to UUID and fit in much the same space.

Formatting

In all of these cases, there is one other issue to address: formatting.

UUIDs and GUIDs are terrible in terms of how much space they use. Hex is great as a case-insensitive format, but the grouping of 8-4-4-4-12 makes it difficult to read over a phone or to find differences in the numbers. I struggle with more than 3-4 numbers at a time, so trying to enter in the twelve at the end can be a trial while being distracted.

The Crockford encoding of ULIDs is much better because it uses a wider range of characters than hex which allows the actual identifier to be fewer characters. It also has some ability to handle difficult letters because there is no “I” or “L” in the encoding which means they can be treated as a “1” and still have the same results.

But, for ULIDs, the resulting ID should be in all capitals and I consider this to be a problem because I have a degree of dyslexia and it is easy to get lost in a 26-character series of letters and numbers all at the same height. It's basically a “bar” of numbers.

In the end, what I found that was easier with customers and end users was to make the identifier lowercase to help distinguish the numbers and then break it into groups of four with dashes (favoring the right side). Using the above example of “01HZ613S22K8NR6W6GRV1Y5C52”, that would result in “01-hz61-3s22-k8nr-6w6g-rv1y-5c52” which is slightly longer (32 characters) but makes it less overwhelming to read it over a phone or type it into a computer balanced on a box with a keyboard.

These values can also be elided nicely much like you only have to enter the first 6-8 characters of a Git hash to reference it, there is a “reasonable” chance of avoiding overlaps if you only go with the last two blocks. I like to put “rv1y-5c52” in a web or GUI column but then put the full value in a copy button or as a tooltip.

Conclusion

At this point, any new project I create would be based on UUIDv7. While they have a tiny smaller random space than ULID and UUIDv4, they play well with databases and have good support in most languages (C# and others have native UUID/GUID support, though maybe not v7 generation).

The readability is an important aspect of mine, so I see myself formalizing on the lowercase with dashes with eliding to smaller numbers when the detail isn't needed.

Overall, I like exploring these situations. I'll admit, I wish I knew about ULIDs or UUIDv7 eight years ago, but it is what it is.

Reflections on Nitride

2024-04-21T05:00:00Z

This week's goal was to get mfgames.com migrated over to MfGames.Nitride. It was a lofty goal, mainly because I've been making small incremental changes over the last year but never got something pushed over the finish line.

I succeeded.

Migration

This is my third “big” site for Nitride, one that follows the common pattern of pulling in multiple Git repositories from my forge, integrating the data, and then creating a website out of it. In other words, one of the main reasons I created Nitride.

It isn't exactly a fancy site, but I find that I'm not really creating flashy sites anymore. Part of this is because I still enjoy Gemini, which is a text-only format, but also because the chrome is distracting to me. I want something simple but not the black on gray of Netscape Navigator.

Overall, I'm pretty happy with the results. It creates landing pages for projects, pulls in the ./docs/ folder for each project and formats them (for example, MfGames.Gallium), and merges news posts from the projects (while decorating them) and creates a single unified feed.

There are more things I want to do, such as links to issue tracking, licenses, and other stuff. I also ended up losing some features like the culture sandbox (which I needed to rewrite anyways), the slippery maps, and a few other minor things.

Still happy with it.

News

I was working on debugging creating news posts in individual project repositories anews posts from individual projects into mfgames.com as my “little thing” for today and I came up with the exact period of time I decided to write my latest static site generator (SSG):

$ git log --pretty="%ai %s"
...
2021-05-04 02:27:28 -0500 feat: implementing my own
2021-04-07 22:13:20 -0500 chore: updating Statiq libraries
...
2014-05-07 14:41:27 -0700 Initial commit

It all started a year before, when I migrated from CobblestoneJS (my SSG from the early 2010s) to GatsbyJS. Which was pretty and gorgeous and so difficult to debug. It was also fragile for me because I don't update my libraries to follow the curve and every time I tried to add a new feature to the site, I spent a few days just trying to figure out the node packages to get everything building again first.

When I got into Gemini for the first time, I also realized that GatsbyJS wasn't going to work there because that shiny site was not text friendly. It was also tied into my frustrating that going to a page the first time downloaded the content in one manner, but navigation was done via SPA and pulled it from another; and the cache was killing me there.

(But it looked so pretty.)

So I decided to look for a new SSG. I didn't want to write my own because… I really don't like writing utility programs. But I wanted to do something Gatsby didn't do, which is generate a site that didn't require Javascript.

I figured I was good at C# (this was before Rust, but I'm still best at C#), so I wanted to find a C# static site generator in hopes that the NuGet package management wasn't as excruciating as Node's.

So I found Statiq which seemed like a perfect match. But going from GatsbyJS to Statiq quickly hit a point where I realized I was, once again, forcing a program to do things it wasn't supposed to be doing. The patterns in Statiq were nice but they were too rigid for the complexity of my way of creating sites:

The fragility of pulling in 80+ Git repositories into a CI pipeline
The use of multiple dynamic lists on a single page
The desire to make a Gemini-friendly site

After that month of thinking, I started to write my own. This came when I was looking at Statiq's document type realizing that my desire to create a new type basically meant I had to rebuild the entire framework to support a slightly different enumeration.

Gallium

Enter Entity Component Systems (ECS) which I learned about while pondering writing a game using Bevy. I had never heard of an ECS before that, but the idea was great. Why not write a static site generator that didn't need a single enum to drive processing, where I could add arbitrary components to pages such as nested lists or metadata, and then easily be able to use them.

Of course, then I had to look at existing ECS systems in C#, but I eventually decided to be more idiomatic to C# (another frustration of Statiq) and focus on something that let me use LINQ-like method calls (such as .Select() and .Where()).

List<Entity> newsList = fileList
    .WhereEntity<UPath>(this.OnlyNewsFiles)
    .Run(this.setInstantFromPath, cancellationToken)
    .SelectEntity<UPath, Instant>(this.pathService.SetPathFromInstant)
    .Select(entity => entity.Set(IsPost.Instance))
    .Select(entity => entity.Set(IncludeInFeed.Instance))
    .Where(entity => entity.Has<IsMarkdown>())
    .OrderBy(entity => entity.Get<PageModel>().Title)
    .ToList();

I'm specifically not saying Gallium is a “high performance” because it isn't. It is an ECS for relatively small systems that builds on existing libraries for patterns. I don't need high performance for sites with only a few thousand pages, I want something easy to understand.

Next Steps

There are quite a few steps going forward. I still want to make Nitride easier to use, Gallium to track LINQ patterns, and mfgames.com to present more data. But this is a big step and I'm really happy that I'm proud of (I'm sure there will be tweaks). More importantly, I have something that hasn't given me any major trouble updating since I started three years ago.

Switching Ceph to SeaweedFS on NixOS

2024-03-21T05:00:00Z

At the end of 2023, I realized that I was running out of space on my home Ceph cluster and it was time to add another node to it. While I had space for one more 3.5" drive in one of my servers, I was feeling a little adventurous and decided to get a DeskPi Super6C, a Raspberry Pi CM4, a large NVMe drive, and try to create a new node that way.

Well, over the following few months, a lot of mistakes were made that are worthy of a dedicated post. But, when most of those problems were resolve, I encountered another series of “adventures” which led me to switching out my home's Ceph cluster for a SeaweedFS one.

Unable to Build Ceph

Around the time I was working on the Pi setup, my NixOS flake was unable to build the ceph packages. Part of this is because I was working off unstable and so the few weeks of being unable to build meant I couldn't get Ceph working on the new hardware. I tried even compiling it myself, which takes about six hours on my laptop and longe because I had to remote build on the Pi itself since I have yet to figure out how to get Colmena to build aarch64 on my laptop.

Also, I was dreading setting up Ceph since I remember how many manual steps I had to do to get the OSDs working on my machines. While researching it, I was surprised to see my blog post on it was on the wiki page, which is kind of cool and a nice egoboo.

There was a PR on Github for using the Ceph-provided OSD setup that would have hopefully alleviated it. That looked promising, so I was watching that PR with interest because I was right at the point of needing it.

Sadly, that PR ended up being abandoned for a “better” approach. Given that it takes me six hours to build Ceph, I couldn't really help with that approach which meant I was stuck waiting unless I was willing to dedicate a month or so trying to figure it all out. Given that the last time I tried to do that, my PR was abandoned for a different reason, I was preparing to keeping my Ceph pinned until the next release and just having my Raspberry Pi setup sit there idle.

I was also being impatient and there was something new to try out.

SeaweedFS

Then I noticed a little thing on top of the NixOS wiki for Ceph:

Another distributed filesystem alternative you may evaluate is SeaweedFS.

I vaguely remember looking at it when I first set up my 22 TB Ceph cluster, but I'd been dreaming about having a Ceph cluster for so long that it was dismissed because I really wanted to do Ceph.

Now, the need was less there so I thought I would give it a try. If anything, I still had a running Ceph cluster and I could run them side-by-side.

A big difference I noticed is that SeaweedFS has a single executable that provides everything. You can run it as a all-in-one process, but the three big services can also be run independently. That includes the master (coordinates everything), the volumes (where things are stored), and filer (make it look like a filesystem).

Also, Ceph likes to work at the block level whereas SeaweedFS wants to be pointed to plain directories. So the plan was to take the 1 TB drive for my Raspberry Pi and turn it into a little cluster to try it out.

SeaweedFS and NixOS

The first thing was that SeaweedFS doesn't have any NixOS options. I couldn't find any flakes for it either. My attempt to create one took me three days with little success. Instead, I ended up cheating and just grabbed the best-looking one I could find and dump it directly into my flake. It isn't even an override.

Yeah I would love to have a flake for this but I'm not skilled enough to create it myself.

Masters

With that, a little fumbling got a master† server up and running. You only need one of these, so pick a stable server and set it up.

# src/nodes/node-0.nix
inputs @ { config
, pkgs
, flakes
, ...
}: {
  imports = [
    ../../services/seaweedfs.nix # the file from dermetfan
  ];

  services.seaweedfs.clusters.default = {
    package = pkgs.seaweedfs;

    masters.main = {
      openFirewall = true;
      ip = "fs.home"; # This is what shows up in the links
      mdir = "/var/lib/seaweedfs/master/main";
      volumePreallocate = true;

      defaultReplication = {
        dataCenter = 0;
        rack = 0;
        server = 0;
      };
    };
  };
}

This is a really basic setup that doesn't really do anything. The master server is pretty much a coordinator. But, what is nice is that that it shows something by starting up a web server at fs.home:9333 that lets you see that it is up (sadly, no dark mode) and running. This site will also let you get to all the other servers through web links.

Another important part is the defaultReplication. I made it explicit, but when messing around, setting all three to 0 means that you don't get hung up the first time you try to write a file and it tries to replicate to a second node that isn't set up. All zeros is basically “treat the cluster as a single large disk.”

Later on, you can change that easily. I ended up setting rack = 1; in the above example because I treat each node as a “rack” since I don't really have a server rack.

† I don't like using “master” and prefer main, but that is the terminology that SeaweedFS uses.

Volumes

Next up was configuring a volume server. I ended up doing one per server (I have four nodes in the cluster now) even though three of them had multiple partitions/directories on different physical drives. In all of these cases, I named the directory /mnt/fs-001 and created an ext4 partition on it. I could have used ZFS, but I know and trust ext4 and had trouble with ZFS years ago. But it doesn't matter, just make a drive.

# src/nodes/node-0.nix
inputs @ { config
, pkgs
, flakes
, ...
}: {
  imports = [
    ../../services/seaweedfs.nix # the file from dermetfan
  ];

  services.seaweedfs.clusters.default = {
    package = pkgs.seaweedfs;

    volumes.${config.networking.hostName} = {
      openFirewall = true;
      dataCenter = "home";
      rack = config.networking.hostName;
      ip = "${config.networking.hostName}.home";
      dir = [ "/mnt/fs-001" ];
      disk = [ "hdd" ]; # Replication gets screwy if these don't match
      max = [ 0 ];
      port = 9334;

      mserver = [
        {
          ip = "fs.home";
          port = 9333;
        }
      ];
    };
  };
}

Once started up, this starts a service on http://node-001.home:9333, connects to the master which will then show a link on that page, and basically say there is plenty of space.

The key parts I found are the disk and max.

Replication is based on dataCenter, rack, and server but also only if the disk types agree. So, hdd will only sync to other hdd even if half of them are ssd or nvme. Because I have a mix of NVMe and HDD, I made them all hdd because it works and I don't really care.

The value of 0 for max means use all the available space. Otherwise, it only grabs a small number of 30 GB blocks and stops. Since I was dedicating the entire drive over to the cluster, I wanted to use everything.

Filers

The final service needed is a filer. This is basically the POSIX layer that lets you mount the drive in Linux and start to do fun things with it. Like the others, it just gets put on the server. I only set up one filer and it seems to work, but others set up multiples. I just don't really understand why.

# src/nodes/node-0.nix
inputs @ { config
, pkgs
, flakes
, ...
}: {
  imports = [
    ../../services/seaweedfs.nix # the file from dermetfan
  ];

  services.seaweedfs.clusters.default = {
    package = pkgs.seaweedfs;

    filers.main = {
      openFirewall = true;
      dataCenter = "home";
      encryptVolumeData = false;
      ip = "fs.home";
      peers = [ ];

      master = [ # this is actually in cluster.masters that I import in the real file
        {
          ip = "fs.home";
          port = 9333;
        }
      ];
    };
  };
}

Like the others, this starts up a web service at fs.home:8888 that lets you browse the file system, upload files, and do fun things. Once this is all deployed (by your system of choice, mine is Colmena), then it should be up and running. Which means you should be able to upload a folder with the port 8888 site.

Debugging

I found the error messages are a little confusing a time, but weren't too much of a trouble to find. I just had to tail journalctl and then try to figure it out.

journalctl -f | grep seaweed

If you have multiple servers, debugging requires doing this to all of them.

Secondary Volumes

Adding more volumes is pretty easy. I just add a Nix expression to each node to include drives.

  services.seaweedfs.clusters.default = {
    package = pkgs.seaweedfs;

    volumes.main = {
      openFirewall = true;
      dataCenter = "main";
      rack = config.networking.hostName;
      mserver = cluster.masters; # I have this expanded out above
      ip = "${config.networking.hostName}.home";
      dir = [ "/mnt/fs-002" "/mnt/fs-007" ]; # These are two 6 TB red drives
      disk = [ "hdd" "hdd" ]; # Replication gets screwy if these don't match
      max = [ 0 0 ];
      port = 9334;
    };
  };

As soon as they deploy, they hook up automatically and increase the size of the cluster.

Mounting

Mounting… this gave me a lot of trouble. Nix does not play well with the auto-mount and SeaweedFS, so I had to jump through a few hoops. In the end, I created a mount.nix file that I include on any node that I have to mount the cluster on, which always goes into /mnt/cluster.

inputs @ { config
, pkgs
, ...
}:
let
  mountDir = "/mnt/cluster";

  # A script to go directly to the shell.
  shellScript = (pkgs.writeShellScriptBin
    "weed-shell"
    ''
      weed shell -filer fs.home:8888 -master fs.home:9333 "$@"
    '');

  # A script to list the volumes.
  volumeListScript = (pkgs.writeShellScriptBin
    "weed-volume-list"
    ''
      echo "volume.list" | weed-shell
    '');

  # A script to allow the file system to be mounted using Nix services.
  mountScript = (pkgs.writeShellScriptBin
    "mount.seaweedfs"
    ''
      if ${pkgs.gnugrep}/bin/grep -q ${mountDir} /proc/self/mountinfo
      then
        echo "already mounted, unmounting"
        exit 0
      fi

      echo "mounting weed: ${pkgs.seaweedfs}/bin/weed" "$@"
      ${pkgs.seaweedfs}/bin/weed "$@"
      status=$?

      for i in 1 1 2 3 4 8 16
      do
        echo "checking if mounted yet: $i"
        if ${pkgs.gnugrep}/bin/grep -q ${mountDir} /proc/self/mountinfo
        then
          echo "mounted"
          exit 0
        fi

        ${pkgs.coreutils-full}/bin/sleep $i
      done

      echo "gave up: status=$status"
      exit $status
    '');
in
{
  imports = [
    ../../seaweedfs.nix
  ];

  # The `weed fuse` returns too fast and systemd doesn't think it has succeeded
  # so we have a little delay put in here to give the file system a chance to
  # finish mounting and populate /proc/self/mountinfo before returning.
  environment.systemPackages = [
    pkgs.seaweedfs
    shellScript
    volumeListScript
    mountScript
  ];

  systemd.mounts = [
    {
      type = "seaweedfs";
      what = "fuse";
      where = "${mountDir}";
      mountConfig = {
        Options = "filer=fs.home:8888";
      };
    }
  ];
}

So, let me break this into the parts. SeaweedFS has a nice little interactive shell where you can query status, change replication, and do lots of little things. However, it requires a few parameters, so the first thing I do is create a shell script called weed-shell that provides those parameters so I don't have to type them.

$ weed-shell

The second thing while doing this is that I wanted to see a list of all the volumes. SeaweedFS creates 30 GB blobs for storage instead of thousands of little files. This makes things more efficient in a lot of way (replication is done on volume blocks).

$ weed-volume-list | head
.> Topology volumeSizeLimit:30000 MB hdd(volume:810/1046 active:808 free:236 remote:0)
  DataCenter main hdd(volume:810/1046 active:808 free:236 remote:0)
    Rack node-0 hdd(volume:276/371 active:275 free:95 remote:0)
      DataNode node-0.home:9334 hdd(volume:276/371 active:275 free:95 remote:0)
        Disk hdd(volume:276/371 active:275 free:95 remote:0)
          volume id:77618  size:31474091232  file_count:16345  replica_placement:10  version:3  modified_at_second:1708137673 
          volume id:77620  size:31501725624  file_count:16342  delete_count:4  deleted_byte_count:7990733  replica_placement:10  version:3  modified_at_second:1708268248 
          volume id:77591  size:31470805832  file_count:15095  replica_placement:10  version:3  modified_at_second:1708104961 
          volume id:77439  size:31489572176  file_count:15067  replica_placement:10  version:3  modified_at_second:1708027468 
          volume id:77480  size:31528095736  file_count:15118  delete_count:1  deleted_byte_count:1133  replica_placement:10  version:3  modified_at_second:1708093312

When doing things manually, that is all I needed to see things working and get the warm and fuzzy feeling that it worked.

Getting it to automatically mount (or even systemctl start mnt-cluser.mount) is that the command to do so is weed fuse /mnt/cluster -o "filer=fs.home:8888".

NixOS doesn't like that.

So my answer was to write a shell script that fakes a mount.seaweedfs and calls the right thing. Unfortunately, it rarely worked and it took me a few days to figure out why. While weed fuse returns right away, I'm guessing network latency means that /proc/self/mountinfo doesn't update for a few seconds later. But systemd had already queried the mountinfo file, saw that it wasn't mounted, and then declared the mount failed.

But, by the time I (as a slow human) looked at it, the mountinfo showed success.

The answer was to delay returning from mount.seaweedfs until we give SeaweedFS a chance to finish so systemd could see it was mounted and didn't fail the unit. Hence the loop, grep, and sleeping inside mount.seaweedfs. Figuring that out required a lot of reading code and puzzling through things to figure out, so hopefully that will help someone else.

After I did, though, it was working pretty smoothly since, including recovering on reboot.

Changing Replication

As I mentioned above, once I was able to migrate the Ceph cluster, I changed replication to rack = 1; to create one extra copy across all four nodes. However, SeaweedFS doesn't automatically rebalance like Ceph does. Instead, you have to go into the shell and run some commands.

$ weed-shell
lock
volume.deleteEmpty -quietFor=24h -force
volume.balance -force
volume.fix.replication
unlock
exit

You can also set it up to do it automatically, I'm not entirely sure I've done that so I'm not going to show my attempt.

Observations

One of the biggest thing I noticed is that Ceph does proactive maintenance on drives. It doesn't sound like much, but I feel more comfortable that Ceph would detect errors. It also means that the hard drives are always running in my basement; just the slow grind of physical hardware as Ceph scrubs and shuffles things around.

SeaweedFS is more passive in that regard. I don't trust that it won't catch a failed hard drive a fast, but it still doesn't have the failures of RAID, lets me spread out data across multiple servers and locations. There is also a feature for uploading to a S3 server if I wanted. I use a Restic service for my S3 uploads.

That passivity also means it hasn't been grinding my drives as much and I don't have to worry about the SSDs burning out too quickly.

Another minor thing is that while there are a lot less options with SeaweedFS, it took me about a third of the time to get the cluster up and running. There were a few error messages that threw me, but for the most part, I understood the errors and what SeaweedFS was looking for. Not always the case with Ceph and I had a few year-long warnings that I never figured out how to fix that I was content to leave as-is.

I do not like the lack of dark mode on SeaweedFS's websites.

Opinions

I continue to like Ceph, but I also like SeaweedFS. I would use either, depending on the expected load. If I was running Docker images or doing coding on the cluster, I would use a Ceph cluster. But, in my case, I'm using it for long-term storage, video files, assets, and photo shoots. Not to mention my dad's backups. So I don't need the interactive of Ceph along its higher level of maintenance.

Also, it is a relatively simple Go project, doesn't take six hours to build, and uses more concepts that I understand (mkfs.ext4) that I'm more comfortable with it.

It was also available at the point I wanted to play (though Ceph is building on NixOS unstable again, so that is moot problem. I was just being impatient and wanted to learn something new.)

At the moment, SeaweedFS works out nice for my use case and decided to switch my entire Ceph cluster over. I don't feel as safe with SeaweedFS, but I feel Safe Enough™.

NixOS and NextCloud

2024-03-03T06:00:00Z

I spent a few hours upgrading my NextCloud server from version 26 to 28. Sadly, it was not a pleasant process, mainly because there were some index changes that were conflicting and the migration code kept having a problem with creating indexes with one name for oc_table_name because it conflicted with table_name. This is probably because I have a relatively old NextCloud setup that has been migrated from OwnCloud over the years and didn't use table prefixes.

Unfortunately, that meant I ended up having to do a lot of:

DROP INDEX index_name ON table_name; in mysql
systemctl restart nextcloud-setup.service || systemctl status nextcloud-setup.service
Look at the error, go back to the beginning with the new name

That took about an hour to get through and pretty much points out that I also aren't using the proper indexes because half of those tables are empty and the other have isn't. However, table_name in the above example was sometimes the prefixes (oc_table_name) and other times it was the non-prefixed (table_name) version so that is just something I'm going to live with.

Once I finished, it complained about missing indexes, so I had to run this:

nextcloud-occ db:add-missing-indices

Which probably put all the indexes I dropped and the migrations put on the wrong table back to the table that is actually used.

Finally, there were still a few other errors about “interned_strings_buffer” and memcached not working. I figured some of these were related to me losing my database instance on my hosting provider, which meant I moved the database from a dedicated server to the same one as the NextCloud server.

Below the Nix file I use to set up NextCloud so it doesn't give me any warnings and has backups properly set up.

# nextcloud.nix
{ config
, pkgs
, ...
}:
let
  host = "nextcloud.example.com";
  backup-name = "restic";
in
{
  # Set up the user in case you need consistent UIDs and GIDs. And also to make
  # sure we can write out the secrets file with the proper permissions.
  users.groups.nextcloud = { };
  users.users.nextcloud = {
    isSystemUser = true;
    group = "nextcloud";
  };

  # Set up backing up the database automatically. The path will be in
  # `/var/backups/mysql/nextcloud.gz`.
  services.mysqlBackup.databases = [ "nextcloud" ];

  # Restic is already set up to back up the mysql directory but
  # we also set it up to backup the data.
  services.restic.backups.${backup-name}.paths = ["/var/lib/nextcloud/data"];

  # Set up secrets. This is a sops-nix file checked in at the same folder as
  # this file.
  sops.secrets = {
    nextcloud-admin-password = {
      sopsFile = ./secrets.yaml;
      mode = "0600";
      owner = "nextcloud";
      group = "nextcloud";
    };

    nextcloud-db-password = {
      sopsFile = ./secrets.yaml;
      mode = "0600";
      owner = "nextcloud";
      group = "nextcloud";
    };

    nextcloud-secrets = {
      sopsFile = ./secrets.yaml;
      mode = "0600";
      owner = "nextcloud";
      group = "nextcloud";
    };
  };

  # Set up Nextcloud.
  services.nextcloud = {
    enable = true;
    package = pkgs.nextcloud28;
    https = true;
    hostName = host;
    secretFile = "/var/run/secrets/nextcloud-secrets";

    phpOptions."opcache.interned_strings_buffer" = "13";

    config = {
      dbtype = "mysql";
      dbname = "nextcloud";
      dbhost = "localhost";
      dbpassFile = "/var/run/secrets/nextcloud-db-password";

      adminuser = "admin";
      adminpassFile = "/var/run/secrets/nextcloud-admin-password";
    };

    settings = {
      maintenance_window_start = 2; # 02:00
      default_phone_region = "en";
      filelocking.enabled = true;

      redis = {
        host = config.services.redis.servers.nextcloud.bind;
        port = config.services.redis.servers.nextcloud.port;
        dbindex = 0;
        timeout = 1.5;
      };
    };

    caching = {
      redis = true;
      memcached = true;
    };
  };

  # Set up Redis because the admin page was complaining about it.
  # https://discourse.nixos.org/t/nextlcoud-with-redis-distributed-cashing-and-file-locking/25321/3
  services.redis.servers.nextcloud = {
    enable = true;
    bind = "::1";
    port = 6379;
  };

  # Setup Nginx because we have multiple services on this server.
  services.nginx = {
    enable = true;
    virtualHosts."${host}" = {
      forceSSL = true;
      enableACME = true;
    };
  };

  networking.firewall.allowedTCPPorts = [ 80 443 ];
}

Hopefully, this will help others if they encounter the same problems as me since I couldn't find these answers with web searching.

New Site Colors

2024-02-24T06:00:00Z

I should have been working on something else today, but a series of events lead me down a path of reworking how I implemented my themes here on https://d.moonfire.us/ and https://fedran.com/.

I have not updated the color themes on Fedran at this point.

In specific, it was someone mentioning they didn't like my dark theme on their recently announced blogroll. Now, there is absolutely nothing wrong with someone not liking my sense of colors or even how I code websites, but a desire to change the colors has been sitting in the back of my head for a few months. That meant that the commentary reminded me of my own desires to change it and that lead into my fixating on redoing the colors.

Color Preferences

One of the problems is that I have a fondness for mono- and bichromatic color themes. You can see that on my covers, but I also realize that it creates rather plain (e.g., uninteresting) site theme. I started to really notice it once I started using Catppuccin as one of my most common UI themes. There were colors on the screen! Multiple of theme!

And I started wanting a few distinct of my own.

At the same time, I continue to struggle with contrasts. When I was younger (teenager and 20s), I loved drawing on quadraile graph paper. But somewhere in my 30s, I started having trouble because the lines were making it hard for me to see my writing. So I went on a few adventures to find lighter lines and evetually switched to dotted grid pages. Then in my 40s, even those became too dark and I had to switch to the same thing my father used: plain white paper.

POV Colors

Related to this is my decision that every point of view in Fedran had a different color. I like that idea, mainly because I love colors, I'm just not great with seeing differences. This is why Sand and Blood has a different color set than Allegro.

Each POV has a color hue and everything else is built off of that using LCH colors. When I switched the covers to be bichromatic, I made the second color always be the contrast (rotate 180 degrees) from the primary color.

Easy huh?

However, this entire excercise pointed out that most people can't tell the difference between say 220 and 225 hue, so I really should spread out the colors a lot more. Even if I say a 6 degree difference in hues, that would limit me to sixty POVs. (Of course, since I'm still somewhat burned out, sixty is a lot right now.)

So, I'm going to change it so each POV color is three colors: the primary color I had before and then two accent colors (which the first will be the contrast color for the existing assigned colors). I'm not quite ready for the full gaumt of colors, but maybe increasing them to three would give me more than enough space and still cater to my reduced color palette.

It also harkens back to CGA colors, which is the first colors I saw on computers.

Segmenting Colors

The general idea was to break the color space into ten segments, numbers from zero to nine. The first color (c0) would be based on the hue and the others would be 36° from that so they are evenly spread across the entire range of hues.

If the hue changes (like the POV hues), then everything else rotates to fit.

As you can see in the above images, there is also ten ranges of brightness (combination of saturation and lightness) that goes from zero to nine with zero being a not-quite-black version of the color and nine being a not-quite-white version.

This effectively gives me a wide range of colors that aren't very close to each other but still a small enough set I'm not struggling with the differences of 218° and 223°.

CSS

Because I've really embraced CSS variables instead of going directly to SASS or LESS, this entire thing uses various CSS variables such as --color-priduck-c0b0 and --color-priduck-c9b9. They use calc() and lch() to build the colors from the base hue of --color-priduck-hue.

Eventually, I hope to create some symbolic ones like --color-priduck-comment or something that would allow customizations.

This will also lead into the rgb() colors that I need to do the Fedran covers.

Libraries

I ended up writing two little NPM packages to support these and let me try it out. These are not even remotely documented because… I would consider them alpha until I get Fedran integrated with them. By then, I should have something I feel is “right” but also expand it out into the other components I need such as the color-rotation code to make the covers and other stuff.

Needless to say, if I like this, it will take me about six months to a year to finish integrating it into my system. Which means I want to make sure this is the direction I want to go.

I also decided to call this the Priduck Color Theme, because “priduck” sounded funny, doesn't have a lot of search hits, and it reminds me a lot of the Useless Duck Company, a set of hilarious shorts about inventing.

@priduck-color-theme/base for Node

https://src.mfgames.com/priduck-color-theme/priduck-color-theme-base-js

Naturally, trying to build all of these was a little tedious so I wrote an ad-hoc programs to generate the color schemes using CSS variables. Then put it into a NPM package because I don't like copy/pasting things when I know I'm going to be tweaking it over time (including adding symbolic colors so I can have a default colors for language keywords, surfaces, and accents).

This lets me just include it into the CSS files as needed.

@import "~@priduck-color-theme/base/colors.css";

:root {
    --color-priduck-hue: 220;
}

@priduck-color-theme/theme for Node

https://src.mfgames.com/priduck-color-theme/priduck-color-theme-theme-js

I also wrote another because there isn't a good way to write a DRY version of CSS themes that can handle default values, the browser providing prefers-color-scheme and prefers-contrast, and setting them via attributes in various combinations. So I wanted to be able to provide six files (dark/light and more/less contrast for both of those).

Re: Unsolicited opinions about CLI design

2024-01-06T06:00:00Z

This week is one of those “little things” week where I get to do fun things, work on the little broke things around the house, and just relax. It also means I get more verbose and start doing blog posts because why not?

Over on Gemini, there was a recent post about CLI design by Lark that caught my attention this morning. Well, and one about last names, but that is a much different topic.

As such, I just had a long conversation with one of my developers about our semi-annual goals. They wanted to document our primary CLI and asked me about opinions of their tasks. Ultimately, I suggested that their idea of creating a document that lists every option would be ultimately useless, expanding on the help from inside the CLI would be beneficial.

This also gives me the impetus to talk about some of my own evolving opinions about CLI.

External Documentation

I'll start with external documentation. Our application, call it bob, is a command-line that reuses the same business logic as our front end and services. It is intended to be a “big” one which means it has nested commands for individual tasks.

Many of those tasks are written because we need to solve a problem Right Now™. Other are the tasks because upper management has, in their infinite wisdom, separated our DBAs into a separate team which handles all the database woes of our company and we no longer have the dedicated individual that we've enjoyed for thirteen years. Since there is now a 1-3 hour delay on getting a DBA, they have to follow a strict system of access, and I have to puppet (re, take over their screen and type because I can't have direct access and they don't know the system or type quickly), I've been writing tools for things that I can programmatically do through our normal users.

This means that any external documentation is going to go stale. In a perfect world, I would write formal documentation as I code, but I also make a point of documenting every argument and command line as I go, so I'd rather have the bob handle the documentation instead of trying to also update our internal wiki, which customers don't have access to despite the fact we ship bob with our product also (for the same reasons I need it).

No Arguments

I completely agree with the -long argument name that find and PowerShell uses. It annoys the hell out of me. I also hate the /v of other Windows programs.

Same with exit codes. Different codes are awesome, when they are used properly. I'm looking at you, MadCap Flare.

Discovering

I think it would be obvious, but bob is inspired by git and az (thought more toward Azure's CLI). It uses nested verbs and they can go down reasonably deep.

az account list
bob direct property calculate

In both cases, I want the container verbs (az account, bob direct, bob direct property) to provide a short help, maybe some examples, than do a default behavior. This is because people who don't know the system need to be able to explore, but also know that there is more.

This is contrast to git remote which lists the remote, but git remote add adds remotes. There is no git remote list or git remote ls, so that makes it hard to discover what the tool can do because there is no obvious indicator that there are more nested verbs.

The Gitea CLI, tea, also does this but in a slightly different manner. Both tea repo and tea repo ls both list repositories but tea repo create will create a new one. That means if someone enters tea repo, there is nothing to indicate the create exists but at least there is a way to explicitly list repositories.

Since I value exploration and curiosity, I strongly lean toward container verbs are for discovery, not functionality. Ideally, a command that just gives input should also have a dedicated exit code (253 in most of my programs) for “showed help”.

Breaking Changes

The problem with the above statement is when a command needs to be broken into separate bits. For example, we used to have bob direct export to import data directly into the database (our direct commands are straight to the DB, our api commands use the OpenAPI layer). However, we split export into export data and export tables (bob direct export table).

What about the existing scripts that made the assumption about bob direct export? I don't want to be a versioning layer on our CLI and our customers don't understand semantic versioning even if we used it (business insists on romantic versioning). I also don't want to limit our ability to evolve as our understanding of the tools and how they work changes.

Ultimately, I go with three options:

Plan ahead and add the nested verb if I think we're going to need it.
Reorganize so we don't have to, which is why we have bob direct criteria export and bob direct property export.
Just break it (bump the major version if you can) and document the change.

Levels of Help

I prefer three levels of help: synopsis, option, and verb.

Synopsis is when you just run a program and it doesn't include the requirement arguments. Just give a little summary of what is needed. Depending on the parsing library, I'd rather see most command options, a list of verbs and what they do.

Option is when someone passes in --help (I do not like using -h for help). That should be the list of all verbs, options, and arguments and details.

This is where my opinions different from Lark. I hate when git clone --help opens a pager. I know what I'm doing and either I'm passing it already through a pager of my choice or I'm scanning because PowerShell is slow enough I can read almost as fast as it prints. I want it to dump data, I want to be able to scroll up as I need. Needless to say, I despise that git clone --help opens up a web browser on Windows. Switching programs is the last thing I want to do and it makes me feel like I'm losing control.

Also, if I touch the mouse, I failed. And browsers on Windows almost always require mice touching.

On the other hand, if it is given as a command or verb, such as git clone help, then I'm okay with a novel-length list of help files and throwing it into a pager is fine. I want the help verb to be the full details with examples, discussions, and links. Need more? Then make help have nested verbs that let me discover the detailed topics (but please no interactive exploration).

That means I'm also okay with the example:

$ go get --help
usage: go get [-t] [-u] [-v] [build flags] [packages]
Run 'go help get' for details.

Though I would have preferred that go get did the synopsis given above, go get --help gave the synopsis and explained what -t, -u, and -v are, and go help get (I'd rather it be go get help though) gave details like modules, examples, etc.

I believe that go get does something by itself, but I'd rather require an option that says “get all” (which is what I assume it does") instead of inferring it, but that is a point I feel strongly about. If go get does something, then skip that but my opinions that go get --help should at least list the verbs and explain the options.

Related to that, it frustrates me when --help does not show help screens. I don't care if I have every argument and option set, --help should always take priority.

Arguments and Options

There is one thing I struggle with the CLI design:

Prefer flags to args. It’s a bit more typing, but it makes it much clearer what is going on. It also makes it easier to make changes to how you accept input in the future. Sometimes when using args, it’s impossible to add new input without breaking existing behavior or creating ambiguity.

There isn't a good way of knowing when an flag is required. No standard convention that says “you must have this” because we are so inconsistent with indicating optional verses not optional. I guess if the synopsis said gary [--verbose] --input FILE maybe?

But this is one that I need to work out in my head because I usually equate positions are required, flags are optional. But, intellectually, I agree with the statement, but it isn't what I'm doing these days.

After thinking about it, I think I want to change to follow this one more.

Data Type of Options and Arguments

This is one of my frustrations, when I don't know if a flag takes a value or not, and what type of value does it take. In our system, we identify files or the type of input.

$ bob direct user list --help
...
--no-color
--table-search REGEX
--output FILENAME
--user-search LOOKUP

(“LOOKUP” has a special meaning for us such as starts:XXX, regex:XXX, id:999, key:XXXX, and contains:XXX verses XXX which is a case-insensitive exact match.)

There are cases when I want to pass in the value, so there is a different between --foo and --foo yes and the help should say that. Also, using a generic value, like tea does is frustrating.'

$ git repo create --help
...
   --gitignores value, --git value    list of gitignore templates (need --init)

Um, what is the value? And the help isn't what it does. It doesn't list the templates, it needs to have a template set but it doesn't tell me where to get a list of the templates.

Terminal Columns

Another frustration is that I don't like fancy tables in my CLI help. It might sound strange, but creating a nicely formatted table is great when you use standard columns:

$ tea repo create --help
...
   --branch value                     use custom default branch (need --init)
   --description value, --desc value  add description to repo

Now, saw you are jamming a narrow column of shell because your editor needs the bulk of the window but you need to handle some sidebar tasks or are writing code to use that CLI:

$ tea repo create --help
...
   --branch value                     use custom def
ault branch (need --init)
   --description value, --desc value  add descriptio
n to repo
   --gitignores value, --git value    list of gitign
ore templates (need --init)

... yeah, that isn't really readable. Wrapping stuff is a problem unless you take into account the number of columns that are currently on the screen. This is somewhere either the default needs to be screen wrapping, or we need a reactive formatting.

In a “perfect” world, I'd rather the narrow one look like:

$ tea repo create --help --faked
--branch value
  use custom default branch (need --init)
--description value, --des value
  add description to repo
--gitignores value, --git value
  list of gitignore templates (need --init)

Also, infrequently options should be longer and or multiple aliases mean there is a bigger chance of word-wrapping. Overall, ripgrep has a nicer default format for what I'm looking for:

$ rg --help
    --no-config
        When set, ripgrep will never read configuration files. When this flag
        is present, ripgrep will not respect the RIPGREP_CONFIG_PATH
        environment variable.

        If ripgrep ever grows a feature to automatically read configuration
        files in pre-defined locations, then this flag will also disable that
        behavior as well.

But even that doesn't handle columns well. I'd love to see a better convention here, even if it comes down to when everything should just be on a single line and let the terminal wrap.

$ bob --help
--log-level LEVEL, -l LEVEL

Sets the logging level for the
console output. Possible optio
ns are: Error, Warning, Info, 
Debug, Verbose. Case-insensiti
vie and shortened values allow
ed. Default: $BOB_LOG_LEVEL, I
nfo.

And since it usually comes up, I can't always zoom out the terminal and make the text smaller because a certain point, the blur radius from my eye surgery causes everything to turn into a muddled mess of colors.

Environment Variables

If an flag/option is driven by an environment variable (then it isn't a argument/positional), then list that in the --help. I believe the Woodpecker CLI does that (but I don't have it installed right now), but seeing something like this is nice:

$ bob direct property export --help
...
--connection DOTNET-SQL-CONNECTION The connection string to connect to
  the primary database. Value comes from $BOB_CONNECTION_STRING,
  "ConnectionString" from bob.config. Required.

That makes it a lot easier to understand, more so when the order of processing is also listed. If I'm using the example of --help verses help, those additional details are better suited for the command instead of the option.

Plural verses Singular

Singular. The same with my REST opinions. tea uses both and it looks wrong to me:

$ tea --help
...
     issues, issue, i                  List, create and update issues

Colors and Emojis

This is a hard one because I love and hate colors. Pretty colors are great, but I use terminals with black background, ones with light ones, and PowerShell with its hideous blue background. Sooner or later, a command with color screws up one of them because after decades of writing CLIs, we haven't come up with a convention to provide color preferences via environment variables.

Also, a decent percentage of the population is color blind. I also find that I stop processing color in certain situations and start seeing things in desaturated colors. So, as much as I want to see all those pretty colors in output, either don't touch my colors so I can use whatever background I want or come up with a standard that everyone follows.

I also love emojis, but there is a disconnect when some are black and white and others have full color. They are also dependant on fonts on the terminal.

Logging

Remarkably, logging flags cause me a lot of stress. As I see it, there are two philosophies when it comes to logging: flags or option.

Flags is when you have --verbose and --quiet. The problem is the conflict. What if you provide both? Should one take priority? Should it blow up with an invalid options selected? Mutually exclusive options for those libraries who handle it? And then there is the -vvvvvvv.

The other is an option, such as a lot of Microsoft tools use (also woodpecker-cli), which is a --log-level LEVEL.

Lately, I've been leaning toward the --log-level approach, but give it options. Since I use Serilog heavily and my users are sloppy, I like the level to be as flexible as possible. So, while I might say “error, warning, info, debug, verbose” (I always think debug and verbose should be reversed with Serilog), I want them to be able to say --log-level e or --log-level ERR because that's how they think.

I also frequently use the same tools in a monitored environment as my local machine, so occasionally I want more details in the log messages. That is also why I want to see more --log-format FORMAT where format can be one-line JSON, plain, with full timestamps, extra details, etc.

Finally, I have a common need to have log files written out to a text file. Usually this is --log-file FILENAME, but that format and level should be independently configurable. --log-file-format and --log-file-level with the same options, defaulting to the top level ones if not provided.

Of course, if --verbose is just an alias for --log-level verbose, that would be fine also. If I had to pick common or uncommon options, I would pick one set for the common.

Consistency

There is a bunch in here. Not all of them make sense for everyone, but they make sense for me. That comes down to the Standards Problem so I can't really say if they are “good” opinions or not, just my opinions and ones that I like and ones that I don't like.

But I like talking about it because it helps refine my opinions and find new ideas that I have never thought about.

Teaching NixOS about OpenTofu

2024-01-05T06:00:00Z

In my endless quest to come up with a completely data-driven and reproducible environment, I decided to take a stab at a new automation tool: OpenTofu. I've already gotten a good NixOS setup, but I wanted to also be able to check in the setup for my instances (and to a smaller degree, my bare metal servers in my home lab) to expand on the functionality. It didn't hurt that work had settled on Terraform.

Previously, I had taken a stab at Pulumi (during my wedding anniversary trip in 2022). It was fun and I liked the code but I ended up gutting it later. For some reason, it quickly ended up feeling like a chore to play with. At that point, I figured I would just do things manually. But then I saw the announcement that OpenTofu had forked from Terraform because of enshittification of licenses (develop with an open license, then switch to a more limited one once profits became important). That little thing set me off and I decided to try it out.

Installing

Since all my infrastructure code is in a Nix flake, to get started just required me to add to the shell's packages.

devShell.${system} = pkgs.mkShell {
  buildInputs = [
    pkgs.just
    pkgs.opentofu
    pkgs.openstackclient
  ];
};

I also grabbed the OpenStack client because it made easier to find some of the nasty little identifiers I needed to import.

Configuration Files

The way tofu works, it grabs all the *.tf files in the same directory. So inside my infrastructure flake, I have a src/tofu directory with configuration files that make sense to me:

000-providers.tf
050-secrets.tf (.gitignored)
050-secrets.tf.enc (SOPS encrypted based on my user key)
200-networks.tf
500-instance0.tf
500-instance1.tf

All of them are picked up, merged together, and made into a single set of settings. I also use tofu fmt a lot since I like to normalize my files on every commit.

OpenStack and DreamHost

Fortunately, my hosting provider of choice is DreamHost. They aren't the cheapest or the best, but they appear to be ethetical. Mostly, I stick with them because they went to the court to fight some overreaching gag orders.

(I also tried DigitalOcean at the same time as Pulumi but dropped that also.)

OpenTofu (via Terraform plugins) does a wonderful job of supporting both OpenStack and DreamHost DNS to tie everything together.

terraform {
  required_version = ">= 0.14.0"

  required_providers {
    dreamhost = {
      source  = "adamantal/dreamhost"
      version = "0.3.2"
    }

    openstack = {
      source  = "terraform-provider-openstack/openstack"
      version = "~> 1.53.0"
    }
  }
}

I added the adamantal/dreamhost plugin so I could also assign the DNS record directly from my script and have everything working.

Secrets

Even though my infrastructure flake is in a private repository, I still encrypt all my secrets. I use SOPS for this, which means setting up the .sops.yaml file to encrypt and then decrypting/encrypting files using `just``:

decrypt: decrypt-clouds decrypt-secrets

decrypt-clouds:
    if [ ! -f clouds.yaml ];then sops -d clouds.yaml.enc > clouds.yaml;fi

encrypt-clouds:
    cp clouds.yaml clouds.yaml.enc
    sops -i -e clouds.yaml.enc

decrypt-secrets:
    if [ ! -f 050-secrets.tf ];then sops -d 050-secrets.tf.enc > 050-secrets.tf;fi

encrypt-secrets:
    cp 050-secrets.tf 050-secrets.tf.enc
    sops -i -e 050-secrets.tf.enc

I do the same with my .env file so I can get the information I need set up properly.

Creating an Instance

Here is a short segment for creating an instance on DreamHost (or most OpenStack providers).

resource "openstack_compute_instance_v2" "instance1" {
  provider    = openstack.dreamhost
  name        = "instance1" // It really isn't instance1, but just pretend
  key_pair    = "keypair1" // This is my SSH key set up somewhere else
  flavor_name = "gp1.supersonic" // gp1.supersonic means I don't need a swap disk

  user_data = <<-EOT
    #cloud-config
    runcmd:
      - curl https://raw.githubusercontent.com/elitak/nixos-infect/master/nixos-infect | NIX_CHANNEL=nixos-unstable bash 2>&1 | tee /tmp/infect.log
    EOT

  # This sets up the boot device as /
  block_device {
    source_type           = "image"
    uuid                  = "2b2c61c6-324c-47f4-88c1-9ae8a978ddfd" # Ubuntu
    boot_index            = 0
    delete_on_termination = true
    destination_type      = "volume"
    multiattach           = false
    volume_size           = 80
  }

  network { // Also configured somewhere else
    name = openstack_networking_network_v2.public.name
  }
}

resource "dreamhost_dns_record" "instance1" {
  record = "instance1.mfgames.com"
  value  = openstack_compute_instance_v2.instance1.network[0].fixed_ip_v4
  type   = "A"
}

For me, the part is really cool is that I can bake in the NixOS infect script right in. To my surprise, it just ran the first time without errors (thought I had to wait about ten minutes after OpenTofu said it was done).

All I had to do was either show the results:

tofu plan

Or apply the changes:

tofu apply

Importing an Instance

Actually, the first thing I did was import my existing instances into the system. This involves creating a .tf file with the same basic setup at the other instance (some fields can be skipped but I was still learning), then import with the ID from OpenStack. Of course, getting the IDs was the hard part. Fortunately, this is where the OpenStack client comes into play. I can use that to get the list of servers, figure out the ID, then import it into Tofu.

$ openstack --os-cloud dreamhost server list
+--------------------------------------+-----------+--------+------------+--------------------------+----------------+
| ID                                   | Name      | Status | Networks   | Image                    | Flavor         |
+--------------------------------------+-----------+--------+------------+--------------------------+----------------+
| 55f8ee35-31b2-4137-af1d-b7597d348271 | instance0 | ACTIVE | public=*** | N/A (booted from volume) | gp1.supersonic |
| 1a38092b-bbc5-46bd-9092-0df979ca8fe4 | instance1 | ACTIVE | public=*** | N/A (booted from volume) | gp1.supersonic |
$ tofu import openstack_compute_instance_v2.instance0 55f8ee35-31b2-4137-af1d-b7597d348271

NixOS

Now, while this was great for setting up things, I also wanted to pull that data into my Nix infrastructure flake. Fortunately, OpenTofu has a way of exporting the data pulled from the cloud. To do that, I need to add an output stanza at the bottom of my 500-instance1.tf file:

output "instance1_ipv4" {
  value = openstack_compute_instance_v2.instance1.network[0].fixed_ip_v4
}

Since I'm (recently) fond of using Just for automation, I banged up a little stanza that automatically creates a default.nix file inside that directory every time I apply.

apply: decrypt format && export
    tofu apply

format:
    tofu fmt

plan: decrypt format
    tofu plan

export:
    echo "inputs: {" > default.nix
    tofu output | sort | perl -ne 'chomp;s@_@.@g;print "  $_;\n"' >> default.nix
    echo "}" >> default.nix

I had to use _ in the output since dotted notation isn't accepted, but I use perl to convert those underscores into Nix-happy format.

inputs: {
  instance0.ipv4 = "1.2.3.4";
  instance1.ipv4 = "2.3.4.5";
}

I use this to pull into my networking.nix which is used to drive things like configuring AdGuard, services like Maddy (for DeltaChat) and other services.

inputs:
let
  tofu = import ./tofu/default.nix { };
in
{
  instance0 = tofu.instance0;
  instance1 = tofu.instance1;
  instance2.ipv4 = "192.168.0.2";
}

From there, I have a single place to get all my IP addresses:

inputs:
let
  ip = (import ../../../../networks.nix { }).instance0.ipv4;
in
{
}

Conclusion

It isn't the best or most graceful way of doing things, but I'm pretty happy how everything turned out. I made a few mistakes along the way of setting up Gitea Actions and had to drop and rebuild my instance0. That was just a matter of renaming 500-instance0.tf, applying to drop, and then name the file back. Then I had nice clean slate to push out a new closure.

OpenTofu is much nicer than Pulmui. It didn't insist on having a cloud to maintain state, the file is checked into Git instead. It has a declarative language instead of code, and since I really don't need a lot of that logical flow, it just works for me. Plus I was able to inject into my just deploy top-level script that pushes out changes to my home lab and all my instances in a single call.

Passing NixOS Flake Inputs Through Colmena

2023-12-03T06:00:00Z

I'm pass the one year anniversary of using NixOS and I'm still learning a lot about the language and infrastructure. For the most part, I still feel good about the results even with my troubles playing video games and my underpowered hardware.

In the beginning, I also decided to go “all in” on Flakes. They covered a lot of the things I wanted in Nix, has the isolation that played well with my writing and programming environment. They were also barely understood, shiny, and I like taking the hardest path possible.

Once I realized I liked NixOS, I started putting it on other machines. For that, I settled on Colmena. It had pretty output and, for the most part, did what I wanted instead of the more complex systems that I couldn't understand due to my low skill in the Nix language.

Flake Inputs

One thing that had been vexing me for the last year is how to pass a flake input from infrastructure project (the repository that builds out all the servers) into the individual nodes (servers). This would let me install custom tools on servers or just use something outside of Nixpkgs that wouldn't require me to hunt down hashes every time the project built. Eventually, I hope to include:

Author Intrusion (if it ever works)
Fedran CLI
FlakeHub (current curiosity)
Catappuccin theme for Gitea

Since I was messing with NixOS over the last few days, I decided to make another attempt at solving the problem. A few questions on Matrix didn't answer, so I decided that a real-time chat isn't the best place for this type of question and went with the NixOS Discourse and posted there. Lo and behold, someone answered with exactly the answer I needed.

# flake.nix
{
  # An example flake to pass into the node.
  inputs.fh.url = "https://flakehub.com/f/DeterminateSystems/fh/*.tar.gz";

  outputs = inputs @ {self, ...}: let
    system = "x86_64-linux";
    pkgs = import inputs.nixpkgs {inherit system;};
  in rec {
    colmena = {
      meta = {
        # Inject the inputs for the top-level flake into a variable so the
        # nodes can pull them down consistently. I renamed this to flakes
        # instead of inputs since I always use `inputs` for my parameters
        # for nested Nix files.
        specialArgs.flakes = inputs; = {
          flakes = inputs;
          system = system; # I cannot figure out the "correct" way to do this yet.
        };
      };

      machine-a = import ./machine-a.nix;
    };
  }
}

# machine-a.nix
{config, pkgs, flakes, system, ...}: {
  environment.systemPackages = [
    flakes.fh.packages.${system}.default
  ]
};

Nix as a Language

I still don't really “like” Nix, but I'm getting better at it. It is one of those things that grows on you, even though I have a strong preference for statically compiled languages because they catch many trivial typos.

Automated handling of trivial things has become something I need. While I have pretty good attention to detail, I don't always have the bandwidth to full focus on an item. So anything that I can do to fix those without thinking is a good thing.

Trying to solve this is a good example of my problems. I spent a year trying to solve this, occasionally asking on Matrix and other places. And a GitHub issues page really isn't the best place to ask beginning questions, though I have done that in the past.

I also couldn't find an example of how to do things, which is one reason why I'm creating these posts. At least then, hopefully, someone who is struggling with the same things can stumble on the answer.

Enforcing Standards with NixOS

2023-12-02T06:00:00Z

Some time ago, I stumbled into std, a batteries-included development stuff. It looks like something I would really like to get into, mainly because it gave off notes of Buck2, which is something that interests me when dealing with microservice ecosystems and polyglot frameworks. And I know I love a polyglot solution to problems.

There were a few things that I fought again. I didn't care for the menu system that always shows up (noise), it's instability (still alpha), and the difficulty getting it to work with my way of thinking. I could have worked on some of those and figure out how to accept what I couldn't change and alter what I needed to be productive.

I don't really have that energy at the moment. I'm painfully aware that my time and attempt budget has been eroded by my family, drama, and the other things going on in my life. I find that I don't have the energy to do much and getting std to play with me was one of those things I decided to bump.

Automation Tools

However, there were some things I really liked about std that I wasn't aware of, namely Nixago which is a way of having the shell hook of a Nix setup automatically write out the various configuration files for things like Conform for Git messages (I like my conventional commits), EditorConfig for formatting, and Lefthook to make sure everything is honored. Standard also taught me about Treefmt which is a single command to reformat a command base.

There was also a way of doing arbitrary configurations, such as maybe setting up my project configuration files or handling other things, but I couldn't figure out how with a cursory look.

In short, all of those things I like to handle automatically instead of remembering all the little details.

With me getting rid of std, I wanted to keep this. Ideally in a manner that I could eventually create a flake of my common configurations and then apply them to every story or programming project.

Necessity Calls

Getting rid of std meant I hand to figure it out. Last night, I sat down and went through the code with my growing skill at Nix (I still do not enjoy the language but I'm getting more fluent with it). I'm also messing with FlakeHub so you'll see some elements from that library.

I already use direnv for setting up my flakes as I enter directories. That is part of my normal tool set and I plan on using that for a great deal of time.

Layout

I like small, individual files. Naturally this means I would like every automated system to have its own file but grouped together in a folder to make it obvious how they are used. (Needless to say, I don't advocate coding without folders but that is also how I work/)

$ find src -type d
src
src/configs

Inputs

The first part is pulling in the inputs for Nixago and its extensions.

# flake.nix
{
  inputs = {
    nixpkgs.url = "https://flakehub.com/f/NixOS/nixpkgs/*.tar.gz";

    nixago.url = "github:jmgilman/nixago";
    nixago.inputs.nixpkgs.follows = "nixpkgs";

    nixago-exts.url = "github:nix-community/nixago-extensions";
    nixago-exts.inputs.nixpkgs.follows = "nixpkgs";
  };

  # Flake outputs that other flakes can use
  outputs = inputs @ { self, nixpkgs, nixago, nixago-exts }:
    let
      # This bit comes from Flakehub's init and seems to be a reasonable pattern.
      supportedSystems = [ "x86_64-linux" ];
      forEachSupportedSystem = f: nixpkgs.lib.genAttrs supportedSystems (system: f {
        inherit system;
        pkgs = import nixpkgs { inherit system; };
      });
    in
    {
      devShells = forEachSupportedSystem ({ system, pkgs }:
        let
          # This pulls in the configurations from the configuration directory.
          configs = import ./src/configs/default.nix { inherit system pkgs nixago nixago-exts; };
        in
        {
          default = pkgs.mkShell {
            # Pinned packages available in the environment
            packages = with pkgs; [
              git         # Needed for life until I find something more awesome
              nixpkgs-fmt # Needed for Lefthook
              treefmt     # Needed for Lefthook
              lefthook    # Needed for Lefthook
            ];

            # Configuration setup
            shellHook = ''
              ${configs.shellHook}
              lefthook install
            '';
          };
        });
    };
}

Configurations

The basic default is just so I have a single line to configure all the libraries. This just acts as an index file.

# src/configs/default.nix
inputs:
inputs.nixago.lib.${inputs.system}.makeAll [
  (import ./conform.nix (inputs))
  (import ./editorconfig.nix (inputs))
  (import ./lefthook.nix (inputs))
  (import ./prettier.nix (inputs))
  (import ./treefmt.nix (inputs))
]

Prettier

Prettier was the first one I used, since I have very little customization in it. The nixago-exts is an extension library that figures out a lot of the formats so I don't have to.

# src/configs/prettier.nix
inputs @ { system, nixago, nixago-exts, ... }:
nixago-exts.prettier.${system} {
  data = {
    printWidth = 80;
    proseWrap = "always";
  };
}

Lefthook

Lefthook's configuration is the same, but you'll notice there is no data = element like most of the others. This threw me because it is different than the others. I also found that I had to add && git add {staged_files} from most of the examples I saw others when I commit, it would reformat the code but then leave them modified for the next check in. Adding the files fixes that and keeps things relatively speedy.

You also can see how I refer to specific paths for the executables while cleaning up the code.

# src/configs/lefthook.nix
inputs @ { system, pkgs, nixago, nixago-exts, ... }:
nixago-exts.lefthook.${system} {
  commit-msg = {
    commands = {
      # Runs conform on commit-msg hook to ensure commit messages are
      # compliant.
      conform = {
        run = "${pkgs.conform}/bin/conform enforce --commit-msg-file {1}";
      };
    };
  };
  pre-commit = {
    commands = {
      # Runs treefmt on pre-commit hook to ensure checked-in source code is
      # properly formatted.
      treefmt = {
        run = "${pkgs.treefmt}/bin/treefmt {staged_files} && git add {staged_files}";
      };
    };
  };
}

As a side note, I have not found a single fast C# reformatter that only handles a few files. It has been intensely frustrating because I really like ReSharper's “Silently Clean” feature and I don't have a way of doing it from the command line in a reasonable period of time.

Conform

Conform is nice because it enforces Git commit messages for conventional commits.

# src/configs/conform.nix
inputs @ { system, nixago, nixago-exts, ... }:
nixago-exts.conform.${system} {
  commit = {
    header = { length = 89; };
    conventional = {
      # Only allow these types of conventional commits (inspired by Angular)
      types = [
        "build"
        "chore"
        "ci"
        "docs"
        "feat"
        "fix"
        "perf"
        "refactor"
        "style"
        "test"
      ];

      # If you want scopes, then add:
      #scopes = ["allows" "scopes" "here"];
    };
  };
}

Treefmt

Cleaning up code is something that is tedious but really needs to be done to lower the bar of allowing others into the code. It is also something that can be “mostly” automated, which I'm also in favor of. Sadly, there are gaps in the tools that I want, like a C# or Rust formatter that will organize members (such as grouping public properties together and making them alphabetical), but I can live without those.

# src/configs/treefmt.nix
inputs @ { pkgs, ... }:
let
  data = {
    formatter = {
      prettier = {
        command = "${pkgs.nodePackages.prettier}/bin/prettier";
        options = [ "--write" ];
        includes = [
          "*.css"
          "*.html"
          "*.js"
          "*.json"
          "*.jsx"
          "*.md"
          "*.mdx"
          "*.scss"
          "*.ts"
          "*.yaml"
          #"*.toml"
        ];
        excludes = [ "**.min.js" ];
      };

      nix = {
        command = "nixpkgs-fmt";
        includes = [ "*.nix" ];
      };
    };
  };
in
{
  # I don't understand the reason why many Nix examples define
  # the data in the let section and then just inherit it here.
  inherit data;
  output = "treefmt.toml";
}

EditorConfig

I think one of the best things that came out of the last decade or so of coding was a slow migration to having a semi-universal file for configuring line endings, trimming white space, and the others. Also, both Microsoft and Jetbrains have embraced EditorConfig so I can check in a file that reduces the trivial (but needed) rejects for pull requests. I want more tools to use this and extend them because I don't want to bother with line indents, tabs verses spaces (tabs lost, but I've accepted spaces now), and formatting rules (braces on new lines).

# src/configs/editorconfig.nix
inputs: # I don't use the inputs, I just wanted all the calls in `default.nix` to be consistent.
let
  data = {
    root = true;

    "*" = {
      end_of_line = "lf";
      insert_final_newline = true;
      trim_trailing_whitespace = true;
      charset = "utf-8";
      indent_style = "space";
      indent_size = 4;
      indent_brace_style = "K&R";
      max_line_length = 80;
      tab_width = 4;
      curly_bracket_next_line = true;
    };

    "*.md" = {
      max_line_length = "off";
    };

    "package.json" = {
      indent_style = "space";
      indent_size = 2;
      tab_width = 2;
    };

    "{LICENSES/**,LICENSE}" = {
      end_of_line = "unset";
      insert_final_newline = "unset";
      trim_trailing_whitespace = "unset";
      charset = "unset";
      indent_style = "unset";
      indent_size = "unset";
    };
  };
in
{
  inherit data;
  hook.mode = "copy";
  output = ".editorconfig";
  format = "toml";
}

Obviously, the C# version is huge with lots of settings to fit my standards.

Benefits

The nice part about this is all I have to do is change into the directory and direnv will automatically make sure all the files are correct and up to date. Since I'm mostly on the command line, this works out beautifully for me and how I work.

$ cd bakfu
direnv: loading ~/src/bakfu/.envrc
direnv: using flake
evaluating derivation 'git+file:///home/dmoonfire/src/bakfu#devShells.x86_64-linux.default'
nixago: updating repository files
nixago: '.conform.yaml' link updated
nixago: '.editorconfig' copy is up to date
nixago: 'lefthook.yml' link updated
nixago: '.prettierrc.json' link updated
nixago: 'treefmt.toml' link updated
direnv: export +AR +AS +CC +CONFIG_SHELL +CXX +HOST_PATH +IN_NIX_SHELL +LD +NIX_BINTOOLS +NIX_BINTOOLS_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu +NIX_BUILD_CORES +NIX_BUILD_TOP +NIX_CC +NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu +NIX_CFLAGS_COMPILE +NIX_ENFORCE_NO_NATIVE +NIX_HARDENING_ENABLE +NIX_LDFLAGS +NIX_STORE +NM +OBJCOPY +OBJDUMP +RANLIB +READELF +SIZE +SOURCE_DATE_EPOCH +STRINGS +STRIP +TEMP +TEMPDIR +TMP +TMPDIR +__structuredAttrs +buildInputs +buildPhase +builder +cmakeFlags +configureFlags +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +doCheck +doInstallCheck +dontAddDisableDepTrack +mesonFlags +name +nativeBuildInputs +out +outputs +patches +phases +preferLocalBuild +propagatedBuildInputs +propagatedNativeBuildInputs +shell +shellHook +stdenv +strictDeps +system ~PATH ~XDG_DATA_DIRS

Moving Parts

There is a problem with this, in that it is a lot of moving parts to basically write out a file that could easily be checked into code once and be done with. I fully admit that we are going through a lot to hoops that could easily be done with simple files with only one exception.

The biggest exception is Lefthook. Someone needs to run lefthook init after the repository is cloned to ensure the hooks are all configured so it enforces the commit messages and makes sure the code is formatted before committing. Since Git won't ever provide that, having the shell hook from the flake enforce it cuts out a tedious step that is easily overlooked.

The other reason for going with this approach is my ability to update it. Each of my stories are in their own repository for a variety of reasons, but the structure and layout of them are typically manipulated en masse as I update a standard. I also do theme and style changes across the boards, such as finding a new font or fixing the ebook generation.

With the flake setup, I could easily migrate these configurations to a dedicated flake that is shared across all of them, and then just update the lock for that file to enforce the latest iteration of an evolving standard.

And standards are evolving. While I have common patterns (braces on newlines), how I format the code, organize files, or hook up patterns changes. Sometimes it is a little incremental change, sometimes it is a sweeping change as I switch build systems or introduce a standard format. Last year, I set it up so every project would generate a EPUB and PDF file and work with my Gitea and Woodpecker CI setup.

If I can make those changes cut across all the projects, then it is less effort for me to get conformity but also let me work in an environment I'm comfortable with. Being able to make sure every tool I want is available (such as Author Intrusion or markdowny) means I don't have to think about the plumbing and just do the part that is fun: write.

Package Management - Formats and Registries

2023-11-30T06:00:00Z

Since my mind has been on it, I wanted to work out some of the ideas I had for formats in my packaging system. In this case, I'm going to focus on a single one, NuGet, because I have a fair amount of experience with this and it has some of the complexities that are throwing me.

Series

This is going to be a series of posts, but I have no idea of how fast I'll be writing them out. I want to work out my ideas, maybe have a few conversations, and then start to move to more technical concepts.

2023-02-07 Package Management - Introduction - The beginning of a short series as I write down my ideas of an ideal package system with the future goal of writing a generic system called Bakfu.
2023-02-08 Package Management - Versions - A short discussion on packages and their versions.
2023-02-12 Package Management - Identifiers - A short discussion on how to identify packages.
2023-02-13 Package Management - Dependencies - A review of different styles of package dependencies, including across different ecosystems, and then an attempt to consolidate them into a single unified system.
2023-09-20 Package Management - Identifiers 2 - Using URNs to identify packages and some re-thinking of concepts.
2023-11-30 Package Management - Formats and Registries - Thoughts on setting up formats and layering registries for those formats on top of each other.

Configuration Files

All of the configuration for the system will be in a series of JSON5 (or JSON or whatever formats are officially supported) that are merged together with any conflict producing an error message and stopping the systems.

Assuming $GITDIR is the top-level directory for a Git repository, then the configuration would be in $GITDIR/.config/bakfu. All the files will be gathered together, but a .gitignore could ignore *.user.* which means authentication information could be stored locally but have a common configuration on top of that.

In this case, the files all have the same schema which is a required component because it also identifies the version of the file.

{
    "$schema": "...",
}

Any more details on how we look for configuration files will have to wait for another post.

What is a Format?

Right now, a “format” is a specific format of a package, such as NuGet package or a NPM one. Inspired by SGML catalogs and how I like to see things in Git repositories, a format looks roughly like this JSON5.

{
    // In this file, the various components don't have to be URL encoded.
    "formats": {
        "nuget": {
            "defaults": {
                "authority": "nuget",
            },
            "authorities": {
                "nuget": {},
            },
        },
    },
}

Since we merge files, that means I could create project-specific authority for packages that aren't (and probably never will be) on the official NuGet server. For example, my personal Forgejo instance at https://src.mfgames.com/mfgames-cil.

{
    // In this file, the various components don't have to be URL encoded.
    "formats": {
        "nuget": {
            "authorities": {
                "src.mfgames.com/mfgames-cil": {},
            },
        },
    },
}

Authorities

The authority itself is the complex part of the file because it needs to handle how to search for packages, how to download them, authorization needed, and how verification is done.

// merged formats.nuget.authorities:
"nuget": {
    // "enabled": true, // Implied so it can be disabled
    "registries": {
        "nuget.org": {
            "protocol": "nuget-v3",
            "url": "https://api.nuget.org/v3/index.json",
        },
    ],
}

Another file could merge additional registries in, much like you can have a proxy feed in DevOps.

// merged formats.nuget.authorities:
"nuget": {
    "registries": {
        "nuget.org": {
            "protocol": "nuget-v3",
            "url": "https://api.nuget.org/v3/index.json",
        },
        "example": {
            "protocol": "nuget-v3",
            "url": "https://example.org/proxied-feed/v3/index.json",
        },
    },
}

Controlling Order

In the NuGet.config file, there is also the ability to clear out the list of registries and use only a single set of identified registries. In this case, it would be a combination of disabling the known ones, changing the search for other files (the later post), and using a set of ordering controls.

// merged formats.nuget.authorities:
"nuget": {
    "registries": {
        "nuget.org": {
            "protocol": "nuget-v3",
            "url": "https://api.nuget.org/v3/index.json",
            "enabled": "false",
            // Below implies "search": { "after": ["example"] } }
        },
        "example": {
            "protocol": "nuget-v3",
            "url": "https://example.org/proxied-feed/v3/index.json",
            "search": {
                "before": ["nuget.org"],
            }
        },
    },
}

Additional Packages

In NuGet.config, it is possible to map a set of packages to a given URL, such as all MfGames* can only be found at a specific server. In those cases, that should be treated as a separate authority with its own set of registries (that may be a duplicate if it is also a proxy feed).

In the example below, contoso would be treated as a separate authority than the nuget default.

<!-- NuGet.config -->
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSourceMapping>
    <packageSource key="nuget.org">
      <package pattern="*" />
    </packageSource>
    <packageSource key="contoso.com">
      <package pattern="Contoso.*" />
      <package pattern="NuGet.Common" />
    </packageSource>
  </packageSourceMapping>
</configuration>

Protocol

The protocol determines how the registry is accessed. I could see a number of possibilities:

NuGet V3 protocol, obviously NuGet-centric
NPM access
Directory location
gRPC proxy server

Since this library needs to be implemented across a number of platforms and libraries, I would expect that unknown protocols would be filtered out (maybe with a warning) and then the ones that can be accessed are used. If there are no valid ones, then the system should blow up.

The protocol also determines what additional settings might be required such as authentication, file system layout, or the like. It would be obvious specific to that protocol, so it is thrown into a generic “settings” object to control those things.

"local": {
    "protocol": "file-v1",
    "url": "file:///${env:HOME}/src/other/project",
    "settings": {
        // The layout for "bob" would be "b/bob".
        "package": "${PACKAGE_NAME:0-0}/${PACKAGE_NAME}/${PACKAGE_VERSION}",
    },
},

Search Controls

Another aspect if how to handle searching. NuGet will search all the sources at the same time and the first one that responds gets it. However, in some cases, one might want only certain ones to be searched and then stop if it isn't there (such as a full feed proxy verses a subset proxy feed).

I would see this as controlled by two components, at the authority and for an individual repository.

// merged formats
"nuget": {
    "authorities": {
        "nuget": {
            "search": {
                "concurrent": true,
                "defaultOrder": "Alphabetical",
            },
            "registries": {
                "nuget.org": {
                    "protocol": "nuget-v3",
                    "url": "https://api.nuget.org/v3/index.json",
                    "search": {
                        "notFound": "Stop",
                        "timeout": {
                            "time": "00:01:00",
                            "action": "Retry",
                            "maximumRetries": 3,
                        },
                    },
                },
                "example": {
                    "protocol": "nuget-v3",
                    "url": "https://example.org/proxied-feed/v3/index.json",
                    "search": {
                        "notFound": "Continue",
                    },
                },
            },
        },
    },
}

Conclusion

Well, that's my thoughts on authorities and how to search them for packages. One thing you might notice is that I don't have offline packages in the above examples. I want to treat offline (cached) files as a first-class concepts in the packaging and so that requires its own discussion. Not to mention, the cache is for all formats, not just one.

I also want to eventually introduce the ability to have services provide opinions on packages. This way, I could set up a service that translates CVE alerts into controls of the packages found or allow a project-specific settings that would hide packages that were incompatible with the current project. I just don't know how to call them.

Computer science cannot solve two things: cache invalidation, how to name things, and off-by-one errors.

Package Management - Identifiers 2

2023-09-20T05:00:00Z

It's been a many (seven) months since I started working on the ideas of a package management system. A lot's happened but something got me down the path again and I decided to look at it again. This isn't a straight journey where I write every single post ahead of time but instead its more of ramblings and thoughts.

Or, in the words of one of my favorite puzzle games, The Thirtieth Guest:

Feeling lonely?

Series

2023-02-07 Package Management - Introduction - The beginning of a short series as I write down my ideas of an ideal package system with the future goal of writing a generic system called Bakfu.
2023-02-08 Package Management - Versions - A short discussion on packages and their versions.
2023-02-12 Package Management - Identifiers - A short discussion on how to identify packages.
2023-02-13 Package Management - Dependencies - A review of different styles of package dependencies, including across different ecosystems, and then an attempt to consolidate them into a single unified system.
2023-09-20 Package Management - Identifiers 2 - Using URNs to identify packages and some re-thinking of concepts.
2023-11-30 Package Management - Formats and Registries - Thoughts on setting up formats and layering registries for those formats on top of each other.

Mistakes Were Made

While I was reading the first identifiers post, I realized I made a few mistakes. One was that I was using a URL instead of a URN to identify a package:

In contrast, URNs were conceived as persistent, location-independent identifiers assigned within defined namespaces, typically by an authority responsible for the namespace, so that they are globally unique and persistent over long periods of time, even after the resource which they identify ceases to exist or becomes unavailable.

When it comes to identifying a package, that is exactly what we want. A persistent identifier that doesn't point to a specific location. When we want Markdowny, an important part is that we don't want to mandate where to get it, but enough to identify it.

Rehashing as URN Components

URNs always start with urn: and a registered “code”. We're going to pretend bakfu is the registered code, so that means all the package identifiers would be urn:bakfu: then something.

Likewise, I think it is important to know when something is a package identifier (say pkg:) or a reference to a package which has ranges (ref:).

EDIT: After looking at my notes, I already figured out why I didn't need this so I removed it from the content below.

From earlier posts, I decided on the package format being a component with well-known versions (npm, nuget, deno) and arbitrary ones (domain-based).

urn:bakfu:npm
urn:bakfu:minetest
urn:bakfu:authorintrusion.com/spell-check

In the later examples below, I'm going to cut off the urn:bakfu: as noise so npm.

Authority

I don't have a lot of problems or doubt with the components above. The next part, on the other hand is a bit more complicated and fluid. As I develop more, I think we should have separate package repositories/registeries instead of putting everything at npmjs.com or nuget.org. However, that leads into potential name and identifier conflicts.

Using the example from my life, when I started Nitride, I made all the namespaces “Nitride” and was going to buy a developer SSL to push it up to nuget.org. However, by the time I got to a stable point, there was then a Nitride already there and that didn't work.

(This is also one reason why I don't like identifiers that aren't namespaced.)

At the same time, I want to keep these URNs relatively “simple” for the 99% cases. In those cases, that means I want to aim for something like:

npm:markdowny
npm:@mfgames-writing/format
nuget:Humanizer

But, if there is a non-default location, the URN needs to have some mechanism that identifies the “authority” of a package. This authority doesn't need to be a URL, just a unique key to distinguish between two packages with the same identifier.

Originally I thought about something like npm:///markdowny based on using file:/// to reference the local file system but allow a domain and directory to be used:

npm://mfgames.com/markdown
npm://mfgames.com/@mfgames-writing/epub2
npm://example.org/~user/@example-organization/example-package

The problem with that is the last one. Where does the directory structure end, where does the package begin? If the entire URL is opaque, then it would be easy to leave as-is, but because this has to translate, there needs to be an unequivocal way of splitting them into an authority and a package identifier.

URL encoding to the rescue.

If we treat the optional directory structure (on the optional authority domain) as a single “unit”, then we can keep the slash to separate authority from the identifier but still keep the identifier in its most common format (NPM uses slashes):

npm:markdowny
npm:///markdowny (same as above)
npm://npmjs.com/@mfgames-writing/epub2
npm://npmjs.com%2F~dmoonfire/@mfgames-writing/epub2

I think this would work because we can have a rule that states that the component after the package type has either two slashes for an authority and the next slash ends that component. Everything after that is the full identifier.

Well-Known URLs

I'm fond of the .well-known/ infrastructure that has built up over the years. I could easily envision that this could translate into an actual URL to help identify the location of the packages if not known.

https://npmjs.com/.well-known/bakfu/npm/npmjs.com%2F~dmoonfire/@mfgames-writing/epub2

The resulting JSON file would give common locations where to find it. So going to the @mfgames-writing/epub well-known URL would then give the URLs for the official servers or locations, such as npmjs.com, my local package repository, an IPFS address, or whatever makes sense.

The reason it won't use query strings like webfinger is because query strings don't play well with static sites and I use static sites pretty heavily.

Qualified Identifiers

I think the ideas from the original identifiers post for qualified identifiers still have merit, but without the bakfu: prefix because it ends up just being noise. I think these should be limited and defined ahead of time since there is flexibility on the features.

java:org.example.hyphenated_name
npm:markdowny?version=1.1.0
cargo:serde?version=1.0.152&feature[]=derive&feature[]=rc
cargo:serde?version=1.0.152&platform=x86_64-unknown-linux-gnu

Additional Versions

If the package version (as opposed to the content version) is needed, then &package=1.0.0 can be used. Likewise, if the Bakfu itself needs to be bumped, then &bakfu=1.2.0 can be used.

I thought about making versions arbitrary, but I couldn't imagine a case where a package would have two different versions for two purposes. Those would be two separate packages in that case.

Overriding Packages

One of the reasons of this exercise is how to do a modification to a library that is already releases but the users learn after the fact that it breaks semantic versioning (SlimMessageBus). One constraint to this is that according to the specification, the version of the content cannot change once published.

That is why the package has its own version, to indicate that the package metadata such as the dependencies and requirements, can change independently of the contents. In most cases, package=1.0.0 but a proxy service could add in the modified dependency and call it package=1.0.1-service which would then cause the packaging system to prefer the highest version of the package with the same version.

There is some gaps because if we had a Bakfu-aware packaging system, someone could create a package and then keep bumping the package version higher to override anyone's overrides but I think this is a case where an upstream package modification should be blocked if there is an override given. At least until that new version can be reviewed and accepted.

Conclusion

The main reason to have these package identifiers is just to distinguish a package uniquely across the entire ecosystem. I strongly believe there needs to be a decoupling of the location verses identifier because of the other goals in this project: moving from one host to another, caching packages, being able to provide a curated list, blocking malicious packages, and to add after-the-fact changes.

Overall, I think this fits my need for something that is roughly ascetic (urn:bakfu:npm:markdowny?version=1.0.1), has a most-common use of something simple and readable (urn:bakfu:npm:markdowny), but still allows distributed packages and cases where there are name collisions (urn:bakfu:nuget://mfgames.com/Nitride).

It also can be reduced to a common form based on context such as removing the urn:bakfu: which makes the simplest version npm:markdowny.

Also, it shouldn't be hard to create a normalized rule to turn it into a proper C# or Rust structure for doing the next steps.

Fedran Infrastructure Redux

2023-06-27T05:00:00Z

I know I said that I was going to work on writing after working on the previous post, but I tried to speed up the pipelines because an hour was overwhelming and bothered me.

Thanks to a bit of obsession, I figured it out.

The bulk of the pipeline was spent download and building Nix packages. This normally isn't a problem, but I want to be able to run an agent in my home lab where I'm charged for bandwidth overages. Pulling down half a gig of packages every time I built my website was also not being a good steward of the Internet as a whole.

I had bookmarked a link to kotatsuyaki's Locally Cached Nix CI with Woodpecker which was based on Kevin Cox's work and tried it a couple of times. I usually got hung up on switching from Docker to Podman but eventually got it mostly working, hit some snags, and found a slightly-less-than-optimal approach instead.

Lazy Configuration

It took me a day to figure out how to get Podman working with my setup. Naturally, OCI images in Nix change name so it was docker-woodpecker-server in Docker and podman-woodpecker-server on Podman. This meant I had to also switch the various systemd links that tie in changing secrets with restarting the OCI images.

As usual, once I realized I was going back and forth (usually on the third time), I decided to automate that.

# woodpecker-agent.nix
{
  config,
  pkgs,
  lib,
  ...
}: let
  tag = "next-eaae6b44c7";
  container =
    if config.virtualisation.podman.enable
    then "podman"
    else "docker";
in {
  sops.secrets = {
    woodpecker-ci-agent = {
      restartUnits = ["${container}-woodpecker-ci-agent.service"];
    };
  };

I'm not fond of Nix as a language, but it was nice being able to have it pick up which configuration was set up properly and then rename the various systemd units as appropriate.

Mounting Nix Store

The crux of the problem was storing Nix packages so I didn't have to download them every time I built the package. This is really important when I start doing mass changes to my writing projects since one of those causes ~137 pipelines to be triggered. Since each one uses an identical flake.lock (because of the Rust CLI), caching those packages also means that it can generate each of the PDFs and EPUBs automatically when I update packages, change style, or need to rebuild things.

Originally I tried the example in the above link and it worked beautifully. I did have to make the pipeline trusted, but I run my own CI server and I don't build pull requests, so I have that locked down to avoid too much security exposure.

# .woodpecker.yaml
pipeline:
  run-greet:
    image: nixos/nix
    commands:
      - echo 'experimental-features = flakes nix-command' >> /etc/nix/nix.conf
      - nix run --store unix:///mnt/nix/var/nix/daemon-socket/socket?root=/mnt .#greet -L
    volumes:
      - /nix:/mnt/nix:ro

I do think I may have had better luck if I included Kevin's command in the above link.

I have seen some issues when using –store. I found that this can be fixed by additionally passing –eval-store local.

Though by the time I realized I should have tried that, I found a working solution and needed to stop messing with an incremental change without significant improvement over what I had.

When I migrated my Fedran pipeline over to the above --store implementation, it blew up with being unable to link DLLs/shared libraries. First it was sodium23, then lowdown, then something else. Each time, I added more packages to the build until I hit libnixeval.so. That one… I couldn't fix.

The main problem was the run-greet pipeline didn't need to call the nix executable, but I was using Standard which does. That means, I couldn't use std //cli/apps/default:run to run anything which basically meant I had to use cargo build which negated the entire purpose of using Nix packages to cache. I also couldn't use nix run . to take advantage of the Nix caching.

In the end, I switched back to Docker but with Kevin's first suggestion: just create a volume in Docker to share the Nix store. This worked out well… but I had to make a tweak from Kevin's original suggestion:

# .gitlab.yaml (I don't remember what it is called anymore)
[[runners]]
    executor = "docker"
    [runners.docker]
        volumes = ["/nix"]

I found it worked better if I used a named volume:

# .woodpecker.yaml
pipeline:
    deploy:
        image: nixpkgs/nix-flakes
        commands:
            - nix develop --command ./src/website/scripts/ci.sh
        when:
            event: [push, manual, cron]
            branch: main
        volumes:
            - woodpecker-nix-store:/nix # named instead of just `- /nix`

And that worked beautifully. The build time when down from 48 minutes for the last run to 6 minutes at the most current one. Plus it barely downloaded anything thanks to using Crane for Rust in Nix, and the general package management. If I created a second store to keep the Git repositories and only did a git pull instead, I could shave it down even more.

Cleanup

It did occur to me that if I left what I figured out as-is, sooner or later I would run out of space. (I also was reminded by I always create a dedicated /var/lib/docker partition in this). So on my nightly job, I added this stanza to my .woodpecker.yaml file (which I haven't tested very well):

# .woodpecker.yaml
pipeline:
    clean:
        image: nixpkgs/nix-flakes
        commands:
            - nix-collect-garbage --delete-older-than 15d
        when:
            event: [cron]
            branch: main
        volumes:
            - woodpecker-nix-store:/nix

If I did it right, then it should keep everything cleaned up and tidy. I'll find out in a month or so if that is true.

Checkout

One thing to be said about having so many project pipelines is that I hammer my system when I do a mass change (updating the locks for example). This meant I encountered a strange bug where Woodpecker's checkout plugin starts to fail somewhere around the tenth rapid-fire pipeline.

To get around it, I switched from using the built-in clone to do it manually:

# .woodpecker.yaml
skip_clone: true

pipeline:
    clone:
        image: alpine/git
        commands:
            - git clone https://fedran:$GITEA_TOKEN@src.mfgames.com/$DRONE_REPO.git . --depth 1
            - git checkout $DRONE_COMMIT
        secrets:
            - gitea_token
        when:
            event: [push, manual, cron]
            branch: main

Since I put it in, I haven't seen check out failures due to being unable to get the user name and password from the prompt. I could have also used nixpkgs/nix-flakes instead of alpine/git, but didn't. This works and it's fast.

What's Next

I really need to write. I got a closure point. Everything works, I wrote issues up for the ones that aren't breaking, so I can stop obsessing and instead focus on the 10 kwd obligation I have by the end of the month. Which is to say… probably not going to happen but I'm going to still try.

Fedran Infrastructure

2023-06-24T05:00:00Z

In the first weeks of March, June, September, and December, life gets complicated for my day job. Usually that means the pressure gathers in intensity in the preceding weeks and then winds down after the quarterly release.

It is time when my other obligations start to press into my attention and I'm too busy to focus on them. Naturally that means as the release winds down, I try to get focus on the others.

This month, I managed to get twenty thousand words on a commissioned novel. I also stumbled into a few things on the Internet that set off my fixation and I ended up working on them for a few weeks as a “play coding” to relieve the stress. This month, it was almost entirely related to Fedran, my fantasy setting that I also use to create what I'm hoping is a full-featured website suitable for spoilers, “who is this character again,” and just background stories.

Fedran Monorepo

Some time ago, I decided to bulk most of the “backend” of Fedran to a single Git repository. I'm still keeping my sources (all the stories, novels, and poetry) in individual repositories though I did go through the exercise to see if I wanted it (I do, there are too many benefits). But the infrastructure is a different story.

There isn't much there right now, just a standard directory structure to locate the files, the Nitride website, and a Rust CLI which sets up the individual stories in a standard configuration, lets me do queries across the projects, and starts to help me coordinate everything together.

Both have built up over the years as I've tried to automate one problem after the other. Moving them into a single repository meant I needed to back out a lot of the cruft that was to coordinate the multiple repositories. I didn't have to manually build the Rust CLI anymore, I didn't need to integrate download the projects into it, I didn't need all the configurations and tokens because there was now one official layout and structure to work against.

Last last month, I got the website migrated over. This week, I got the CLI fully integrated, actually building, and doing things. This means when I edit a chapter to add content warnings or add details, the website deployment will pick up the changes without me having to remember (e.g., not getting it done more twice or thrice a year).

It does add a half hour to the build process because I'm apparently incapable of picking a single language to work on and I haven't figured out caching on my Woodpecker CI server, so it downloads the entire repository, a half GB of Rust packages, another quarter of NuGet packages, and about the same in Node packages to build it.

That I can work on.

This took a while, mainly because I was struggling to get it integrated into a standalone and remove the bulk of the overhead I needed when it was separate.

NixOS and Standard

Last week, while wandering around the internet, I stumbled on Standard, a setup for NixOS flakes (like I use for Fedran's monorepo) that is opinionated but also seems to fit a lot with my thing.

That was a fun few days (four) to migrate the Fedran site over, and it fixed some problems I was having in general (read, I actually have the CLI integrated into the build now).

The success means I want to change over my other infrastructure repository which drives most of my home lab but… I need to say no to that until after August.

But, one thing that really stuck with me is that it hits the same buttons that I found when I was reading about Buck2 and Bazel, two other monorepo systems that fit well with a lot of professional and personal development going on in my life. Plus, I just like typing this:

std //cli/apps/default:run

std //website/app/default:build

And have it do all the right things including getting all the dependencies. I can easily see that tying in my aggregate books, timelines, and other stuff nicely into a single unified system.

Future Plans

I like building Fedran's infrastructure as much as writing. More so lately because I burned out a little bit and need to work on finishing Nor Curse Be Found and Second-Hand Dresses.

Speaking of Dresses, I'm down to a single editing question but that ties into a different program I need to work on, rebuilding my calendar system so it can play with C# and Rust. I've been using my Typescript based on for about a decade now and I have a huge list of things that need to work.

Who knew an editor question of “what day is it” would trigger that paralysis, but I want to solve it because one of the things I lost with my migration to Nitride was my (awesome) timeline functionality. I was proud of that and want it back.

The calendar is also critical for my constraint system. Because I'm coordinating across multiple process, and I had a terrible time with Raging Alone's documentation process, I realized I need to have something that lets me say things like “this character is eight years older than this one” or “this happens two months after the events of this story”.

(I also need a map, but I'm trying really hard not to think about that.)

Nor Curse Be Found

Speaking of Nor Curse Be Found. I hit a little bit of a writer's block on the current chapter and I'm not happy with it. But work and my writing obligations are making it difficult to focus on it, which is why I haven't done much with it lately. I can only have so many writing projects going on and writing that pays bills has a higher priority over writing that makes me feel good.

While Reddit was imploding, I stumbled onto a post in /r/fantasy asking for resident authors. Since I've been commenting off and one for a few years (and being good about not saying "read my book!"), I decided to throw my hat into the ring. Not sure what would come of it, but it wouldn't hurt since right now Flight of the Scions currently has one review on Amazon and zero reviews on Goodreads (and just about nowhere else).

Self-Published Fantasy Blog Off

Over the years, I've withdrawn from both Facebook and Twitter. One of the conquences of this action is that I don't know about a lot of things going on, but one thing I've been occasionally checking has been the Self-Published Fantasy Blog-Off.

Sand and Blood was a semi-finalist in the first one and I felt I was so close to getting to at least finalist. I lost the finalist for The Thief Who Pulled on Trouble's Braids who would eventually be the winner.

But I was hoping that maybe Flight of the Scions could finally have a chance. So I was waiting for the announcement from Mark Lawrence but it never came. Last night, when I decided to check one more time, I saw the reason.

SPFBO 9 is closed. We reached the 300 entries in 41 minutes.

Without watching for when the next round comes around, I might have to just hang up my hat on trying to make another SPFBO and just chalk it up to an “almost win” with Blood.

What's Next?

Wel, I need to put down the programming hat for a while because we're coming up to the end of the month and I need to write another ten thousand words on my end of month obligation. The beginning of the month will be my commissioned novel (hoping to get 10-20k words by the 15th).

That means my next “fun break” is in about three weeks. I need to work on the Typewriter Press website because I published another book for an author last week and I need to do my due diligence to help them sell as many copies as possible.

NixOS, Gitea/Forgejo, and Catppuccin

2023-05-13T05:00:00Z

I spent a couple hours today figuring out how to get the official Catppuccin theme for Gitea and Forgejo integrated into my NixOS environment. Originally, I had taken the user theme for Codeberg, tweaked and cleaned it up for my server, and then used that. But, I decided to go a little further and submitted a request to make it official. Someone then made an entirely new package and made it the official version. After a little pouting, I decided that I'd rather have someone else maintain it so I can focus on other things.

The first part was pulling it down. Originally, I thought I would grab it via the Nix flake, but that ended up being too difficult. After fumbling around, I came up with using fetchzip (which apparently does tarballs also) because the release files don't have a top-level directory (all the CSS are on the root).

inputs @ {
  config,
  pkgs,
  lib,
  ...
}: let
  theme = pkgs.fetchzip {
    url = "https://github.com/catppuccin/gitea/releases/download/v0.2.1/catppuccin-gitea.tar.gz";
    sha256 = "sha256-HqVLW58lKPn81p3gTSjzkACHSBbmqPqeobAlJMubb8Y=";
    stripRoot = false;
  };
in {}

From there, I needed to set it up. There aren't a lot of good ways (that I've found) with setting up data structures for services on NixOS, so I set everything up in the preStart of the service. The stanza below gets all the files into the right place.

systemd.services.gitea = {
  preStart = lib.mkAfter ''
    rm -rf ${config.services.gitea.stateDir}/custom/public
    mkdir -p ${config.services.gitea.stateDir}/custom/public
    ln -sf ${theme} ${config.services.gitea.stateDir}/custom/public/css
  '';
};

The last bit is to tell Gitea about the themes. I considered writing them all down, but there are quite a few of them (four basic types plus accents for each one), so I decided to dynamically build the list from the input.

services.gitea = {
  settings = {
    ui = {
      THEMES =
        builtins.concatStringsSep
        ","
        (["auto"]
          ++ (map (name: lib.removePrefix "theme-" (lib.removeSuffix ".css" name))
            (builtins.attrNames (builtins.readDir theme))));
      DEFAULT_THEME = "catppuccin-mocha-blue";
    };
  };
};

Thanks to ottidmes from the #nix:nixos.org matrix room for all the help.

In other news, I switched from Gitea to Forgejo on src.mfgames.com. There wasn't a compelling reason other than I feel that I fit the philosophy more. I also disabled registrations on that server for now. If you want to send me an update, just use my contact page and send directly.

Rumors of my dead garden links were exaggerated

2023-03-15T05:00:00Z

Whenever tax season comes around, I start to talk about Exit Planning because it is “death and taxes.” I'm pretty proud of it as a project and as a living documentation of something I think a lot of us are missing, how to prepare for our deaths (or incapacitation).

What I didn't realize was that when I put in the dead link removal code for the package management to let me write posts ahead of time, I also broke all the links in the gardens.

That cornucopia of links that seems to represent my thought posts had disappeared so everyone just so a bland introduction page without knowing there were two dozen links interspersed inside it.

Thankfully, Benjamin Hollon on Mastodon noted that all my links were dead, so I finished tracking it down and fixed it. Thank you!

Now, to move that HTML dead link code into Nitride proper so I can use it with all my sites.

Package Management - Dependencies

2023-02-13T06:00:00Z

While packages can be self-contained, usually they have dependencies with other packages or even frameworks to avoid duplicating code. Without dependencies, packages would be forced to contain common logic that other packages share, and that code may become stale or not updated as frequently introducing potential bugs.

Series

2023-02-07 Package Management - Introduction - The beginning of a short series as I write down my ideas of an ideal package system with the future goal of writing a generic system called Bakfu.
2023-02-08 Package Management - Versions - A short discussion on packages and their versions.
2023-02-12 Package Management - Identifiers - A short discussion on how to identify packages.
2023-02-13 Package Management - Dependencies - A review of different styles of package dependencies, including across different ecosystems, and then an attempt to consolidate them into a single unified system.
2023-09-20 Package Management - Identifiers 2 - Using URNs to identify packages and some re-thinking of concepts.
2023-11-30 Package Management - Formats and Registries - Thoughts on setting up formats and layering registries for those formats on top of each other.

Simple Dependencies

For most systems, dependencies are very simple: this package requires these packages in this range. When working with Bakfu, this is almost identical to the package identifier but without the bakfu:version and instead a range of values that are acceptable.

Using MfGames.Nitride?version=0.15.1 as an example, here is a fragment of what that could look like:

{
    // Basically this is "bakfu:nuget:MfGames.Nitride?framework=net6.0"
    type: "nuget",
    id: "MfGames.Nitride",
    attributes: {
        "framework": "net6.0",
    },
    content: {
        version: "0.15.1",
        dependencies: [
            "bakfu:nuget:Autofac?framework=net6.0": "[6.4.0, 7.0.0)",
            "bakfu:nuget:FluentValidation?framework=netstandard2.1": "[11.2.1, 12.0.0)",
            "bakfu:nuget:MfGames.Gallium?framework=net6.0": "[0.4.0, 0.5.0)",
        ],
    },
}

Replacements and Substitutions

While the above works in most cases, there are situations when it doesn't. Probably the easiest one to explain in in Minetest with multiple mods that replace the built-in beds module: TenPlus1's and SorceryKid's. In all three of the cases, they are installed as beds which is used for the dependency, but they are distinguished enough for our package system despite the same name.

bakfu:minetest:beds
bakfu:minetest:TenPlus1/beds
bakfu:minetest:sorcerykid/beds

In the C# world, I've encountered this with the BouncyCastle libraries. In specific, we have:

Both of these libraries have a 1.8.9 version and both of them have exactly the same DLL once it is build, BouncyCastle.dll. In the past, this has caused us a lot of problem because while the two versions are API compatible (they have the same application binary interface, or ABI), they also overlap each other and have different packaging systems (one is only for .NET Framework, portable is more universally usable and currently maintained).

Likewise, if someone abandoned a package in a fit of anger, say the leftpad drama in the JavaScript, a strict package identifier means a replacement cannot be find. The same thing happens if a project maintainer dies without planning for a successor, there is a schism in the community causing a soft- or even hard-fork, or even someone using a Git repository or local copy to add some customization they need. Regardless, there are reasonable reasons for having two packages that proclaim to be the same version.

To handle this, packages should list the aliases of what they provide (including themselves if needed) and that is what dependent packages would use. Using the BouncyCastle for an example:

// BouncyCastle
{
    content: {
        id: "BouncyCastle",
        version: "1.8.9",
        attributes: {
            "framework": "net",
        },
        provides: [ "bakfu:nuget:BouncyCastle?bakfu:version=1.8.9&framework=net" ],
    },
}

// Portable.BouncyCastle
{
    content: {
        id: "Portable.BouncyCastle",
        version: "1.8.9",
        attributes: {
            "framework": "net40",
        },
        provides: [
            "bakfu:nuget:Portable.BouncyCastle?bakfu:version=1.8.9&framework=net40",
            "bakfu:nuget:BouncyCastle?bakfu:version=1.8.9&framework=net",
        ],
    },
}

// A third package
{
    content: {
        id: "Something.Else",
        version: "1.0.0",
        requires: {
            "bakfu:nuget:BouncyCastle?framework=net": "[1.8.9,2.0.0)",
        },
        // A missing provides would just produce the identity package
        // in this case, it would imply:
        // provides: { "bakfu.nuget:Something.Else?bakfu:version=1.0.0" }
    },
}

In the above example, if a module doesn't have a provides element, then we could assume that it is just the identity version. “Something.Else” version 1.0.0 becomes bakfu.nuget:Something.Else?version=1.0.0.

Publish and Subscribe

The above example doesn't handle optional dependencies. For example, Minetest modules can check for the presence of other modules and then hook up to them. Atom (one of my favorite editors that was sadly killed) had the idea of services where a package could provide a service (that had a version) and other packages could “consume” them. Using a fragment of my own spell-check, it looked like this:

// from spell-check
"consumedServices": {
  "spell-check": {
    "versions": {
      "^1.0.0": "consumeSpellCheckers"
    }
  }
},

// from spell-check-project
"providedServices": {
  "spell-check": {
    "versions": {
      "1.0.0": "provideSpellCheck"
    }
  }
},

In the above case, the editor would call use the callback method for every package that “provided” a service which would return an opaque object, which would then be handed to spell-check consumed callback (consumeSpellCheckers) as an array. That allowed optional dependencies by publishing a service and then having other packages subscribe to them.

Even if the publisher (spell-check in the above example) would get every subscriber, then doesn't mean that the publisher then could call something back into every subscriber that would let you wire the two together to allow for a reverse- or even bi-directional flow of data.

Since this also has system-specific elements, we use the same attributes construct to give flexibility to extend outside of Bakfu.

{
    content: {
        id: "spell-check",
        version: "0.77.1",
        publish: {
            "bakfu:atom-service:spell-check": {
                accept: "[1.0.0,2.0.0)",
                attributes: {
                    callback: "consumeSpellCheckers",
                },
            },
        },
    },
}

{
    content: {
        id: "spell-check-project",
        version: "0.7.2",
        subscribe: [
            "bakfu:atom-service:spell-check": {
                accept: "[1.0.0,2.0.0)",
                attributes: {
                    callback: "provideSpellCheck",
                },
            },
        ],
    },
}

How the application handles those services isn't in the domain of Bakfu, just the gathering and providing the information that links the two. It would also be based on the application if there can be multiple conflicting publishers (can two packages both publish the same service and version?) or not.

Categorizing Dependencies

The last major topic to cover for this post is how to categorize dependencies. NPM and C# both have the concept of development and production dependencies. NPM is a little easier to work with:

// package.json
{
    dependencies: {
        "package-1": "1.2.3",
    },
    devDependencies: {
        "package-2": "4.5.6",
    },
}

Rust is more complicated because enabling a feature also changes the dependencies of the package. Many crates don't include serialization, but if you add the serde feature, it would include serde and any other requirements. In effect, this is an optional dependency but determined at build item.

It also changes how we organize dependencies above by adding an extra layer inside the requires element that provides an application-specific set of keys. In Rust, this may also include every feature as a separate element of dependencies.

(I could have started with this, but I wanted to build the blocks and work out the ideas on a single dependency before getting into the more complex situations.)

{
    content: {
        id: "Something.Else",
        version: "1.0.0",
        requires: {
            production: {
                "bakfu:nuget:BouncyCastle?framework=net": "[1.8.9,2.0.0)",
            },
            development: {
                "bakfu:nuget:DoesNotValidateInput": "[1.0.0,2.0.0)",
            },
        },
    },
}

Another reason to have this categorization is to allow us to identify what packages are vulnerable in an intelligent mannner instead of just assuming any and all vulnerabilities are important. In other words, if you trust your developers not to use a DDoSing regular expression in their code, you don't have to patch that code.

Expanding Dependency References

One of the areas I'm not sure about is now to describe dependences and provides. In the above examples, I'm using the URI format but my gut feeling is that it should be the more verbose form to allow for extensions.

The previous example would then look like this:

{
    content: {
        id: "Something.Else",
        version: "1.0.0",
        requires: {
            production: [
                {
                    type: "nuget",
                    id: "BouncyCastle",
                    accept: "[1.8.9,2.0.0)",
                    attributes: {
                        framework: "net",
                    },
                },
            ],
            development: [
                {
                    type: "nuget",
                    id: "DoesNotValidateInput",
                    accept: "[1.0.0,2.0.0)",
                },
            },
        },
    },
}

I'm just not entirely sure about which one is the “best” but I'm also trying to work out ideas for a complete system, so I will probably use the URI unless the expanded makes sense. Ultimately, I really won't know until I've gotten through all the analysis and ideas before I can say one way or the other.

(Though, I do lean toward more strict schema which means that the expanded form will probably be the final one.)

I'm also not sure if the Bakfu format really needs to have bakfu: in front of identifiers. I mean, it is inside the “bakfu” file.

Package Management - Identifiers

2023-02-12T06:00:00Z

The other really big topic when it comes to package is how to identify them. This is a lot more complicated because I view packages in a polyglot manner instead of a single foreign ecosystem.

Series

2023-02-07 Package Management - Introduction - The beginning of a short series as I write down my ideas of an ideal package system with the future goal of writing a generic system called Bakfu.
2023-02-08 Package Management - Versions - A short discussion on packages and their versions.
2023-02-12 Package Management - Identifiers - A short discussion on how to identify packages.
2023-02-13 Package Management - Dependencies - A review of different styles of package dependencies, including across different ecosystems, and then an attempt to consolidate them into a single unified system.
2023-09-20 Package Management - Identifiers 2 - Using URNs to identify packages and some re-thinking of concepts.
2023-11-30 Package Management - Formats and Registries - Thoughts on setting up formats and layering registries for those formats on top of each other.

Previous Updates

While I got to thinking (this is a living series), I didn't like the upgrade format in the versions post of this series. In there, I suggested this for Bakfu:

{
    content: {
        version: "1.10.0",
        majorUpdate: "[1.10.0, 1.15.0)",
        minorUpdate: "[1.10.0, 1.11.0)",
    },
    package: {
        version: "2.0.0",
    }
}

I think putting the update rules in their own category makes sense since they related to each other but also to handle adding other rules as needed in there. I thought about putting everything in a version, but then I would have {version: { current: "1.2.3" }} which didn't feel right.

We also should have the package format version.

{
    bakfu: {
        version: "0.0.1",
    },
    content: {
        version: "1.10.0",
        update: {
            major: "[1.10.0, 1.15.0)",
            minor: "[1.10.0, 1.11.0)",
        },
    },
    package: {
        version: "2.0.0",
    }
}

Identifier Schemas

A package identifier is one of those things that varies greatly between different languages:

NPM: markdowny, @mfgames-writing/format
NuGet: MfGames.Nitride, System.Collections.Generic
Minetest: beds, moreores
Java: com.mfgames.ruby (an old game of mine)
Go: github.com/callicoder/packer/numbers
Cargo: serde
Atom: spell-check

That gives a pretty good overview of most languages that use packages. I don't think there are many other ways of referencing them. In addition to the ones above, I want to be able to allow anyone to create an arbitrary package ecosystem because I don't need to have a game using my calendar's packages (unless it wants to) but they are distinct.

Package Groups and Organizations

I will be up front, I don't like flat package systems such as Node and Cargo. They have a significant problem with name squatting (where someone “reserves” a package name that is ideal) and also because many of the packages are generic (Minetest's beds or Node's time) even if they are as capable.

If you notice, almost all of my packages start with mfgames or MfGames. I do that to avoid conflicts with other packages. I thought I wouldn't for Nitride but when I finally created the packages, someone had put one up on nuget.org and I promptly turned it back to MfGames.Nitride. At this point, if someone is using MfGames. in their packages, it's intentional (and probably malicious).

Scoped packages are better, more so if there is a way of controlling who can use that scope. On nuget.org, you can reserve a prefix which is how they prevent someone from uploading a Microsoft.HackYourComputer package. In theory, I could do the same with MfGames but it requires some dependencies that I don't have lying around (cash).

That said, I don't like NPM's @organization because they also use @1.3.2 for versions. So, then you have package and version identifiers as @mfgames-writing/format@3.4.0. It isn't “pretty” and I am frequently driven by what I think looks good. As such, I'm inclined to avoid the @ whenever possible.

Polyglot Identifiers

pol·y·glot /ˈpälēˌɡlät/ -> adj. knowing or using several languages.

When working with code, not only do we use different packages, we use packages that require entirely different systems. Probably one of the most common one for me is the bane of my JavaScript and TypeScript development: node-gyp. Node-gyp requires a C compiler and Python to run, but it is for Node packages. So, that means that if we really want to identify a package (and its dependencies, a later post), we need to be able to address all of those dependencies in the system addressing scheme.

Uniform Resource Identifiers (URIs)

Since I want to be able to identify packages by something that jumps languages, it seems like a perfect use of a URI. In short, a URI looks like this:

URI = scheme “:” ["//" authority] path ["?" query] ["#" fragment]

The most common one we see are URLs:

https://fedran.com/flight-of-the-scions/

In the above example, https is the scheme, //fedran.com is the authority, and flight-of-the-scions/ is the path. If we take the same idea, we could create a pseudo version of package identifiers that use the same thing.

(Side note, Humanizer is an awesome package for C# and everything Charm.sh is very pretty. Markdowny is my own tool for working with Markdown + YAML files, so I think it is fantastic.)

This works for well-known package systems, but what about arbitrary ones? Say I create something for Author Intrusion, I could use authorintrusion:spell-check but that would require registering a URI if I want to avoid conflict. A more efficient way (that I also will help with some future ideas) is to prefix them with bakfu: instead and then allow domain or scoped registrations for non-well-known ecosystems.

bakfu:npm:markdown
bakfu:go:github.com/charmbracelet/lipgloss
bakfu:authorintrusion.com/spell-check (I really should do something about that website.)

Well-Known Systems and Aliases

Sadly, this does assume a centralized repository for the well-known (e.g., npm, go, nuget) packages. It also leans into DNS instead of using other addressable schemas or pet names, I could also use dotnet.microsoft.com for nuget (since that is the general packaging system), nodejs.org, and go.dev for those languages.

If I do go with the well-known, I would also have to establish a extension process that would allow communities to introduce new aliases or “register” their own to avoid conflict. While that isn't something I'd want to do, if this entire idea goes beyond me, it would be the first thing I would establish because I like to play well with others.

I would also call it BEEP - Bakfu Emerging Extension Process so then I could have BEEP-0000 be the first one.

Qualified Identifiers

Using the URI, we can use URL-style query parameters to create a key/value pair for additional details about the package such as Cargo's features, .NET's frameworks, or whatever variants are important to include.

For ones used by Bakfu, using a XML-style prefix would be more clear than either having a name collision (version= could be common) or an arbitrary character prefix (_). In these cases, bakfu:version would be useful and allow for further extensions.

For keys that need multiple, such as crate's features, the key would be suffixed by [] to indicate an array (“Ruby style” is what I've heard this called).

Rust has both platforms and featuers:

bakfu:cargo:serde?bakfu:version=1.0.152&feature[]=derive&feature[]=rc
bakfu:cargo:serde?bakfu:version=1.0.152&platform=x86_64-unknown-linux-gnu

NuGet has frameworks:

bakfu:nuget:Autofac?bakfu:version=6.5.0&framework=netstandard2.0
bakfu:nuget:Autofac?bakfu:version=6.5.0&framework=net6.0

Given that, and the future discussion on dependencies, I think it would make sense to make it a query parameter instead of using the # fragment. Also, some foreign ecosystems don't have versions (Minetest, at least one of the Python systems) which means even a version is optional.

Expanded Form

To simplify coding, the above URL could easily be broken down into a standard JSON structure for less condensed format with the “bakfu:” attributes pulled up a level.

{
    // bakfu:cargo:serde?bakfu:version=1.0.152&feature[]=derive&feature[]=rc&platform=x86_64-unknown-linux-gnu

    type: "cargo",
    id: "serde",
    version: "1.0.152",
    attributes: {
        "platform": "x86_64-unknown-linux-gnu",
        "features": ["derive", "rc"],
    },
}

At the moment, I'm going to use both formats but I lean toward the more expanded form in actual files with the condensed URI form when trying to explain things. I'm sure I will bounce back and forth between something that feels “right”. You may also notice that I spell out most things because I don't like abbreviations while communicating concepts.

Distributed Packages

This is for a future topic, but I am planning on addressing distributed packages that don't encourage using only a single source for the data. So, the URI above only addresses the framework-level components that identify a distinct package irregardless if you find it on nuget.org, npmjs.com, gitlab.com, or even src.mfgames.com.

Package Management - Versions

2023-02-08T06:00:00Z

The start of almost every package journey starts with versions. This is how one distinguishes between different copies and hopefully makes sense of when to get the latest version verses only updating a few or not updating at all.

Series

2023-02-07 Package Management - Introduction - The beginning of a short series as I write down my ideas of an ideal package system with the future goal of writing a generic system called Bakfu.
2023-02-08 Package Management - Versions - A short discussion on packages and their versions.
2023-02-12 Package Management - Identifiers - A short discussion on how to identify packages.
2023-02-13 Package Management - Dependencies - A review of different styles of package dependencies, including across different ecosystems, and then an attempt to consolidate them into a single unified system.
2023-09-20 Package Management - Identifiers 2 - Using URNs to identify packages and some re-thinking of concepts.
2023-11-30 Package Management - Formats and Registries - Thoughts on setting up formats and layering registries for those formats on top of each other.

The Semantic Dream

These days, almost every package system proclaims that they use semantic versions and write the infrastructure as such. I love semantic versions with a passion, including putting them in my novels and stories. They are relatively clear and concise, and easy for me to understand: breaking-things.adding-things.fixing-things.

The problem is, semantic versions are hard to manage when it comes to packages and dependencies, not to mention the business practices.

Romantic Versions

The biggest one are the ones who won't use it. One of the library I use during my day job is EasyQuery because they have a .NET and a JS library and give me a lot of flexibility in creating dynamic queries. They also are responsive to bug issues. What they don't do is follow semantic versions. They use romantic versions with every x.y version being a breaking change from the previous one. This applies to their code on NPM and NuGet.

It is easy to manage, but you have to know that you can't go from 7.0.2 to 7.2.0 without major work, though semantic version says that it should have been a safe change.

My day job product is also romantically versioned, so I deal with this on a quarterly basis.

Dependency Changes

Another example is SlimMessageBus.Host. In version 1.14.1, it requires “.netstandard2.0”. In 1.15.0, it is “.netstandard2.1”. While this seems like safe change, it was a minor version bump that made it impossible for us to update because our product is still bound to “.netstandard2.0” until we can get off WebForms.

Version Ranges

Since it is important, there are also a number of ways of communicating version ranges. Node uses the ^ for semantic versions and ~ for “only patch updates” (basically what EasyQuery uses).

NuGet uses a different system such as [1.0.0, 1.15.0) with the [] meaning inclusive and () meaning exclusive. This is my preferred system, mainly because it hearkens back to my computer science days and allows a mid-version break such as SlimMessageBus. In that regard, it is more flexible because ^ and ~ can translate directly into the verison range.

Being Explicit

I haven't gotten to dependencies yet, but the developer knows when something is safe update a package verses when it is not. EasyQuery knows that a patch is the breaking point, it would be better that they be able to communicate that as part of their metadata instead of requiring a customer support call. In effect, using the first two semantic version components (major and minor), EasyQuery 7.0.0 would be:

{
    version: "7.0.0",
    majorUpdate: "[7.0.0, 7.1.0)",
    minorUpdate: "[7.0.0, 7.1.1)", // The third number is their minor versions
}

On the other hand, SlimMessageBus would use (using an older version for illustrative purposes):

{
    version: "1.10.0",
    majorUpdate: "[1.10.0, 1.15.0)",
    minorUpdate: "[1.10.0, 1.11.0)",
}

I'm assuming that anything more precise is a patch and is safe to update. I'm also think Bakfu should lean heavily on well-defined undefined values. So if majorUpdate, then assume semantic version. If minorUpdate is missing, then use majorUpdate (which may be assumed).

Package Versions

The last bit is one that I've encountered a number of times while packaging books, but also while I was doing Debian packages. The version of the package should be independent of the content. Using the above example, if SlimMessageBus realized they had made a breaking change, they should be able to update the 1.14.0 version of their package to indicate that 1.15.0 is a major breaking change.

Today, their only choice would be to remove the 1.15.0 (and any later) version and create a 2.0.0 version. This can be a lot of work because most likely someone realized it after there were a number of updates already out in the field.

Instead, if the package version was decoupled from the content version, then it would be possible to retroactively update critical information such as dependencies and safe upgrade ranges. Using SlimMessageBus again:

{
    package: {
        version: "1.0.0",
    },
    content: {
        version: "1.10.0",
    },
}

After they realize the breaking change:

{
    package: {
        version: "2.0.0",
    },
    content: {
        version: "1.10.0",
        majorUpdate: "[1.10.0, 1.15.0)",
        minorUpdate: "[1.10.0, 1.11.0)",
    },
}

Package Management - Introduction

2023-02-07T06:00:00Z

As I'm working through one project, I'm frequently planning the next. I noticed that a number of my future ideas, fantasies as you will, all have one thing in common: packages. It doesn't if it is a game idea or working on programming calendars or Author Intrusion, they all need some form of package management.

Given that and the rule of three, it makes sense to sit back and working that problem independent of the individual projects. Some people call this procrastinations. I like to call it composition. After a few months of rolling it around in my head, plus a bit of thoughts from Drew DeVault's various posts on the topic, I figured I'd codify my ideas and see what I can do to create something I'm happy with in the future.

Series

This is going to be a series of posts, but I have no idea of how fast I'll be writing them out or if it will pan out, but this is a list of the related posts. I want to work out my ideas, maybe have a few conversations, and then start to move to more technical concepts.

2023-02-07 Package Management - Introduction - The beginning of a short series as I write down my ideas of an ideal package system with the future goal of writing a generic system called Bakfu.
2023-02-08 Package Management - Versions - A short discussion on packages and their versions.
2023-02-12 Package Management - Identifiers - A short discussion on how to identify packages.
2023-02-13 Package Management - Dependencies - A review of different styles of package dependencies, including across different ecosystems, and then an attempt to consolidate them into a single unified system.
2023-09-20 Package Management - Identifiers 2 - Using URNs to identify packages and some re-thinking of concepts.
2023-11-30 Package Management - Formats and Registries - Thoughts on setting up formats and layering registries for those formats on top of each other.

Package Systems

Packages are awesome because they allow over developers provide components, they split apart logic into smaller chunks which make it easier to understand, and they can have different update paths instead of needing a single monolithic deployment. This also means, I'm more likely to be able to focus on a small part instead of having to “swap in” an entire project. That is one thing that has made Nitride a lot more enjoyable to work with.

At their core idea, a package is just a “blob” of something along with metadata. For purposes of this series, I don't care what is in the blob. It could be C# code, a WASM binary, or Rust source. The thing I've gotten hung up is that we have a ton of different package systems (e.g., LISP modules, Python packages, NPM, Minetest, Debian packages) and each one was written for a specific use and has different rules. Some are nicer on the disk, others support signed code, and some have methods to avoid squatters. Every community wants to write their own for their specific uses, but to me it feels like we are reinventing the wheel so many times when we could be focusing on more fun things (unless packaging systems is your idea of fun, it is mine).

(Yes, I'm aware that I want to write my own for my own specific purpose, but that's going to happen anyways. It's just a matter of what framework I write it in.)

Terminology

Since I'm thinking about doing a series of posts, here are some terms I plan on using:

Application - Whatever needs a package, be it a game, library, or utility.
Bakfu - My hypothetical system for managing packages. This is the Lojban word for “package” and seems like a decent handle for what I'm working for.
Foreign - Another packaging system.
Package Ecosystem - All the servers, tools, and the packages themselves for a given type of packaging. This includes bakfu and foreign packaging ecosystems.
Package Server - The servers used to query and distribute packages.
Package - A specific type of package. This will have bakfu, application, or foreign as an adjective when necessary.
Package Content - The “thing” being packaged. This can be a set of files, a word list, assets for a game, or even a foreign package. For purposes of these posts, this is a generic “blob” of data that is specific to an application.

Bakfu

I want something generic. More specifically, I want to find what I consider are the “best” parts of the various systems I've worked with, solve problems that I've experienced, and maybe come up with a set of libraries and utilities that let me implement a packaging ecosystem that only requires me to override the application-specific components while delegating the rest of the ecosystem to bakfu.

I have no intent on this being a “standard” because everyone likes to reinvent the wheel. Everyone has their own idea of what is the “right” way of handling packages, even when there is a defacto standard already implemented.

That isn't to say this is only going to be for me. Like most of my work, I'm hoping to create something generic enough that others can use it for any purpose and in the languages of their choice and still get some of the benefits I envision.

A Week of Dependencies plus Nor Curse Be Found 13

2023-01-29T06:00:00Z

I'm trying to get get back to the one-week iterations/sprints for my personal projects. As one of my goals for the year, I want to write at least one short story or a chapter a month. “At least” being the goal since I want to write more, but writing Fedran is still a struggle because of burnout, entanglement, and just life.

Markdowny

So I dedicated this week to writing chapter thirteen of Nor Curse Be Found, but in the process of setting it up, I finally hit the point where bugs in Markdowny were too much for me. The main one is that I used the wrong shebang in the scripts but didn't realize it until I switched to NixOs. The key part is that I need to use /usr/bin/env bash for all my shebangs instead of assuming the path to bash.

# !/usr/bin/env bash

So that meant I needed to go into that project to fix it (and add a few features I've been wanting). But that was so old that I spent a few days bringing it up to fix my new project layout so I had a consistent environment. While doing that, I ended up on yet another tangent to create a script that could create my script files. (Not sure where to store that, to be honest.)

But then I finished the script to get the layout working to get Markdown working. After another day or so to coerce markdowny into Woodpecker CI, I was finally ready to actual write a feature I've been needing for a while.

New Feature

To submit entries to my writing group, I need to post a “events so far” document because it could be weeks or months between submissions. I use markdowny to do that with the list command:

$ markdowny list chapters/*.md | head -n 5
1. Always Moving: While Linsan waits for her mother to come home, she bounces around on the furniture and talks to her father. She announces that she has named a violin her father is making Palisis and learns that the violin is for her father's first wife who got married to her mother's best friend.

2. Early Lessons: Years later, Linsan is learning how to play the violin from her father. The lesson is interrupted when Dukan, her father's best friend and manager for the business, visits in a panic to tell him that the family's workshop in the valley is on fire.

3. Home Early: Unable to visit the burnt remains of the family's workshop, Linsan comes home to find her father work in depression. He had given up working on instruments and switched to writing articles about music. She goes into the attic to put some books away and finds Palisis in a corner, returned after Marin's death. She plays it, but then finds out that no one had ever played it before.

I needed a bit more details, so I decided to expand it to use Handlebars to give me a template for the chapters and to access all of the components inside the YAML header.

$ markdowny list ~/src/fedran/sources/allegro/chapters/*.md --template='- Chapter {{_number}} - {{{title}}} => {{{summary}}}' --trim-whitespace | head -n 5
- Chapter 1 - Always Moving => While Linsan waits for her mother to come home, she bounces around on the furniture and talks to her father. She announces that she has named a violin her father is making Palisis and learns that the violin is for her father's first wife who got married to her mother's best friend.
- Chapter 2 - Early Lessons => Years later, Linsan is learning how to play the violin from her father. The lesson is interrupted when Dukan, her father's best friend and manager for the business, visits in a panic to tell him that the family's workshop in the valley is on fire.
- Chapter 3 - Home Early => Unable to visit the burnt remains of the family's workshop, Linsan comes home to find her father work in depression. He had given up working on instruments and switched to writing articles about music. She goes into the attic to put some books away and finds Palisis in a corner, returned after Marin's death. She plays it, but then finds out that no one had ever played it before.
- Chapter 4 - Solace in Memories => As Linsan frequently did, she visited the family's ruins after school. The spot gave her peace despite everything they had lost. However, a bully from school, Dukan's daughter Brook, follows after her and they fight. During the brawl, they both manifest their powers: Linsan with music and Brook with concussion powers.
- Chapter 5 - Bitter Partings => Linsan comes limping home after her fight with Brook. Her parents are surprised she is there, but then Dukan and Brook show up. Dukan has his daughter apologizes and then offers to send money to the Sterlig's. Linsan's father tries to refuse it, but Dukan phrases it as helping Linsan and they accept.

It might not be much, but it was something I've been looking for. It also means that I can include things like POV, when it happened, or other details in the list to help work timelines, locations, and the like. Or use it as a simple index page generator for HTML and Gemini pages, if someone found a use for that.

Tables

Of course, there is also a table approach to the same thing:

markdowny table chapters/*.md --fields _basename title when.start locations.primary | head -n 5

_basename	title	when.start	locations.primary
chapter-01.md	Rutejìmo	1471/3/28 MTR 4::22	Shimusogo Valley
chapter-02.md	Confession	1471/3/28 MTR 4::75	Shimusogo Valley
chapter-03.md	Morning	1471/3/28 MTR 11::71	Shimusogo Valley

Nor Curse Be Found

Despite all that, I got another chapter of Nor Curse Be Found written this afternoon. It isn't polished as I hope, I'm still struggling with the “feel” for the chapter and how the prince responds to things, but that is also something I have to finish to really understand and go back to edit.

Overall, it was a good day (it took me three days to write the chapter).

Faking Time Magic with Nitride

2023-01-21T06:00:00Z

It's been a while since I've talked about MfGames.Nitride and I thought I would do a short post about working with time-sensitive posts. In most cases, this is dating blob posts but it can also be how I dole out weekly chapters while building up a buffer while I enjoy some down time.

NodaTime

All the time elements in Nitride use NodaTime instead of the base class library's DateTime. Part of this is because it predates the advent of LocalDate but also because NodaTime has good abstractions around providing the time for reliable manipulations of time, an obsession with correctness, and also a deep understanding of the complexities of temporal elements.

None of these really matter, but I like the library and I felt it was a solid base. This is also one reason why Nitride is opinionated.

Getting Started

To use the temporal library, you need to include the NuGet package into your project and then either inject the module into the class or use the extension method.

NitrideBuilder builder = new NitrideBuilder(args)
    .UseTemporal(
        config =>
        {
            config
                .WithDateTimeZone("America/Chicago")
                .WithDateOptionCommandLineOption();
        });

The configuration is optional, but this shows the two most common options. The first sets the time zone for all the local dates to handle when your site generates even if your server or build machine is UTC or in a different time zone.

The other adds a --date YYYY-MM-DD option to the command line to let you choose the “now” that the site is running, such as to preview next week's post or see how everything will turn out. If the --date is not provided or the command line option isn't used, then “now” will be now. Well, then, but it was now then, but if you run it now, then it will now. Well, then. (Spaceballs reference.)

dotnet run -- build --date 2023-01-25

The Basics

Even with the module, nothing more is going to happen than additional logging line in the output. Because this is an Entity-Component-System, adding time to an Entity is as simple as setting the instant.

Instant instant;

var entity = new Entity().Set(instant);

Of course, this only applies to a single entry, so there are two operations included that will mass assign Instant components to entities. The first is SetInstantFromComponent. This gets a LocalDate, DateTime, DateTimeOffset, or Instant from the entity (Model.Date in this example) and returns it. If this returns a null, then no instant will be assigned.

public class Model
{
    public string Access { get; set; } = "public";
    public DateTime? Date { get; set; }
}

public Pipeline : PipelineBase
{
    private SetInstantFromComponent<PageModel> op;

    public Pipeline(
        SetInstantFromComponent<PageModel> op)
    {
        this.op = op
        .WithGetDateTimeObject(entity => entity.Get<Model>().Date);
    }

    public override IAsyncEnumerable<Entity> RunAsync(
        IEnumerable<Entity> entities,
        CancellationToken cancellationToken = default)
    {
        return entities
            .Run(op)
            .ToList()
            .ToAsyncEnumerable();
    }
}

As a basic functionality, it is pretty simple. However, the most common way I need to assign an Instant are my blog posts which all have the following pattern:

./posts/2023-01-21/faking-time-magic-with-nitride/index.md

To handle this common condition, there is a second operation: SetInstantFromPath.

SetInstantFromPath op;

return entities
    .Run(op)
    .ToList()
    .ToAsyncEnumerable();

This one doesn't require any customization to it because it uses a regular expression with named groups to calculate the three components. This is the same as setting the regular expression:

return entities
    .Run(op.WithPathRegex(@"(?<year>\d{4})[/-](?<month>\d{2})[/-](?<day>\d{2})"))
    .ToList()
    .ToAsyncEnumerable();

Obviously the default is ISO date time, but I'm not going to support anything besides that in the defaults since you can writ a custom regular expression with named groups to handle any arbitrary format. The default also handles 2028/01-21-faking-time-magic-with-nitride.md as well as my normal format. In the end, the Entity has an Instant component.

Using Instants

Of course, setting an Instant requires it to be used somewhere. Since it is a component, it is relatively easy to get the latest X posts from a list of entities.

var latest = entities
    .WhereEntity<Instant>()
    .OrderByDescening(entity => entity.Get<Instant>())
    .Take(x)
    .ToList();

This is used to create feeds, but that is a topic for later since generating Atom feeds needs some more attention before I'm comfortable with it.

The instant can also be used to remove future instants via the creatively named FilterOutFutureInstant which has no configurations thank to the use of the --date parameter and TimeService in its dependency-injected constructor.

FilterOutFutureInstant op;

return entities
    .Run(op)
    .ToList()
    .ToAsyncEnumerable();

In most cases, I gather up all the pages, generate the project pages so I can show future chapters but grayed out, then remove the future ones before producing the HTML so no one can “cheat” by just adding one to the page.

Schedules

The above operations and concepts pretty much got me through generating d.moonfire.us. On the other hand, fedran.com needed something more since I don't date my chapters. Instead, they are things like fedran.com/sand-and-blood/chapter-01/ and they need to be sent out every week until I run out of buffer or fail to keep up.

To support that, I added a second NuGet package, MfGames.Temporal.Schedules which contains more complex operations that adjust or change an entity's attributes based on the current date. In both cases, they are driven off a “path” which may be any arbitrary component but default to the entity's Zio component. They allow me to set up a schedule in a separate YAML file (could be JSON, could be on the page) and then they change.

PeriodicPathRegexSchedule

The first is PeriodicPathRegexSchedule which is the initial evolution of writing a schedule for static sites. It uses a regular expression to capture the numeric part of a path (12 from chapter-12) and figure out how long to show it. If we use the Model.Access above to control access, we can easily make chapters available for subscribers and then spread them out a week at a time starting at the beginning of the year.

schedules:
  # Patron and Ko-Fi subscribers get it all at once
  - pathRegex: chapters/chapter-(\d+)
    scheduleStart: 2000-01-01
    schedulePeriod: instant
    access: subscribers
  - pathRegex: chapters/chapter-(\d+)
    scheduleStart: 2023-01-01
    schedulePeriod: 1 week
    access: public

Depending on the “now” when the site is run, all the chapters are available to patrons but they will be made public at the rate of one per week starting with the first chapter on January 1st. Of course, with regular expressions, I could easily stop posting chapters for a while by splitting the regular expression in two to cover two ranges. Or I can have two schedules that have a tier-1 and a tier-2 release with a public release a week after tier-2 gets it.

schedules:
  - pathRegex: chapters/chapter-(\d+)
    scheduleStart: 2000-01-01
    schedulePeriod: tier-1
    access: subscribers
  - pathRegex: chapters/chapter-(\d+)
    scheduleStart: 2023-01-01
    schedulePeriod: 1 week
    access: tier-2
  - pathRegex: chapters/chapter-(\d+)
    scheduleStart: 2023-01-08
    schedulePeriod: 1 week
    access: public

This can also get complicated when I had to take breaks from chapters:

schedules:
  # Patron and Ko-Fi subscribers get it all at once
  - pathRegex: chapters/chapter-([0-1]\d|2[0-4])
    scheduleStart: 2000-01-01
    schedulePeriod: 2 weeks
    access: public
  - pathRegex: chapters/chapter-(2[5-9]|[3-9]\d)
    scheduleStart: 2023-01-01
    schedulePeriod: 2 weeks
    access: public

Obviously, implementing this is more complicated then setting the instant from the path.

public class PageSchedule : NumericalPathSchedule
{
    public Access { get; set; }

    protected override Entity Apply(Entity entity, int number, Instant instant)
    {
        var model = entity.Get<Model>();
        model.Access = this.Access;
        return entity.Set(instant, model);
    }
}

List<PageSchedule> schedules;
Ienumerable<Entity> entities;

return applySchedules.WithSchedules(_ => schedules).Run(entities);

IndexedPathRegexSchedule

The second method for schedules is IndexedPathRegexSchedule. This is the most complicated to set up in C# code, but I find the easiest to comprehend in the YAML code. Instead of having a complex set of schedules and multiple regular expressions, it goes with a single regular expression to identify the numerical component and then has a dictionary of elements that say “from chapters X until the next, for chapters Y and later, do this, etc.”

schedules:
    pathRegex: chapters/chapter-(\d+)
    indexes:
        1:
            scheduleStart: 2025-01-01
            schedulePeriod: instant # means all the chapters 1-10 at once
            access: subscribers
        10:
            scheduleStart: 2030-01-01 # chapter 10 on 2030-01-01, chapter 11 on 2023-01-08, etc.
            schedulePeriod: 1 week
            access: subscribers
        30:
            schedulePeriod: never # never going to see these

This one works easier for me, mainly because Fedran doesn't have subscriber tiers or complex logic but I do have frequent pauses while posting. Also, the schedule isn't an outer list (though it could be, ApplySchedules applies every schedule in order so you can have two of these without a problem).

public class Schedule : IndexedSchedule
{
    public string? Access { get; set; }

    protected override Entity Apply(
        Entity entity,
        int number,
        Instant instant)
    {
        TestModel model = entity.Get<TestModel>();
        model.Access = this.Access;
        return entity.SetAll(instant, model);
    }
}

IndexedPathRegexSchedule<Schedule> schedules;
Ienumerable<Entity> entities;

return applySchedules.WithSchedules(_ => new [] { schedules }).Run(entities);

Conclusion

As you may notice, these don't filter out or do anything other than set the Instant and maybe some other components of an Entity. This follows the Single Responsibility Principle (SRP) but also “do one thing well”. If you want to filter it out, use an operation for that. If you want to dynamically change the contents of a page based on the current day, that's a schedule.

This is also a starting point. Everything is fluid. I have thirteen websites I'm planning on converting over to Nitride over the next year or so. The fourth site is mfgames.com when I'll start documenting all of this and probably start pushing to see if others want to use it. Right now, the only documentation comes from https://src.mfgames.com/mfgames-cil/ in the examples, read me files, and tests.

Capabilities

2022-12-21T06:00:00Z

Lately, I've been thinking about capabilities of server APIs and how the clients interact with them. This is driven by a couple things I'm considering doing in the future but also with the frustration that certain features in the fediverse are difficult to implement for clients.

In specific, I'm very fond of Markdown posts and emoji responses. I love Markdown for all things and I like the low-attention responses of emojis when you don't need to have a big post to say “I'm sorry” or “that blows”. My family and work buddies use emoji responses fairly heavily, and it is well supported with Slack, Discord, and even Microsoft Teams as of this week.

Mastodon decided not to implement them which is one reason I like Pleroma and Misskey, though Misskey has historically been too heavy to put on my servers so one of my accounts uses Pleroma.

However, my favorite Android client, Tusky handles Markdown fairly well on the Glitch instance I'm on, but doesn't for Pleroma and it doesn't do emoji responses at all. Huksy supports these as a fork of Tusky specifically for Pleroma. But Husky doesn't really play well with the Glitch, which means I struggle to find One Client to Rule Them All™.

Likewise, I want to switch my Pleroma for GoToSocial for various reasons, but they also support Markdown but not emoji responses. Mainly because there is no standard for emoji responses and if I want to get emoji responses, I need broader acceptance or patterns to work with.

So, this comes down to capabilities. I want there was a way of having Tusky be able to query the server to determine it can handle emoji responses, so it can turn on that logic so I don't have to have Tusky and Husky. Simple, but I also want it to have the ability to know if the server can handle Markdown (base Mastodon can't, Glitch can) without doing various server identification scans.

This means a fedi server should have some ability to say “I can handle this feature” that can be queried. If there was a standard for that, then Tusky or a CLI or any other client could selectively enable or disable the feature as needed.

Fine-Grained

This is talking about a fine-grained approach to APIs. We have many different fedi servers out there, being developed and also in production. Likewise, we have a number of clients that are also growing as the fedi becomes more popular. I think we're going to see features moving from server to server as they become interesting and there is no way every client can make that choice for every server.

Identifying those abilities will make it easier to say “this client supports Markdown for those servers who have it”. It decouples the tie between the servers and the clients by giving a bit of an abstraction, not unlike how the Language Server Protocol has made developing language refactoring a lot easier across multiple clients by working to an interface.

Approach 1 - Instance Information

The first approach, which is somewhat specific to fedi servers, is to update the instance information (Mastodon, Pleroma doesn't have it implemented) to provide that information to clients. This would require updating Glitch and submitting a patch to Pleroma (and maybe Misskey) once some sort of understand/agreement could be found on how to identify these features.

This is probably the easiest and I think would really benefit the client ecosystem as a whole, not only to give me the features I want.

Approach 2 - OpenAPI Support

A more generic approach, which is beyond just fedi servers, is to encourage most API servers to also expose their OpenAPI v3.1.0 file in a well-known location that describes all the endpoints they support for that specific instance. This has benefits beyond just describing what they can handle.

However, knowing there is an endpoint that can handle something doesn't mean it works the same way. Pleroma may have the same abilities as Glitch for Markdown, but they may not be communicating them in the same way or require different inputs. There needs to be a signal that indicates a specification is in use.

References

OpenAPI has a $ref which can be used to describe an endpoint; this reference is a JSON Reference that allows an endpoint to point to another specification either in the same document or to an external URL.

It is the external URL that interests me. If that points to a fragment of OpenAPI specification that is documented for a specific feature, that would be a signal that the endpoint impelments that specific feature. In other words, if there was an /api/emoji endpoint that is described as:

{
    "$ref": "https://well.known.location/emoji.xml"
}

Then a client could use that information to assume that the endpoint has those abilities and can handle those requests.

Capabilities

As mentioned before, Mastodon doesn't have that ability since a posting is a posting, be it a response or a full text. OpenAPI also has the ability to add additional specifications to an endpoint, such as a x-additional-refs that has an array of $ref object that do point to specific ones.

That way, then Mastodon post status endpoint could add a x-additional-$refs that would allow it to describe the ability to have polls (added in 2.8.0), schedules (2.7.0), in a clear manner that clients would know it could handle those correctly without guessing or inferring from server information. Or… emoji.

{
    "paths": {
        "/api/v1/statuses": {
            "post": {
                "operationId": "postStatus",
                "parameters": [],
                "x-additional-$refs": [
                    "https://well.known.location/emoji.xml"
                ]
            }
        }
    }
}

This approach also appeals to some other projects I have in mind, so I'm curious to see what others think of it.

Thoughts

Obviously, I have a specific goal in mind: Markdown and emoji. But hopefully there is also a benefit for allowing other servers to provide this information and make it easier to create clients and interfaces to their servers.

Ceph and NixOS

2022-12-10T06:00:00Z

Recently along the continuing chain of my entanglement, the media server died. We've been ignoring the symptoms for a while such as randomly crashing with hard drive failures, refusing to start up, and occasionally making a high-pitched noise. All signs that something terrible was going to happy but we were in the middle of other problems and just pushed it off.

Then it died properly and wouldn't get back up.

The Plans

Over the years, I've been looking at Ceph ever since DreamHost did a blog post about it. It seemed perfect for my previous experiences in crashing RAIDs and trying to find enough disk space to ram “just one more database” needed to get an analysis done. These same problems leaked into my media collection and keeping track of Partner's photo shoots.

When the drives crashed, I figured I'd sit down and do a Ceph cluster instead and see if it would be easier to bring more drives online as I run out without having to tear down servers and replace drives or find another spot to fit the server. Plus the whole idea of being able to have duplicates appealed to me.

After a bit of hemming and hawing, I picked up five 6 TB drives and waited half a month for them to show up (because I'm avoiding Amazon as much as possible and NewEgg is great (except that I cannot run their application on my phone).

Initial Set Up

After putting in the first drive, I promptly set up the other four for Ceph. Because of various bugs, frustrations, and learning curve difficulties, it took me almost a week. I was expecting NixOS to be able to set up drives, but it couldn't. It has options for things, but there are a lot of little fiddly bits and dials to do manually and then turn them into the Nixian way after the fact.

OSDs and NixOS

Setting up an OSD on Nix seems pretty simple:

services.ceph.osd = {
  enable = true;
  daemons = ["0"];
};

But that was not meant to be. NixOS doesn't really set things up for you, so you have to do it manually. Also, Ceph plays well with systemd but it does not play with Nix's version of systemd.

ceph-volume lvm create --data /dev/sdb --no-systemd

The above command will create the proper entries in /var/lib/ceph/osd/osd-0 and things will appear to be working fine. But I found this is a lie. When the system restarts, it will helpfully wipe out the entire contents of /var/lib/ceph/osd/osd-0.

To fix that, after I call ceph-volume above, I had to do this to get the state to recover restarting.

cd /var/lib/ceph/osd
tar -cjf ~/osd-0.tar.bz2 osd-0
systemctl restart ceph-osd-0.service
tar -xjf ~/osd-0.tar.bz2
chown -R ceph:disk osd-0
systemctl restart ceph-osd-0.service

After that, systemctl restart ceph-osd.target didn't blow away all my files. This is because there are some tmpfile rules in the systemd configuration for OSD that don't seem to be in the mon, mds, or mgr entries.

Corrupted Drives

It took another week of copying as much as I could off the existing corrupted drives. Thankful, I could mount them with ntfs-3g which means rsync could grind through them, spending about an hour for every file that was unrecoverable before moving on.

Over the years, I've had to recover bad drives thrice now. One time it took me almost two weeks to recover what I could off of Partner's laptop drives. This was one of the big reasons why I decided to go with Ceph, to handle the cases when we have photoshoots and large files and then lose the hard drive they are stored on.

Mistake One

I managed to get it up to find out my first mistake: Ceph needs 1 GB of RAM for every 1 TB of disk and I had 24 TB on a 16 GB machine. I also only had a 8 GB swap partition for the machine, which means everything was just grinding away.

To compensate, I could have removed the drives but I had just spent a week copying files over and basically the old drives were toast. So I spent a few hundred dollars and picked up two cheap Dell business towers from NewEgg instead. When they showed up, I started the process of moving one of the drive which involves “out” the OSD (the disk) and then Ceph gracefully moved the files off that drive so it can be safely moved.

Mistake Two

The second mistake was a minor one, the new computers didn't have drive rails so I needed to order a few more ports while I waited for the first disk to be cleared off so I could move it to a second drive I decided to put the 3.5" drives into the 5.25" bay because I didn't need the DVD and it was easier to run the wires.

Mistake Three

The third mistake was probably a big one. I picked the wrong drive to “out” and moved a live drive into the second machine. I also learned that Ceph is very tolerant of moving said drives but it gets very cranky. Since the old PC (my first Ceph server) was struggling, the rubber for the mounting screws cracked so I decided to go the slow approach and just “in” the drive I thought I was going to remove, “out” the one I'm actually moving, and used a Sharpie to identify said drives so I don't make that mistake again.

Adding Monitors

When I had three systems, I decided I needed to bring up a monitor on all three to get some balancing. Not to mention having a monitor go down means the entire system crashed. However, this took me a few tries because of how NixOS handles systems.

export MID=$(hostname)
export MIP=$(host $MID.local | cut -f 4 -d ' ')

cd /var/lib/ceph/mon
mkdir ceph-$MID
cd ceph-$MID
mkdir /tmp/add-ceph-mon
ceph auth get mon. -o /tmp/add-ceph-mon/keyring
ceph mon getmap -o /tmp/add-ceph-mon/map
ceph-mon -i $MID --mkfs --monmap /tmp/add-ceph-mon/map --keyring /tmp/add-ceph-mon/keyring
ceph-mon -i $MID --public-addr $MIP
rm -rf /tmp/add-ceph-mon

Starting up the new monitor is fine as long as you do not touch the NixOS configuration files. This first part uses killall and host to make the monitor ID equal to the server. When it comes up, it will be running in daemon mode which means NixOS will forget it on restart:

root        1792  1.0  0.3 575324 59284 ?        Ssl  17:00   0:00 ceph-mon -i notil --public-addr 192.168.2.36

In this case, I kill that process, either with kill 1792 or killall ceph-mon.

Then, I configure the server to start up the server and use colmena to push out the changes.

services.ceph = {
  mon.enable = true;
  mon.daemons = ["paruk"];
};

Once I make sure everything is finally up, then I add them to the initial hosts.

services.ceph = {
  global = {
    monInitialMembers = "muliq,notil,paruk";
    monHost = "muliq.local,notil.local,paruk.local";
  };
};

Next Steps

Overall, I'm still happy with this. There is a huge learning curve when it comes to Ceph and NixOS, mainly in how they interact with each other. I assume there will be more difficulties but I seem to be heading in the right direction.

Migrated Fedran to Nitride

2022-11-25T06:00:00Z

Part of my push to get Flight of the Scions published, I had to rewrite my website because the landing page couldn't get any commerical links or be updated. That meant I had two options before me, either to figure out the Gatsby errors that caused me to stop adding new content years ago or move forward and migrate the site to Nitride.

Since moving forward with Nitride would benefit me in the long wrong (I could add new content) but would also set me back (not everything moved along), I decided Nitride was the best approach. So, in the middle of everything going on, I sat down and migrated the website.

Appearance

I went with a much different look for the site, mostly based on the CSS theme for https://d.moonfire.us/, but with the “standard” green theme I use for Fedran. I also knew I would be eventually hosting it on Gemini so much of the complex layouts and multiple columns were taken away in favor of CSS Grids and linear lists. This also let me create a “dark” and “light” mode since the original Fedran site predated my evolving preference for dark mode (and the CSS to support it). As a bonus, I went through and made sure everything passed Firefox's accessibility page for AA contrast.

This did make the site a bit more “generic looking” but that is also where my preferences are leading too. Complex sites are pretty to use but I wanted something easier to navigate and less confusing (a frequent complaint). It also meant I got rid of those fancy buttons with the identifiers on the left and the various keywords on the right in favor of something like:

Flight of the Scions (0047-00) → Kanéko • Novel (129k)

Sand and Ash (0100-01) → Rutejìmo • Novel (78k)

Sand and Blood (0100-00) → Rutejìmo • Novel (69k)

Sand and Bone (0100-02) → Rutejìmo • Novel (82k)

On the other hand, as I integrate genres and other information, the bullet list version will scale much easier and I won't struggle to layout it out in neat columns. (You can also see I integrated word counts into the listings, which is new).

Migration

The migration went surprisingly easy. I was originally planning on holding off fedran.com until the end, because it is my “flagship” and most complicated site, but the various styles and concepts that I was working with were too different as long as I didn't try to integrate the custom calendar or build up the time line (two features that were removed but will come back eventually). The idea for multiple category lists came from the MediaWiki version of the site. I change the format how plugins were listed on the page, but that was just to work around MarkDig's and Zio functionality. The only big problem with my wiki links which I originally had a Markdown links with no paths such as [Rutejìmo]() but eventually wrote a proper MarkDig extension so I could say [[Rutejìmo]] instead and fit with better conventions.

The biggest thing was the 80+ Git repositories. I had already decided to move the retrieval out of the Nitride process (as opposed to Gatsby trying to do everything), so I had those prepared in a shell script called before building. And I had already tried out some ideas when integrating my garden where a Nitride pipeline looped through the projects and called a builder on each one before gathering all the results into a single pipeline.

var entities = this.fileSystem    
    .EnumerateItems("/fedran/sources", SearchOption.TopDirectoryOnly)
    .OrderBy(a => a.Path.ToString())
    .SelectMany(this.CreateProject)
    .ToList();

Being able to set the “/” of a project to the input files is a great feature of Zio and I use it for most projects that need to view a directory structure without worrying about where that directory is located.

Overall, it took two weeks.

New Features

There are two really cool features that I'm proud of. The first is that dead links (e.g., pages I haven't written) are not automatically turned into spans with a class. That means if a link is there, it will go to a page as opposed to having so many be missing. I think this will improve the experience a lot while still giving me the hooks to fill in the pages and eventually aim for a 100% coverage of pages.

The other related to the Fedran's POV coloring scheme. Every character has a different color associated with them. The entire idea was that you could identify the series by the primary colors in the cover. Well, as part of the rewrite, I switched from SASS variables to CSS variables which meant I could inject a POV-specific stylesheet on the pages so now the POV charcters pages have the same color scheme as their covers, such as Sand and Blood, Rutejìmo, Flight of the Scions, and Allegro.

It's a little touch but I love how the colors change based on the page. And it covers both the POV charcter's entry and all of their sources.

Gemini

One of the other reasons I wanted to migrate to Nitride was because I wanted to also generate Gemtext pages and create a Gemini pod. This is a bit of a “silly” project since I know most readers aren't going to ever be on Gemini, but the “small network” is nice and I like sticking my work out where outliers might find it.

Because I wrote Nitride with Gemini in mind and had already done this site, it ended up only being a few days to create something I thought was pleasing and to push it out as a pod. It was nice as things I've been working on for over a year just fell into place.

Next Steps

There are a lot of next steps when it comes to that site. I still seem to be struggling to write and programming is a nice distraction, but I really want to get my time line back, which requires me to port my custom calendar (MfGames Culture) code to C# and get the “what is every chapter or scene where this character participated in or was referenced from”.

Then, start to fill out the pages since I really should write up Flight of the Scions in the site.

Various Updates - Good and Bad

2022-10-26T05:00:00Z

I made my last post almost two months ago. I was uncomfortable and in pain and was distracting myself with migrating to a new forge. In the time since then, I've pretty much done the same thing: been in a lot of pain and migrating stuff.

Migration

Over the last month, I managed to get all of my stories and novels migrated over to my Git repository, switched to a private CI, and create EPUB/PDFs of every single story. That has been a relief, thought there is one blank page that annoys me. I'll fix it sooner or later, but it does mean in addition to my website, I will have EPUB and PDF versions of every story I wrote.

It also gets me about 80% off GitLab. The others will take longer, but it will happen sooner or later.

Flight of the Scions

I'm still on track with publishing Flight of the Scions on November 8th. I have a review request out, a guest post written and submitted, and it posted almost everywhere.

But… the Fedran page has a problem and I can't fix it without spending a lot of hours fixing it. So my current goal is to get Fedran updated to use Nitride and let me post links to various stores. I was able to disable the later chapters, so if you didn't get a beta copy, it will be an indeterminate amount of time before I release the chapters again.

Naturally, I'm nervous about publishing this book. It has taken me a lot of gets to get to this point and I really want folks to love it. I just really don't know how to get the word out about it, other than to be excited and try to promote myself.

Warning

The last section is about the complications from my surgery. No graphic details, but somewhat negative.

Surgery

At this point, I can safely say, the surgery did not go well like the rest of my luck. It started well and things seem to be right on track. Right before I ended up in the emergency room for excruciating pain for six to ten hours every day. Actually, went to the emergency room twice because when the nurse asked if I wanted to be hospitalized, there was a pressure to go home and “push through it” when I should have said yes.

Ray, when someone asks if you're a god, say “yes!”

Little would I know that it was only the early stages of a series of cascading failures when it came to my surgery. I did end up being hospitalized overnight, which was a terrible experience of doctor's ignoring everything I said including when they would ask if I wanted something for the pain: I said yes, then they gave me nothing. Three times.

I think much of it was that I also had asymptotic Covid. As things are when overworked people are only looking at paper, they focused purely on that and shoved me out the door as soon as I was able to walk around. It took another week or so before I recovered from that.

As a lovely bit of foreshadowing, during the first emergency room visit, I had a CAT scan and they said there was a small fluid pocket from the surgery but “it probably wasn't going to be a problem.”

It did end up being a problem, including more frantic doctor visits. My original surgeon went on a medical leave so I was tossed over to a second doctor who had no clue what had happened. It took a week of agony for him to get the baselines. Then more days until I could get a drain tube put in for a week, to have it go wrong, then back into the doctor's office.

The second doctor decided the drain tube was enough and pulled it out. This ended up being premature, so then I had to get it lanced when it came back with a vengeance. Now, I'm going to the hospital for wound packing for a week now and probably will be going for few more weeks; but there are already signs that I will need another procedure because there are some additional complications beyond the weekend nurses not packing it correctly.

Overall, there are were at least seven points where someone made a decision, with hindsight, was the wrong one. One thing I could handle, but this has been problem after problem after problem.

Currently, the best part is that I'm only slightly uncomfortable (but always exhausted) as opposed to the days of agony that dominated my September.

Migrating to Take Away the Pain

2022-09-06T05:00:00Z

I had surgery last Friday. It was and continues to be painful, so I needed to distract myself so I decided to finish up the process of migrating my C# projects from GitLab over to my Gitea: https://src.mfgames.com/mfgames-cil/.

At this point, all of my libraries had been moved over, CI changed over to Woodpecker CI and deploying to the NuGet repository on the Gitea server. I also went through each of them and set up source mapping which means all of the “MfGames.*” will only point to my host and everything else will go to nuget.org as normal.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="mfgames.com" value="https://src.mfgames.com/api/packages/mfgames-cil/nuget/index.json" protocolVersion="3" />
  </packageSources>
  <packageSourceMapping>
    <packageSource key="nuget.org">
      <package pattern="*" />
    </packageSource>
    <packageSource key="mfgames.com">
      <package pattern="MfGames.*" />
    </packageSource>
  </packageSourceMapping>
</configuration>

There is still a lot more to do in process, but I feel good about having things over on my own system. I did love GitLab a lot, they have taken care of me for many years, they just went in a direction I can't follow.

In specific, they made everything free and open in the beginning, but as they got success, they needed to start to make a profit. That's understandable, but I was unable to find a reasonable way of affording such services and their sales stopped talking to me when they saw how small I was (I have many organizations but but with only 1-2 users and they couldn't tell me how to send them money to allow that). So, my $20/month goes into hosting my instances instead.

Most of the Fedran sources are already migrated, but they are not hooked up to Woodpecker CI so I don't really consider them “migrated”. Also, Woodpecker doesn't have a scheduled post so I can't do my daily update that I use pretty heavily so the websites haven't migrated.

This also ties into my short-term plan over the next few months, so once I get this site (https://d.moonfire.us/) migrated, then I can start pushing to get fedran.com ready to host Flight of the Scions for November 8th.

August in a Nutshell

2022-08-27T05:00:00Z

The last few weeks of August have been very busy, both in terms of obligations, deadlines, and events of note. Some of them are negative while others are positive, so give me a chance to tell how my month has been going.

The Trip

The month started with a trip to the family cabin. It was about ten days, mainly because Partner had things going on and was unable to attend (various reasons of their own, I don't force them to come to my family events). It ended up being ten days and fourteen hours of driving (I do not like driving long distance, period).

My younger brother did a lot to spruce up the cabin and he brought the results of his multi-year project: a boat he had been restoring that my father had helped build. It was cool and nice to see a slice of my childhood in the water again.

I didn't get on it. Not that I didn't want to, it just didn't work out. The younger cousins (my kids and his kids) were having fun with rafting and water skiing. That meant I was well suited for jumping into the water around the dock and occupying them as they took turns, helping guide the boat into position, and assisting in my kids (who don't know how to swim as well as his).

I still had a ball. I like visiting family, seeing my brother, and just remaining in contact since I'm really terrible at initiating conversation outside of those events.

Broadband

One of the projects I started in August of last year was trying to get broadband into the cabin. The city had a large-scale installation and we got free installation of fiber (about twice as fast as I have at home). It was pretty much installed a year to the week, which worked out because my company stopped providing an access point and my phone doesn't work that far north†.

Having that there means that there is a better chance I can have a “writing weekend” where I got up for a couple of days of writing and work since everything I do is remote. I just have to get a better monitor/keyboard set up there.

Still, when it comes to reducing the chaos in my life, this was something that was hanging over me as one more item in the entanglement.

† T-Mobile started saying “99% coverage of the population” which is a lot more accurate because 99% of 329.5 million is still missing out on 3.3 million people. As such, that is Verizon territory and my phone pretty much craps out the second I hit the interstate. (I won't buy non-GSM phones and Verizon screwed me over more than once because I hold onto phones for about a decade before buying a new one.)

Flood Five

Now, I wouldn't be mentioning the entanglement if there wasn't something that went wrong. When I came home and went into my basement office, I could smell moisture. That is rarely a good sign and something that is quickly triggering PTSD.

Apparently, there was a major rain while I was gone and the sump pump had died so the entire basement flooded. Every scrap of carpet, soaked drywall, and ruined boxes were laid out for me to manage. Today, I got all of the carpet and padding out and into the garage; I'm planning on getting a dumpster in late October to get rid of it along with the rotted wood from the shelves and furniture and everything else that needs to get tossed.

One thing I find fascinating is that of all the six floods I've had in the basement (first one didn't really count since that was the week I moved in), they have all had entire different sources:

Flood Zero: Water main cracked
Flood One: Derecho in 2020
Flood Two: Cracked pipe inside the slab
Flood Three: Refrigerator water line snapped
Flood Four: Mother's Day toilet explosion
Flood Five: Sump pump

Floods two through five have all happened in the last twelve months. The end result is that I used to have a finished basement. At this point, I think the entire thing has been gutted enough to consider it no longer finished.

With best estimates, I figured I'll get it back to speed in five to ten years. I'm still working in the basement, I just have everything on naked concrete.

Surgery

In 2020, I caught Covid. I suspect twice, once in February since I had all the symptoms and then in November where I had all the symptoms but even worse. During the first one, I coughed so hard I hurt myself but it was a minor hernia so the doctor's dismissed it (probably because I'm fat) until it became obvious that it was steadily getting worse.

Fast foward to July: that is when we can change our benefits at work. I switched from high-deductible to PPO and then got the doctor to look at it with something more than “well, that looks painful, so how is everything else?”

Next week, I get surgery to fix that and I have so much hopes that it will handle the nigh continuous pain I've been in for the last four months. Given that I'm hypersensitive to input, that has been a distraction for almost everything I do and prevented me from sleeping well.

Of course, I have to get through the pre-op exam, the pre-pre-op tests, the pre-op phone call, and the pre-op billing statement first.

This also put a knife edge on the deadline for the carpet removal from the flooding since I won't be able to pick up anything heavy for a while. Wet carpet is not… light. Nor is moving water-logged furniture.

Allegro

A while ago, I contributed to a fund-raiser to send some writers to Clarion. Call it a wish fulfillment since I won't be able to go myself. One of the rewards was having another author read a book (or a chance to play with an AI novel writing tool). I decided to go with the reading and that meant I needed to finish “something” before September.

Since I was stalled on Allegro, I committed to finishing that novel for September. Earlier this week, I got the first draft done and I'm so excited. For what is a glorified fan-fiction of Lindsey Sterling's Roundtable Rival set in Fedran, I'm pretty happy with a 105k word novel coming from a four minute music video.

It ended up being a much different story that Lindsey's video. It ties into Songbird in a Kitchen and the general build up of stories to Fedran's world war (it's a Phase One story). Regardless, I'm really happy with the results and I hope I can polish it up to make it my sixth published novel (Second-Hand Dresses will be my fifth and Nor Curse Be Found will probably be my seventh).

Flight of the Scions

Before the trip, I got the “release candidate” for Flight of the Scions done and sent to Ingram for a print version. It came while I was at the cabin. It looks beautiful and I'm really happy how the new covers turned out.

I also botched the layout (missed one page of padding so everything has the wrong margins), so I need to order a new one. That also means I have another chance to look for typos. My dad sent me a bunch, I found a few, I'm just going to work on them in September because of Allegro.

I think I'm on-track for the November 8th release of Flight of the Scions and I'm so excited. I will have early copies at ICON this year since I'll have a table in the merchant area along with Shannon Ryan's Merger of Evil.

For a novel I've been working off and on for almost two decades, having this out will be both a relief and a source of constant anxiety because it doesn't have my full evolution of writing (I couldn't rewrite it one more time, I just couldn't). It also means I need to get off my rear and start writing Pack Daughter, the second of four in the series.

Fedran Website

Of course, I want to show off since my books are available to read even before they are for sale. Unfortunately, the site generator that https://fedran.com/ uses, Gatsby JS, proved to be too fragile for how I use it. Plus it doesn't play well with my goal of getting my site on Gemini.

That means I need to get it migrated over to Nitride as soon as reasonable. Until then, the site is pretty much static until I can do that. It also means I can't start posting weekly chapters for Allergo until that comes out (even though everyone will be able to read the entire thing).

I just like having that weekly post. It makes me feel good to see that steady signal. I have no clue if I'm going to try again for three hundred consecutive weeks like before. In most regards, I just effectively took a year off from writing.

September

September is going to be a busy month for me. I have Allegro to get out, Flight to polish, and fix the little things that are needed to get the book out the door.

Semantic Release and Woodpecker CI

2022-08-07T05:00:00Z

With the recent drama of GitLab, both with the CI/CD changes and then more recent possible threat of deleting old repositories, I continue my migration to a local Gitea instance, https://src.mfgames.com/ for the bulk of my code and writing.

For the most part, migrating is just a matter of shuffling data. I have a lot of repositories, both active and inactive, and it will take me months to move them over. Plus I haven't decided if I'm going to purge them from my GitLab account so there is a single source of truth or just mirror back to them.

Currently, the most difficult task was figuring out how to handle the build processing. I've mentioned previously that I use Conventional Commits and Semantic Release fairly heavily. I've branched out a little from there using Lefthook and my project layout.

Currently, the CI does the following:

Build the project
Test various conditions including valid commit messages
If the commits indicate a new build:
1. Tag it
2. Build the release version
3. Create a release on Gitea

This changes over time, but it is the basic pattern.

Tags and Git Depth

Woodpecker does not automatically download the needed tags for semantic-release (and GitVersion). This means that the .woodpecker.yml file needs to include tags.

clone:
    git:
        image: woodpeckerci/plugin-git
        settings:
            tags: true
pipeline:
    # The pipeline elements

Unike GitLab, which only limits to the last ten commits, it appears that Woodpecker downloads the full repository by default which is also needed by GitVersion because it calculates every version. Not entirely sure about semantic-release logging indicates it doesn't need the full repository, just enough back to find a version.

Building and Testing

To support task branches, I have a basic build and test code that runs on pushes and pull requests. This lets me identify bugs earlier and catch typos with my commits.

build:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - nix develop --command scripts/build.sh
    when:
        # We need both "tag" for the next section.
        event: [push, pull_request, tag]
        tag: v*

test:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - nix develop --command scripts/test.sh
    when:
        event: [push, pull_request]

From the build tasks, you can see that I'm using my current project layout which uses scripts in the scripts/ folder instead of npm run or dotnet run. This is to make it easier to work with polyglot plus works around the issue that I need to use nix develop to get into my reproducible environment since the Docker image doesn't automatically do that. This is because both Gitlab and Woodpecker use the image which bypasses initialization files and I couldn't have it run direnv allow automatically to set up environment variables.

One thing that is missing is that Woodpecker doesn't have a clean mechanism for temporary build artifacts. I can't upload the build files and then download them so I can see the final results. Instead, I have to script it out or use a S3 plugin.

Building on Versions

With most cases, I build the release version of the project when the conventional commits indicate that there is a new version (feat and fix). This is an additional pipeline that comes after the test: line.

release-main:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - export DRONE="true" # Required to convince `env-ci`
        # semantic-release needs this locally
        - git branch $DRONE_BRANCH origin/$DRONE_BRANCH
        - nix develop --command scripts/release.sh
    secrets:
        - gitea_token
        - git_credentials
    when:
        event: push
        branch: main

There are a number of things in this block that took me a while. The first is the event and branch. We only do releases on the main (I'm still moving away from master as racist language).

The second is the export DRONE line. At the time I set this up, env-ci wasn't aware of Woodpecker, but it was recently added thanks to 6543 on the Woodpecker Matrix channel, #woodpecker-ci:matrix.org. I don't know when the latest semantic-release will have it, but it shouldn't be needed soon, if not already.

The third is the git branch line in the above script. Woodpecker creates a detached head, as does Gitlab. But when it doesn't do is also create a local branch for the one being created. This causes a problem because the release process appears to “jump” to the branch to figure out the changes between the detached head (the commit being built) and the actual branch.

Finally, we have the secrets. semantic-release automatically picks up $GITEA_TOKEN for the release process but also needs $GIT_CREDENTIALS to verify Git access.

The token is easy, that is what given by Gitea for the user.

GIT_CREDENTIALS is slightly harder, it is a colon-separate tuple of the user name and the Gitea access token. From observations, the Gitea is basically just a bunch of URL-safe characters, so the URL-escaping isn't needed in my case (you need to URL-escape the left and right of the colon but not the colon itself).

$ export GITEA_TOKEN=9fc6d72c72e4b149f07491a0b2d3ec9215d57caf
$ export GIT_CREDENTIALS="dmoonfire:$GITEA_TOKEN"

These need to be set on a per-project basis since Woodpecker, unlike GitLab and sourcehut, there doesn't appear to be a good way of having a shared set of secrets for projects (GitLab has organization/group level secrets, sourcehut has the secret storage). This means I have to set the same GITEA_TOKEN and GIT_CREDENTIALS for all 80+ of my Fedran repositories†.

† Woodpecker has a CLI, woodpecker-cli which will let me automate that. I will use that.

Create Release on Tag

One thing I'm moving toward is creating a release entry on the forge. Since this happens after the build process and Woodpecker uses a different Docker image, I need it to be a post-release event so I hang it off the tagging process instead of the push to main.

6543 came to my rescue again with this one, so that is why there are those two “tag” elements in the script above. I also have a new stanza for the release process:

release-gitea:
    image: plugins/gitea-release
    settings:
        base_url: https://src.mfgames.com
        files:
            - "*.pdf"
            - "*.epub"
        api_key:
            from_secret: gitea_token
    when:
        event: tag
        tag: v*

If I didn't have the event: and tag: in the build: stanza, it wasn't working for the tags. This caused me some difficulties because I usually treat hashes as being unordered, but Woodpecker uses file order for processing pipelines. So, I needed to have the build: target build the file (with the correct version because it was tagged) and then release-gitea: to use that output for the release process. The test: and release-main: are skipped because they don't have those events listed.

In addition, secrets are handled differently when done as a parameter for a plugin. That is why I have the from_secret: element in the above script. This inconsistency threw me for a few days.

Putting it Together

If you want to see the final version, check out this example which has my current version as a single file.

Conclusion

I'm happy to move over to Woodpecker (you know, except for the cost of hosting) both because of the control and the challenge. I also don't have a need for speed, so if it takes a while to get through the queue, I'm okay.

Project Layout (v4.0.0)

2022-07-20T00:34:17Z

I'm a polyglot programmer. I work in a variety of languages but mostly in C#, Typescript, and Rust. Every few years, I try a new language to see if I can pick up new ideas or if one “fits” my current mental state better. This is also why I've done a lot dozens of other languages; I would say I know over thirty languages but I'm only a “master” in a handful.

I also flit from project to project. I have my writing and games. I have little one-off programs and ones that I hope will become a major thing. But, like everything else in my life, I'm “gloriously unfocused” on my tasks which means I have to minimize the speed that I get into a project before the muse escapes me.

Over the years, I've come up with a project structure that works across most languages. It is mostly “for me”, but I think there are aspects of that others may enjoy.

Conventions

Throughout this document, // represents the root of the Git repository. So, //src/just/build.just would be the build.just in the ‘src/just’ file under the root.

When I'm referring to a directory, I will include a trailing lash such as //src/ to make a distinct between that and //Justfile which is a file.

Also, I use forward slashes regardless of the operating system, PowerShell can handle both and it doesn't require escaping in most languages.

With HTML paths, I will use // when a link has both a Gemini and HTML server associated with it or explictly say https:// if it is an HTML-only site.

Philosophy is the general guidelines for setting up a project.
Just has the tool used to run scripts in projects.
Directories describes the preferred directory layout.
Setup for setting up a project to make it “easier” to get into it.
History describes the historical evolution of my project layout.

License

This book is distributed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license. More info can be found at https://creativecommons.org/licenses/by-nc-sa/4.0/

The preferred attribution for this novel is:

“Project Layout” by D. Moonfire is licensed under CC BY-NC-SA 4.0

In the above attribution, use the following links:

Project Layout: https://d.moonfire.us/garden/project-layout/
D. Moonfire: https://d.moonfire.us/
CC BY-NC-SA 4.0: https://creativecommons.org/licenses/by-nc-sa/4.0/

Source

The source of this project can be found on the Moonfire Games forge. Feel free to report any issues, requests for expansion or clarifications. Alternately, you can contact me directly.

Nitride v0.3.4

2022-06-06T05:00:00Z

Last week, I decided to work on Nitride, my static site generator written in C#. One of the outstanding tasks after I wrote it for this site to handle both HTML and Gemtext versions was to look at the various patterns that I had established and then simplify them.

Cleaning up code is good when starting with something that potentially can get large. I had used Cobblestone for almost a decade between all my websites and I fully expected Nitride to last the name. That means I need to be comfortable with the API. Of course, “comfortable with” usually means “I like it until I stop using it for three months and then I want to rewrite it all.” However, I've managed to avoid that with Cobblestone and I'm “mostly” resisting that with MfGames Writing.

Side note: I'm planning on switching MfGames Writing to configuration-as-code instead of declarative files in the future, just not sure when.

The other plan I had was to write unit tests and examples for every option to basically help document the place. This was going to be injected into https://mfgames.com/ as a project, but I realized that I was trying to do too much and I needed to break it down.

This week's effort was just pattern normalization. I've embraced using dependency injection for setting up the various pipelines, so there is no more mix cases of constructor injection and new operations; it's all constructor injection. That way, I can offload the validation logic to FluentValidators and change the signatures, such as to add logging, without breaking existing code.

public PagesMarkdownPipeline(
    ReadFiles readFiles,
    MoveToIndexPath moveToIndexPath,
    ParseYamlHeader<PageModel> parseYamlHeader,
    RemovePathPrefix removePathPrefix,
    IdentifyMarkdownFromPath identifyMarkdownFromPath)
{
    this.readFiles = readFiles.WithPattern("/pages/**/*");
    // ...
}

As you can tell from the example above, I'm also using the fluent pattern a lot more to handle configuration. I think this works since the initial object has mostly sane defaults and validations for the rest of them while the rest can be customized in the constructor.

I also decided to keep the configuration in the constructor of the pipeline. That makes the actual operation of the pipeline much easier to understand.

/// <inheritdoc />
public override Task<IEnumerable<Entity>> RunAsync(
    IEnumerable<Entity> entities)
{
    IEnumerable<Entity> files = this.readFiles.Run()
        .Run(this.removePathPrefix)
        .Run(this.moveToIndexPath)
        .Run(this.identifyMarkdownFromPath)
        .Run(this.parseYamlHeader)
        .ForEachEntity<PageModel>(this.SetInstantFromPageModel);

    return Task.FromResult(files);
}

As you can tell, it also pulls from my history with gulp a lot since I think the chained .Run() methods works. The ForEachEntity is from Gallium, my simply entity component system that I wrote for this (but also want to use in other places).

The other thing in the above example is that the pipelines (but not necessarily the individual operations) are async. I figured this gives a good mix of allowing async operations at the pipeline level but in most cases, it ends up being fairly synchronous.

Thanks to the C#'s enumerations, the actual value sometimes isn't run until after the pipeline “runs”. There are exceptions, like when I need to create a category pages which requires knowing every other page in the website. Those are resolved immediately in the pipeline to process that, and then returned.

Overall, it is moving forward nicely. I still see a lot of shuffling of method names and I want to break up the index creation pages (the annual, monthly, and daily blog listings) into separate components. But, once I get those, I should be ready to start converting the next site, https://moonfire.us/ and then into the next “big” one, https://mfgames.com/. The “end goal,” as it were, is converting https://fedran.com/.