Statiq

Using Nitride - Markdown

2025-06-10T05:00:00Z

Yesterday, I created an over-engineered program to copy a single file from one directory to another. Now, time to make it less overkill by transforming that Markdown file into simple HTML.

Series

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

MarkDig

I don't like reinventing the wheel. I mean, I seem to keep doing it but I don't enjoy it. That was one reason why I tried another static site generators before working on MfGames.Nitride. However, when it comes to redoing a Markdown parser, even I'm not that foolish when there is already MarkDig, an excellent library for turning Markdown into HTML and extendable enough that I could also turn Markdown into Gemini (a later post).

In these cases, we need to tell Nitride how to do anything with Markdown since I didn't make it part of the core library. To do that, we need to pull in the NuGet package. While we're at it, we're also going to add the HTML processing library.

$ cd src/dotnet
$ dotnet add package MfGames.Nitride.Markdown
$ dotnet add package MfGames.Nitride.Html

Once we have the packages installed, we need to add those modules into the system. This is where Autofac came in helpful since I just have to add a module for the package it will handle the registration of any operations, components, and systems that we need to use.

// In //src/dotnet/Program.cs
var builder = new NitrideBuilder(args)
    .UseIO(rootDirectory)
    .UseMarkdown()
    .UseHtml()
    .UseModule<WebsiteModule>();

As you can see, I'm trying to follow the generic host pattern for the setup.

Identifying Markdown

While it may be obvious to convert any entity class that ends in .md into .html, we break this apart into separate steps. First, is that we identify a file as a Markdown file. This does two things, it adds the MfGames.Nitride.Markdown.IsMarkdown as a component, and then treats the contents as text instead of binary.

If you remember previously, we had this output:

[00:41:57 INF] <PagesPipeline> Read in 1 files from /src/pages
[00:41:57 INF] <PagesPipeline> Entity: Path /index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]
[00:41:57 INF] <SimplifiedMarkdownPipeline> Reading 1 entities
[00:41:57 INF] <StyleHtmlPipeline> Reading 1 entities
[00:41:57 INF] <OutputHtmlPipeline> Writing out 1 files
[00:41:57 INF] <OutputHtmlPipeline> Entity: Path /build/typewriter/html/index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]

Now, we're going to use a new operation, MfGames.Nitride.Markdown.IdentifyMarkdownFromPath. This could be put in a central place, such as SimplifiedMarkdownPipeline, but I found it is better to do this earlier than later so I usually put the identify process in the input methods. In this case, PagesPipeline:

// In //src/dotnet/Pipelines/Inputs/PagesPipeline.cs
public PagesPipeline(
    ILogger<PagesPipeline> logger,
    ReadFiles readFiles,
    IdentifyMarkdownFromPath identifyMarkdownFromPath)
{
    _logger = logger;
    _identifyMarkdownFromPath = identifyMarkdownFromPath;

    _readFiles = readFiles
        .WithPattern("/src/pages/typewriter/**/*.md")
        .WithRemovePathPrefix("/src/pages/typewriter");
}

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    var list = _readFiles
        .Run(cancellationToken)
        .Run(_identifyMarkdownFromPath)
        .ToList();

This operation doesn't take any parameters because it attempts to “do the right thing” with a minimal amount of effort. Running the code now produces this:

[00:46:43 INF] <PagesPipeline> Entity: Path /index.md, Components ["MfGames.Nitride.Markdown.IsMarkdown","Zio.UPath","MfGames.Nitride.Contents.ITextContent"]

The big things is that we have a new component, IsMarkdown, and the IBinaryContent changed to ITextComponent which gives us some additional extension methods but also indicates that the file is a text file instead of treating it as a simple binary. It still hasn't loaded the file into memory, it just switched how it is handled.

This is a separate step is because sometimes I want to keep a specific file as Markdown and not convert it to HTML. Also, there are times when I construct an Entity directly without having a valid path and I just have to add the IsHtml and ITextContent and it then acts like every other file without having to have any special rules.

Content

Entities with ITextComponent have a number of useful extension methods associated with them. (Technically, you can use these methods against any Entity class and it will try to convert a binary to text if there is a IBinaryContent and we want a ITextContent).

bool hasText = entity.HasTextContent();
string text = entity.GetTextContent();

entity.SetTextContent(stringValue);
entity.SetTextContent(stringBufferValue);

These are also stored as a ITextContent instead of string or StringBuffer. This is because the default interface is to leave the content on the disk and use Zio to retrieve it. However, as soon as SetTextContent is used, then it keeps that value in memory for the rest of the execution. This is the point where memory pressure begins to increase.

In the future, we could easily create a ITextContent implementation that writes large text files to the disk to get them out of memory. However, even my largest chapter of twenty-five thousand words doesn't create too much of a problem so I haven't bothered trying to implement that at this point (but if I did, it would go into //.cache in some manner).

Converting Markdown to HTML

Just identifying a file as Markdown doesn't do anything in itself. To convert it, we use another operation, MfGames.Nitride.Markdown.ConvertMarkdownToHtml. This easily goes into the StyleHtmlPipeline to handle the conversion and styling. It allows any MarkDig extension or plugin to be called as part of the setup, allow one to customize exactly how the output is generated.

We also want to change the extension from .md to .html. I realize I should have baked that logic into the ConvertMarkdownToHtml since it is a common operation, which I will do, but for now, we also need to use the MfGames.Nitride.IO.Paths.ChangePathExtension to do that.

public StyleHtmlPipeline(
    ILogger<StyleHtmlPipeline> logger,
    SimplifiedMarkdownPipeline simplifiedMarkdownPipeline,
    ConvertMarkdownToHtml convertMarkdownToHtml,
    ChangePathExtension changePathExtension)
{
    _logger = logger;
    _changePathExtension = changePathExtension.WithExtension(".html");

    _convertMarkdownToHtml = convertMarkdownToHtml
        .WithConfigureMarkdown(builder =>
        {
            SmartyPantOptions smartyPantOptions = new();
            SmartyPantsExtension smartyPants = new(smartyPantOptions);

            builder
                .Use<GenericAttributesExtension>()
                .Use(smartyPants);
        });

    AddDependency(simplifiedMarkdownPipeline);
}

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    var list = entities
        .Run(_convertMarkdownToHtml, cancellationToken)
        .Run(_changePathExtension, cancellationToken)
        .ToList();

    _logger.LogInformation("Reading {Count:N0} entities", list.Count);

    return list.ToAsyncEnumerable();
}

Running this gets us:

$ just build
[01:00:43 INF] <PagesPipeline> Read in 1 files from /src/pages
[01:00:43 INF] <PagesPipeline> Entity: Path /index.md, Components ["MfGames.Nitride.Markdown.IsMarkdown","Zio.UPath","MfGames.Nitride.Contents.ITextContent"]
[01:00:43 INF] <SimplifiedMarkdownPipeline> Reading 1 entities
[01:00:44 INF] <StyleHtmlPipeline> Reading 1 entities
[01:00:44 INF] <OutputHtmlPipeline> Writing out 1 files
[01:00:44 INF] <OutputHtmlPipeline> Entity: Path /build/typewriter/html/index.html, Components ["MfGames.Nitride.Html.IsHtml","Zio.UPath","MfGames.Nitride.Contents.ITextContent"]
$ find build -type f
build/typewriter/html/index.html
$ cat src/pages/typewriter/index.md 
# Typewriter Press
$ cat build/typewriter/html/index.html
cat build/typewriter/html/index.html
<h1>Typewriter Press</h1>

And now our over-engineered copy method has duplicated markdown2html for a single file.

Components

You may notice that IsMarkdown has been removed and IsHtml was added. This is part of where I struggled with Statiq and lead me down the path of using components. I don't have to pre-define the different data types, purposes, or even formats of a file. Enums are great, but they don't allow easy extension but with an ECS, it's just a matter of adding and removing components based on the use.

I've used components for a lot of things including identifying pages that should be in the blog archives, special notices, or pages that I want to ignore because they are aliases. I also embed indexes and lists into the pages to allow things like the “next” or “previous” links. If I was doing a web comic, I could have a per-character next/previous system easily implemented via those components.

My other static site generators didn't even have the content type tagging Statiq did, which was a novel concept for me and one that I'm glad I had a chance to use. It simplified a lot of my logic and lead nicely into where I am today.

Planning Ahead

As I'm planning ahead, I'm going to do the following change:

Rename SimplifiedMarkdownPipeline to ContentPipeline
Move the logic I just added into a new pipeline called BareHtmlPipeline and insert it into between ContentPipeline and StyleHtmlPipeline

The reason for this is because RSS/Atom feeds use bare HTML to generate their content, so it makes sense to have that bare pipeline feed both of them while having the ContentPipeline handle a lot of the linking and references that we'll need.

Directory Paths

One last thing for this post: I prefer paths that end in directory slashes instead of files. So, if create a contact page at //src/pages/content.md, we want the HTML to be at https://typewriter.press/content/. This is, creatively enough, another operation: MfGames.Nitride.IO.Paths.MoveToIndexPath.

// In //src/dotnet/Pipelines/Inputs/PagesPipeline.cs
public PagesPipeline(
    ILogger<PagesPipeline> logger,
    ReadFiles readFiles,
    IdentifyMarkdownFromPath identifyMarkdownFromPath,
    MoveToIndexPath moveToIndexPath)
{
    _logger = logger;
    _identifyMarkdownFromPath = identifyMarkdownFromPath;
    _moveToIndexPath = moveToIndexPath;

    _readFiles = readFiles
        .WithPattern("/src/pages/typewriter/**/*.md")
        .WithRemovePathPrefix("/src/pages/typewriter");
}

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    var list = _readFiles
        .Run(cancellationToken)
        .Run(_identifyMarkdownFromPath)
        .Run(_moveToIndexPath)
        .ToList();

And we add a contact.md page which a run gives us this:

$ just build
[01:12:26 INF] <PagesPipeline> Entity: Path /contact/index.md, Components ["MfGames.Nitride.Contents.ITextContent","MfGames.Nitride.Markdown.IsMarkdown","Zio.UPath"]
[01:12:26 INF] <PagesPipeline> Entity: Path /index.md, Components ["MfGames.Nitride.Contents.ITextContent","MfGames.Nitride.Markdown.IsMarkdown","Zio.UPath"]
[01:12:26 INF] <ContentPipeline> Reading 2 entities
[01:12:26 INF] <BareHtmlPipeline> Reading 2 entities
[01:12:26 INF] <StyleHtmlPipeline> Reading 2 entities
[01:12:26 INF] <OutputHtmlPipeline> Writing out 2 files
[01:12:26 INF] <OutputHtmlPipeline> Entity: Path /build/typewriter/html/contact/index.html, Components ["MfGames.Nitride.Contents.ITextContent","MfGames.Nitride.Html.IsHtml","Zio.UPath"]
[01:12:26 INF] <OutputHtmlPipeline> Entity: Path /build/typewriter/html/index.html, Components ["MfGames.Nitride.Contents.ITextContent","MfGames.Nitride.Html.IsHtml","Zio.UPath"]
$ cat src/pages/typewriter/contact.md 
# Contact Us

Our emails is [contact@typewriter.press](mailto:contact@typewriter.press).
$ cat build/typewriter/html/contact/index.html 
<h1>Contact Us</h1>
<p>Our emails is <a href="mailto:contact@typewriter.press">contact@typewriter.press</a>.</p>

As you can see, the input is //src/pages/contact.md, but through the pipeline, it is written out as //build/typewriter/html/contact/index.html.

What's Next

Next up, handling front matter. We need that for many reasons, but one of the biggest reason is to provide metadata to generate styled HTML output.

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

Using Nitride - Entities

2025-06-09T05:00:00Z

From yesterday's post, there was an dangling topic, the Entity class. Unlike most of the static site generators I've worked with, MfGames.Nitride uses an ECS (Entity Component System) to generate its files.

Series

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

History

I've worked on a number of static site generators over the years, including more than a few adhoc ones until I finally got something stable with CobblestoneJS which carried me for almost a decade of heavy use. A lot of the patterns that Nitride was based on were on how I found I had to build my pages in Cobblestone, including having a separate “gather” step which wrote out temporary files and then the “process” step which formatted them and made everything pretty. I had to do that because I wanted to have cross-linking and references work across multiple Git projects.

A good example is the Fedran website. At the bottom of a chapter, I have a list of characters and locations in the chapter. The links are built from data from the wiki project but the chapter information comes from the story Git repository. The gather step let me combine both of those together to generate those links.

That also meant that I needed to have some supplementary information stored in the gather step (which was written to a //build folder) to help with those linking.

Near the end, once Cobblestone began to break under its abstraction and complexity, I wanted to get away from writing my own and switched to GatsbyJS which made a very pretty website, but it was painful to debug and develop. It only lasted a year or so before I wanted to move (though the Gatsby version of Fedran was probably the prettiest I had ever made). If anything, Gatsby ended up being an ecosystem that needed me to pay attention to update things and my cycle of coming around to update a project was too far for it so I spent days trying to get it working again to build it.

So I tried out Statiq. I never successfully made a website out of it, but it introduced me to some really interesting concepts including pulling in data from other sources (Gatsby also did this, just never really used it). But with the C#, it had some really interesting problems. However, I quickly found that I was trying to do something that was outside of the planned usage, which involved me trying to extend an enumeration which involved basically rewriting large hunks of the system just to handle it. Plus, I really like Autofac and I could never get it to play well with that.

Which lead me into creating Nitride.

Now, I want to say, both Gatsby and Statiq are both good systems. They just aren't good systems for me. These don't do what I want them to do and they are designed around patterns that I'm not comfortable with. Even with the desire not to write my own, it wasn't enough.

Entity Component Systems

Nitride is build on MfGames.Gallium which basically means the formal name of the system Gallium Nitride:

Gallium nitride (GaN) is a binary III/V direct bandgap semiconductor commonly used in blue light-emitting diodes since the 1990s. The compound is a very hard material that has a Wurtzite crystal structure. Its wide band gap of 3.4 eV affords it special properties for applications in optoelectronics, high-power and high-frequency devices. For example, GaN is the substrate that makes violet (405 nm) laser diodes possible, without requiring nonlinear optical frequency doubling.

Gallium is a small ECS library. It doesn't intend to be fast or efficient, but it was written to match the idiomatic patterns of System.Linq. I wanted something that wasn't going to reinvent the wheel, which also meant not making custom collection classes or lookup tables. Instead, I wanted to work with BCL primitives which is why much of the system is based on IEnumerable<Entity>.

There is some high-level documentation for Gallium on its project page and its test, but I'll give the basics here.

The core object is MfGames.Gallium.Entity. This is a small class that has an integer identifier and basically a Dictionary<Type, object> called Components. It has some methods to make it easier, but basically you can set or get components from it:

// This creates an empty entity.
Entity entity1 = new();

// This creates an entity with four components:
//   "System.Int32": 123
//   "System.String": "bob"
//   "System.IO.FileInfo": ...
//   "System.IO.DirectoryInfo": ...
Entity entity2 = new()
    .Set(123)
    .Set("bob")
    .SetAll(
        new FileInfo("/tmp/bob.txt"),
        new DirectoryInfo("/tmp")
    );

Assert.True(entity2.HasComponent<FileInfo>);
Assert.False(entity2.HasComponent<decimal>);

Assert.True(123, entity2.Get<int>());

In effect, the class of the component is the key. There are a number of other functions, the test class is probably the best documentation at this point.

The systems part of the system is the LINQ-inspired collection classes. Most of them work off IEnumerable<Entity> with the test class being the bulk of the documentation:

// Set up a list of entities.
List<Entity> list = ...;

// This gets all the entities that have a FileInfo.
var fileOnlyList = list.WhereEntityHas<FileInfo>().ToList();

// This gets all entities that have both a FileInfo and an int.
var fileAndIntList = list.WhereEntityHasAll<FileInfo, int>().ToList();

// Get a modified list based on components. This looks for all
// entities that have a FileInfo and then adds a marker component,
// IsHtml, into it.
IsHtml isHtml = IsHtml.Instance;

var modifiedList = list
    .SelectEntity<FileInfo>((entity, file) => entity.Set(isHtml))
    .ToList();

As a note, Entity is “mostly” immutable. The Set command will return a new Entity instance with the same ID but with a different set of components. The “mostly” bit comes that the objects that it points to can be mutated.

🛈 The lack of formal documentation for Gallium and Nitride is one of the reasons this is still alpha or beta level code.

Operations

The next concept needed for today's post is an operation which extends MfGames.Nitride.IOperation. These are just easy ways of creating some functionality that works on an IEnumerable<Entity> that can be combined together.

Naturally, there are some extension methods that make it easier to work with operations with an IEnumerable<Entity>:

IEnumerable<Entity> entityList = ...;

IEnumerable<Entity> otherList = someOperation1.Run(entityList);

IEnumerable<Entity> resultList = otherList
    .Run(someOperation2)
    .Run(someOperation3)
    .Run(someOperation4);

This makes it relatively easy to create easy processing through composition of one or more operations. Alternatively, we could easily just work directly off the enumerable.

Zio

There is no question, I'm fond of the Zio library. In this case, Nitride uses two big features with it: to create an abstract view of the project that is independent of the working directory and the UPath class which normalizes paths.

If you remember from the previous posts, we set a root directory in the Program.cs:

public static async Task<int> Main(string[] args)
{
    var rootDirectory = new DirectoryInfo(Environment.CurrentDirectory);

    var builder = new NitrideBuilder(args)
        .UseIO(rootDirectory)
        .UseModule<WebsiteModule>();

    return await builder.RunAsync();
}

This caused an Zio.IFileSystem to be injected in the system which treats that directory as the /. So, in our case where we had our index file in //src/pages/index.md, it has a UPath of /src/pages/index.md. It doesn't seem like a lot, but it means that all paths in the system are always relative to the root of the project and there is no need to pass a root directory throughout the system which means path operations are considerably simplified.

And the UPath ensures a consistent pattern for accessing so this works on Windows, Mac, and Linux without a problem. Not to mention, any issues with .. are handled automatically.

It also simplifies writing tests.

Copy File in Six Files

Put all that together, we can use two operations to create a massively over-engineered copy file operation using Nitride. Admittedly, the complexity will make sense later, but at the moment, we're going to read the file in the PagesPipeline, through SimplifiedMarkdownPipeline, through StyleHtmlPipeline, and finally write it out in OutputHtmlPipeline.

Making Some Noise

To start with, let's put some logging to show how everything flows from pipeline to pipeline. To do that, we're going to inject an ILogger<> into the constructor into all four pipelines and then put a logging message into the last three (the PagesPipeline is going to do something different).

// In //src/dotnet/Pipelines/Content/SimplifiedMarkdownPipeline.cs
public class SimplifiedMarkdownPipeline : PipelineBase
{
    private readonly ILogger _logger;

    public SimplifiedMarkdownPipeline(
        ILogger<SimplifiedMarkdownPipeline> logger,
        PagesPipeline pagesPipeline)
    {
        _logger = logger;

        AddDependency(pagesPipeline);
    }

    public override IAsyncEnumerable<Entity> RunAsync(
        IEnumerable<Entity> entities,
        CancellationToken cancellationToken = default)
    {
        var list = entities.ToList();

        _logger.LogInformation("Reading {Count:N0} entities", list.Count);

        return list.ToAsyncEnumerable();
    }
}

Internally, Nitride uses Serilog, so using {Count:N0} does the right thing even thought we are using a logging abstraction for the actual call. When we run this, we get the following output (I'm not going to show the just build call for these today):

[14:00:46 DBG] <DirectoryService> Setting root directory and filesystem to /home/dmoonfire/src/typewriter-press/typewriter-press-website
[14:00:46 DBG] <BuildCommand> Processing 0 pipeline options
[14:00:46 INF] <BuildCommand> Running pipelines
[14:00:46 INF] <SimplifiedMarkdownPipeline> Reading 0 entities
[14:00:46 INF] <StyleHtmlPipeline> Reading 0 entities
[14:00:46 INF] <OutputHtmlPipeline> Reading 0 entities
[14:00:46 INF] <PipelineManager> Completed in 00:00:00.1079413
[14:00:46 INF] <BuildCommand> Command took 00:00:00.1097142

🛈 I moved NuGet.config to the root level because there is where the Website.sln file was located and it felt right.

Reading Files

Now, to read a file, we are going to use the MfGames.Nitride.IO.ReadFiles operation which reads file from the Zio file system and passes them through. We've already included that NuGet package in a previous step, so nothing to do there.

// In //src/dotnet/Inputs/PagesPipeline.cs
using MfGames.Gallium;
using MfGames.Nitride.IO.Contents;
using MfGames.Nitride.Pipelines;
using Microsoft.Extensions.Logging;

namespace Generator.Pipelines.Inputs;

public class PagesPipeline : PipelineBase
{
    private readonly ILogger _logger;
    private readonly ReadFiles _readFiles;

    public PagesPipeline(
        ILogger<PagesPipeline> logger,
        ReadFiles readFiles)
    {
        _logger = logger;
        _readFiles = readFiles.WithPattern("/src/pages/**/*.md");
    }

    public override IAsyncEnumerable<Entity> RunAsync(
        IEnumerable<Entity> entities,
        CancellationToken cancellationToken = default)
    {
        var list = _readFiles
            .Run(cancellationToken)
            .ToList();

        _logger.LogInformation(
            "Read in {Count:N0} files from /src/pages",
            list.Count);

        foreach (var entity in list)
        {
            _logger.LogInformation("Entity: Path {Path}, Components {ComponentList}",
                entity.Get<UPath>(),
                entity.GetComponentTypes().Select(x => x.FullName));
        }

        return list.ToAsyncEnumerable();
    }
}

This requires a bit of explanation. We are using dependency injection to insert the ReadFiles operation into the constructor. This is not a singleton, so you can have multiple ReadFiles if you need to read from multiple locations (or use a globbing operator to do it in one).

_readFiles = readFiles.WithPattern("/src/pages/**/*.md");

In the constructor, I configure it. I could do this in the RunAsync function but I found it made it easier to parse the files when I configure in the constructor. Like most things in Nitride, this uses a fluent coding style where each object sets the properties and returns itself to make a nice chained list of calls.

There are two properties being set, Pattern and RemovePathPrefix. I use a source generator to automatically create the With... pattern for fluent coding. The first one tells which files to read, relative to the root directory. This happen to use a globbing that allows me to grab all Markdown files in the source directory.

var list = _readFiles
    .Run(cancellationToken)
    .ToList();

This run the _readFiles operation and gets a list of the files it read in. (I'm also going to put this in the OutputHtmlPipeline to show how everything writes out.)

foreach (var entity in list)
{
    _logger.LogInformation(
        "Entity: Path {Path}, Components {ComponentList}",
        entity.Get<UPath>(),
        entity.GetComponentTypes().Select(x => x.FullName));
}

This is just some debugging to show how things are working and how Gallium works. In this case, the read files sets two components: the UPath for the path of the file and a IBinaryContent which is an abstraction to the binary content. Since we haven't done anything to the file, this just points to the file system itself without loading it into memory. That way, we can easily deal with gigabyte-sized files without overloading the system.

return list.ToAsyncEnumerable();

With this, we return the list of files that we just read in. With this change, running the build gets us this:

[14:20:03 DBG] <DirectoryService> Setting root directory and filesystem to /home/dmoonfire/src/typewriter-press/typewriter-press-website
[14:20:03 DBG] <BuildCommand> Processing 0 pipeline options
[14:20:03 INF] <BuildCommand> Running pipelines
[14:20:04 INF] <PagesPipeline> Read in 1 files from /src/pages
[14:20:04 INF] <PagesPipeline> Entity: Path /src/pages/typewriter/index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]
[14:20:04 INF] <SimplifiedMarkdownPipeline> Reading 1 entities
[14:20:04 INF] <StyleHtmlPipeline> Reading 1 entities
[14:20:04 INF] <OutputHtmlPipeline> Writing out 1 files
[14:20:04 INF] <OutputHtmlPipeline> Entity: Path /src/pages/typewriter/index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]
[14:20:04 INF] <PipelineManager> Completed in 00:00:00.9543639
[14:20:04 INF] <BuildCommand> Command took 00:00:00.9559424

As you can see, that single file read in is then passed through the other pipelines because each of them return their input list and then the final pipeline gets those same entities and paths.

Changing Paths

Now the problem with the above example is that we are writing our output to the same location as we are reading. That isn't optional. What we want to do is strip off the /src/pages/typewriter from the beginning of the read in path and add /build/typewriter/html in front of it. This is easily done with two additional operators, RemovePathPrefix and AddPathPrefix.

To start with, let's remove the prefix from the read methods to normalize the paths.

// In //src/dotnet/Inputs/PagesPipeline.cs
public PagesPipeline(
    ILogger<PagesPipeline> logger,
    ReadFiles readFiles,
    RemovePathPrefix removePathPrefix)
{
    _logger = logger;
    _removePathPrefix = removePathPrefix.WithPathPrefix("/src/pages/typewriter");
    _readFiles = readFiles.WithPattern("/src/pages/**/*.md");
}

And then we chain the operation using an extension method on operations:

// In //src/dotnet/Inputs/PagesPipeline.cs
public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    var list = _readFiles
        .Run(cancellationToken)
        .Run(_removePathPrefix)
        .ToList();

Since each operation also takes an IEnumerable<Entity> and removes the same type, they can flow from one operation to another. When we run the build, the output line looks like this:

[14:23:35 INF] <PagesPipeline> Read in 1 files from /src/pages
[14:23:35 INF] <PagesPipeline> Entity: Path /index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]

Adding the output prefix is done on the output step:

// In //src/dotnet/Html/OutputHtmlPipeline.cs
public OutputHtmlPipeline(
    ILogger<OutputHtmlPipeline> logger,
    AddPathPrefix addPathPrefix,
    StyleHtmlPipeline styleHtmlPipeline)
{
    _logger = logger;
    _addPathPrefix = addPathPrefix.WithPathPrefix("/build/typewriter/html");

    AddDependency(styleHtmlPipeline);
}

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    var list = entities
        .Run(_addPathPrefix)
        .ToList();

Because we already have a list of entities, we can just use the extension method to pipe it through the operation and produce the following output:

[14:28:35 INF] <PagesPipeline> Read in 1 files from /src/pages
[14:28:35 INF] <PagesPipeline> Entity: Path /index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]
[14:28:35 INF] <SimplifiedMarkdownPipeline> Reading 1 entities
[14:28:35 INF] <StyleHtmlPipeline> Reading 1 entities
[14:28:35 INF] <OutputHtmlPipeline> Writing out 1 files
[14:28:35 INF] <OutputHtmlPipeline> Entity: Path /build/typewriter/html/index.md, Components ["MfGames.Nitride.Contents.IBinaryContent","Zio.UPath"]

The reason PagesPipeline doesn't use entities.Run(...) is because that is the first pipeline and the entities is going to be an empty list. Instead, we need to make entities, which is why we call _readFiles.Run() instead.

Writing Files

And the last bit for today's post is to write out files. This, creatively enough, uses the WriteFiles operation.

// In //src/dotnet/Html/OutputHtmlPipeline.cs
public OutputHtmlPipeline(
    ILogger<OutputHtmlPipeline> logger,
    AddPathPrefix addPathPrefix,
    WriteFiles writeFiles,
    StyleHtmlPipeline styleHtmlPipeline)
{
    _logger = logger;
    _writeFiles = writeFiles;
    _addPathPrefix = addPathPrefix.WithPathPrefix("/build/typewriter/html");

    AddDependency(styleHtmlPipeline);
}

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    var list = entities
        .Run(_addPathPrefix)
        .Run(_writeFiles)
        .ToList();

In this case, we've already done all the fancy operations to adjust the path so it writes out the files in the proper location:

$ find build -type f
build/typewriter/html/index.md

Convenience Methods

Now, adding and removing path prefixes is pretty common, so both the ReadFiles and WriteFiles have methods to do those in a single call.

// In //src/dotnet/Inputs/PagesPipeline.cs
public PagesPipeline(
    ILogger<PagesPipeline> logger,
    ReadFiles readFiles)
{
    _logger = logger;
    _readFiles = readFiles
        .WithPattern("/src/pages/typewriter/**/*.md")
        .WithRemovePathPrefix("/src/pages/typewriter");
}

// In //src/dotnet/Html/OutputHtmlPipeline.cs
public OutputHtmlPipeline(
    ILogger<OutputHtmlPipeline> logger,
    WriteFiles writeFiles,
    StyleHtmlPipeline styleHtmlPipeline)
{
    _logger = logger;
    _writeFiles = writeFiles.WithAddPathPrefix("/build/typewriter/html");

    AddDependency(styleHtmlPipeline);
}

What's Next

And there you go, an over-engineered system for copying a single file from one directory to another. In the next post of this series, we'll turn that Markdown file into an HTML page.

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

Using Nitride - Pipelines

2025-06-08T05:00:00Z

I found a good way of working on a project is being able to see the end product. That means forging a minimal path from the start to end, and then going back to flesh it out. Interestingly, this is the opposite of how I write novels because, in those, the journey is the important part. Not so much with development projects.

In this case, the goal is to get a website generated and the ability to see it locally.

Series

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

Project Setup

Since this mainly a .NET project, we start by creating a new project and solution. This is a fairly simple generator and I don't plan on needing more than one assembly, so I'm going to throw everything into //src/dotnet/ for the website.

$ dotnet new console --name Generator --output src/dotnet
$ dotnet new solution --name Website
$ dotnet sln add src/dotnet/Generator.csproj

And since we want to have the project run, we also update the Justfile:

# //Justfile

# Builds the typewriter.press website
build-typewriter:
    dotnet run --project src/dotnet/Generator.csproj --

And that gives us an easy command to run that works anywhere underneath the Git repository (thanks to Just).

$ just build-typewriter 
dotnet run --project src/dotnet/Generator.csproj --
Hello, World!

Setting up NuGet

At the moment, the MfGames.* assemblies are not on nuget.org. There are a couple of reasons, mostly self-doubt that anyone would find this useful and the dread of setting up a SSL certificate which appeared to be required. My current intent is to do it once I have a number of sites using Nitride or I get a request to do so. Until then, I use NuGet.config file to hook up my libraries to the website.

<!-- //src/dotnet/NuGet.config >
<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="mfgames.com" value="https://src.mfgames.com/api/packages/mfgames-cil/nuget/index.json" protocolVersion="3" />
  </packageSources>
  <packageSourceMapping>
    <packageSource key="nuget.org">
      <package pattern="*" />
    </packageSource>
    <packageSource key="mfgames.com">
      <package pattern="MfGames.*" />
    </packageSource>
  </packageSourceMapping>
</configuration>

Setting up the Generator

While MfGames.Nitride could use configuration files to do the bulk of the setup, I went with a configuration-as-code approach to it since I almost always end up writing custom code. That way, everything is done in the same manner. This also uses a model roughly based on the generic host for command line.

🛈 I'm in the process of reworking the initial setup to use the generic host more completely. The current implementation was my best approach to write a dependency-injected console application that is configured via the CLI instead of JSON files.

The first step is to include the required package. Nitride is organized in a lot of small, focused packages. This is it avoid pulling in too much for a given project but also to help keep everything organized. For the initial work, we can add the first two required assemblies.

$ cd src/dotnet
$ dotnet add package MfGames.Nitride
$ dotnet add package MfGames.Nitride.IO
$ dotnet add package Autofac

Nitride using the excellent Zio library for abstracting file system access, plus making it easier to write test to make sure everything work. We also use Autofac for a lot of the module registration and processing.

🛈 I'm working on removing the Autofac dependency and using Microsoft.Extensions.DependencyInjection instead but it will take a while. There are a lot of useful features in Autofac that the extensions don't have so it will take some time.

First, we create the website module with a very generic, naive implementation (unless otherwise mentioned, all of these files are in //src/dotnet/):

// WebsiteModule.cs
using Autofac;

namespace Generator;

public class WebsiteModule : Module
{
    /// <inheritdoc />
    protected override void Load(ContainerBuilder builder)
    {
        builder
            .RegisterAssemblyTypes(GetType().Assembly)
            .AsSelf()
            .AsImplementedInterfaces()
            .SingleInstance();
    }
}

With that, we change the initial program to use Nitride:

// Program.cs
using MfGames.Nitride;
using MfGames.Nitride.IO.Setup;

namespace Generator;

public static class Program
{
    public static async Task<int> Main(string[] args)
    {
        var rootDirectory = new DirectoryInfo(Environment.CurrentDirectory);

        var builder = new NitrideBuilder(args)
            .UseIO(rootDirectory)
            .UseModule<WebsiteModule>();

        return await builder.RunAsync();
    }
}

If we run this, we'll get an error:

$ just build-typewriter 
dotnet run --project src/dotnet/Generator.csproj --
Required command was not provided.

Description:

Usage:
  MfGames.Nitride [command] [options]

Options:
  --log-level <log-level>                    Controls the verbosity of the output, not case-sensitive and prefixes allowed: 
                                             Verbose, Debug, Information, Warning, Error, Fatal [default: Warning]
  --log-context-format <log-context-format>  Controls the format of the source context for log items, not case-sensitive and 
                                             prefixes allowed: None, Class, Full [default: Class]
  --log-time-format <log-time-format>        Controls the format of the time in the log messages, such as 'HH:mm:ss' (default) or 
                                             'yyyy-MM-ddTHH:mm:ss'. Blank means exclude entirely. [default: HH:mm:ss]
  --log-level-override <log-level-override>  Overrides log levels for certain contexts in the format of either 'Context' or 
                                             'Context=Level' (repeat for multiple) [default: Microsoft=Warning]
  --version                                  Show version information
  -?, -h, --help                             Show help and usage information

Commands:
  build  Generate the website
  watch  Generates the site and then watches for changes

error: Recipe `build-typewriter` failed on line 15 with exit code 1

The watch command doesn't work very well right now, but we want build, so we'll update our Justfile:

# //Justfile
build-typewriter:
    dotnet run --project src/dotnet/Generator.csproj -- build

And then try again:

$ just build-typewriter
dotnet run --project src/dotnet/Generator.csproj -- build
[12:27:17 ERR] <PipelineManager> There are no registered pipelines run, use ConfigureContainer to include IPipeline instances
error: Recipe `build-typewriter` failed on line 15 with exit code 1

Pipelines

Now we get to the first big concept of Nitride: pipelines. Some of the ideas are built up from Statiq and also the patterns I've worked out with CobblestoneJS. They basically are larger “units” of processing for the site that are intended to gather a bunch of files and then feed them into another pipeline which then splits out into different ones for output.

It also allows a group of inputs to be handled differently, such as linking the “next” and “previous” to posts which doesn't make for pages. Another case where I've used this heavily is to copy binary assets directly to the styled output, generate CSS through webpack or esbuild, or generate project-specific pages from the input.

        +-------+ +-------+
        | Pages | | Posts |
        +-------+ +-------+
            |         |
            +----+----+
                 | 
                 v
          +-------------+
          |  Simplified |
          |   Markdown  |
          +-------------+
                 |
     +-----------+-----------+
     |           |           |
     v           v           v
+---------+ +---------+ +---------+
|  Style  | |  Atom   | | Style   |
|  HTML   | |  Feeds  | | Gemtext |
+---------+ +---------+ +---------+
     |                       |
     v                       v
+---------+             +---------+
|  Output |             | Output  |
|  HTML   |             | Gemtext |
+---------+             +---------+

Pipelines attempt to run in parallel to reduce the amount of time on the developer's machine. They are also asynchronous at their core since a pipeline could be used to query another system, require long-running processes, or perform some other action that would require using that functionality.

Obviously, you could use a single pipeline. I use to start that way, but the pattern of gathering the information, combining them together to generate tags and categories, and then styling fits better as separate steps for me. Since this is my website, we're going to create a minimum from the pages, generate simplified Markdown, style it, and then write it out.

Since we use dependency injection throughout Nitride, we pass in the dependencies into constructor of a pipeline and then add it to that project. Internally, Nitride runs every pipeline in a separate task when it is building the site.

// ./Pipelines/Inputs/PagesPipeline.cs
using MfGames.Gallium;
using MfGames.Nitride.Pipelines;

namespace Generator.Pipelines.Inputs;

public class PagesPipeline : PipelineBase
{
    public override IAsyncEnumerable<Entity> RunAsync(
        IEnumerable<Entity> entities,
        CancellationToken cancellationToken = default)
    {
        throw new NotImplementedException();
    }
}

// ./Pipelines/Content/SimplifiedMarkdownPipeline.cs
using Generator.Pipelines.Inputs;
using MfGames.Gallium;
using MfGames.Nitride.Pipelines;

namespace Generator.Pipelines.Content;

public class SimplifiedMarkdownPipeline : PipelineBase
{
    public SimplifiedMarkdownPipeline(
        PagesPipeline pagesPipeline)
    {
        AddDependency(pagesPipeline);
    }

    public override IAsyncEnumerable<Entity> RunAsync(
        IEnumerable<Entity> entities,
        CancellationToken cancellationToken = default)
    {
        throw new NotImplementedException();
    }
}

// ./Pipelines/Html/StyleHtmlPipeline.cs
using Generator.Pipelines.Content;
using MfGames.Gallium;
using MfGames.Nitride.Pipelines;

namespace Generator.Pipelines.Html;

public class StyleHtmlPipeline : PipelineBase
{
    public StyleHtmlPipeline(
        SimplifiedMarkdownPipeline simplifiedMarkdownPipeline)
    {
        AddDependency(simplifiedMarkdownPipeline);
    }

    public override IAsyncEnumerable<Entity> RunAsync(
        IEnumerable<Entity> entities,
        CancellationToken cancellationToken = default)
    {
        throw new NotImplementedException();
    }
}

// ./Pipelines/Html/OutputHtmlPipeline.cs
using MfGames.Gallium;
using MfGames.Nitride.Pipelines;

namespace Generator.Pipelines.Html;

public class OutputHtmlPipeline : PipelineBase
{
    public OutputHtmlPipeline(
        StyleHtmlPipeline styleHtmlPipeline)
    {
        AddDependency(styleHtmlPipeline);
    }

    public override IAsyncEnumerable<Entity> RunAsync(
        IEnumerable<Entity> entities,
        CancellationToken cancellationToken = default)
    {
        throw new NotImplementedException();
    }
}

And since we want to avoid exceptions, we'll ignore the entire Entity bit and do a couple of changes that will be explained in the next post.

In PagesPipeline.cs, we want to return an empty list:

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    return Array.Empty<Entity>().ToAsyncEnumerable();
}

For all the others, we're just going to pass whatever we got in back out:

public override IAsyncEnumerable<Entity> RunAsync(
    IEnumerable<Entity> entities,
    CancellationToken cancellationToken = default)
{
    return entities.ToAsyncEnumerable();
}

I'll admit, I stumbled into the usage of IAsyncEnumerable. It lets me use await calls inside a function without needing to load an entire list into memory just to copy it out again into another list to make the pipelines to work. Given that one of my baseline websites was to process my entire fiction website without killing my machine, I want to reduce the number of copies in memory and reduce the pressure there.

Using IAsyncEnumerable is cumbersome though, so might I might create a convenience method in PipelineBase that allows for a Task<IEnumerable<Entity>> and change it an IAsyncEnumerable but that is a future idea to work out.

When we run this, we get this (after adding ``–log-level debuginto the//Justfile`):

$ just build-typewriter 
dotnet run --project src/dotnet/Generator.csproj -- build --log-level debug
[13:22:48 DBG] <BuildCommand> Processing 0 pipeline options
[13:22:48 INF] <BuildCommand> Running pipelines
[13:22:48 INF] <PipelineManager> Completed in 00:00:00.1051617
[13:22:48 INF] <BuildCommand> Command took 00:00:00.1075694

It doesn't really do anything in the 0.1 seconds that it takes to run, but it basically gets the pipeline chain working so things will be generating with the next in this series.

Observations

I'm also aware that this appears to be overkill for most websites. If I was doing an HTML-only website, I would have started with only a single pipeline but I already know that I'm going to generate Atom feeds out of that and they work better with simple HTML instead of fancy chrome. Being able to style the Atom's output separately gives me some flexibility to do different things there. Also there are going to be multiple inputs eventually so experience tells me that this is the “best” approach for generating output given the circumstances of this specific website.

You'll also notice that I'm not doing a site-specific pipelines. I've tried a couple of approached and my current one is to run the entire generation process once per website using some configuration settings, which I'll get after I finish the main website.

Future Plans

The pipelines also are the units that are triggered on the watch command (which doesn't really work). If a pipeline identifies that it needs to be run, it will run but then trigger a re-run of every pipeline that depends on that.

The advantage of that if I'm working on styling CSS, I can have a pipeline feeding into the StyleHtmlPipeline that generates that. When the hypothetical CssHtmlPipeline triggers a rebuild, it runs itself, StyleHtmlPipeline, and OutputHtmlPipeline only because of those dependencies. But changing a page would trigger everything but the CssHtmlPipeline which should keep watches fairly performant.

What's Next

In the next post, I'll explain what Entity is and start make this actually generate something useful.

2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.

Reflections on Nitride

2024-04-21T05:00:00Z

This week's goal was to get mfgames.com migrated over to MfGames.Nitride. It was a lofty goal, mainly because I've been making small incremental changes over the last year but never got something pushed over the finish line.

I succeeded.

Migration

This is my third “big” site for Nitride, one that follows the common pattern of pulling in multiple Git repositories from my forge, integrating the data, and then creating a website out of it. In other words, one of the main reasons I created Nitride.

It isn't exactly a fancy site, but I find that I'm not really creating flashy sites anymore. Part of this is because I still enjoy Gemini, which is a text-only format, but also because the chrome is distracting to me. I want something simple but not the black on gray of Netscape Navigator.

Overall, I'm pretty happy with the results. It creates landing pages for projects, pulls in the ./docs/ folder for each project and formats them (for example, MfGames.Gallium), and merges news posts from the projects (while decorating them) and creates a single unified feed.

There are more things I want to do, such as links to issue tracking, licenses, and other stuff. I also ended up losing some features like the culture sandbox (which I needed to rewrite anyways), the slippery maps, and a few other minor things.

Still happy with it.

News

I was working on debugging creating news posts in individual project repositories anews posts from individual projects into mfgames.com as my “little thing” for today and I came up with the exact period of time I decided to write my latest static site generator (SSG):

$ git log --pretty="%ai %s"
...
2021-05-04 02:27:28 -0500 feat: implementing my own
2021-04-07 22:13:20 -0500 chore: updating Statiq libraries
...
2014-05-07 14:41:27 -0700 Initial commit

It all started a year before, when I migrated from CobblestoneJS (my SSG from the early 2010s) to GatsbyJS. Which was pretty and gorgeous and so difficult to debug. It was also fragile for me because I don't update my libraries to follow the curve and every time I tried to add a new feature to the site, I spent a few days just trying to figure out the node packages to get everything building again first.

When I got into Gemini for the first time, I also realized that GatsbyJS wasn't going to work there because that shiny site was not text friendly. It was also tied into my frustrating that going to a page the first time downloaded the content in one manner, but navigation was done via SPA and pulled it from another; and the cache was killing me there.

(But it looked so pretty.)

So I decided to look for a new SSG. I didn't want to write my own because… I really don't like writing utility programs. But I wanted to do something Gatsby didn't do, which is generate a site that didn't require Javascript.

I figured I was good at C# (this was before Rust, but I'm still best at C#), so I wanted to find a C# static site generator in hopes that the NuGet package management wasn't as excruciating as Node's.

So I found Statiq which seemed like a perfect match. But going from GatsbyJS to Statiq quickly hit a point where I realized I was, once again, forcing a program to do things it wasn't supposed to be doing. The patterns in Statiq were nice but they were too rigid for the complexity of my way of creating sites:

The fragility of pulling in 80+ Git repositories into a CI pipeline
The use of multiple dynamic lists on a single page
The desire to make a Gemini-friendly site

After that month of thinking, I started to write my own. This came when I was looking at Statiq's document type realizing that my desire to create a new type basically meant I had to rebuild the entire framework to support a slightly different enumeration.

Gallium

Enter Entity Component Systems (ECS) which I learned about while pondering writing a game using Bevy. I had never heard of an ECS before that, but the idea was great. Why not write a static site generator that didn't need a single enum to drive processing, where I could add arbitrary components to pages such as nested lists or metadata, and then easily be able to use them.

Of course, then I had to look at existing ECS systems in C#, but I eventually decided to be more idiomatic to C# (another frustration of Statiq) and focus on something that let me use LINQ-like method calls (such as .Select() and .Where()).

List<Entity> newsList = fileList
    .WhereEntity<UPath>(this.OnlyNewsFiles)
    .Run(this.setInstantFromPath, cancellationToken)
    .SelectEntity<UPath, Instant>(this.pathService.SetPathFromInstant)
    .Select(entity => entity.Set(IsPost.Instance))
    .Select(entity => entity.Set(IncludeInFeed.Instance))
    .Where(entity => entity.Has<IsMarkdown>())
    .OrderBy(entity => entity.Get<PageModel>().Title)
    .ToList();

I'm specifically not saying Gallium is a “high performance” because it isn't. It is an ECS for relatively small systems that builds on existing libraries for patterns. I don't need high performance for sites with only a few thousand pages, I want something easy to understand.

Next Steps

There are quite a few steps going forward. I still want to make Nitride easier to use, Gallium to track LINQ patterns, and mfgames.com to present more data. But this is a big step and I'm really happy that I'm proud of (I'm sure there will be tweaks). More importantly, I have something that hasn't given me any major trouble updating since I started three years ago.

Gallium Nitride and Gemini

2021-07-10T05:00:00Z

Last November, I switched my static site generator from CobblestoneJS (my homebrew Gulp-based one) to Statiq. There were a number of reasons, all of them still good but mainly to support what I want to do with fedran.com and my other sites.

This holiday weekend, I ended up ripping out all of the Statiq and wrote another static site generator, this time in C# and using an Entity-Component-System (ECS). I also migrated my site over to Gemini as part of the effort.

Why?

I liked many of the ideas that Statiq provided:

Treating the website generation as code.
Having pipelines with dependencies for generation.
Immutable objects on the pipeline.

However, I found myself struggling with concepts. It just didn't sit well with me and I would spend two weeks trying to implement a feature and getting stuck. When I realized I had spent a month not fixing something that was bothering me because I didn't want to delve into the code, I knew it was time to change.

That isn't to say Statiq isn't bad. It just isn't for me. That's it.

About once a year, I get 4-7 days of “alone time” to do what I want. This year, I decided to work on a new static site generator that did work the way I work today and that I hoped would carry me over for the next five years or so.

Entity-Component-Systems (ECS)

One thing that Statiq did (but differently) was implemented the system as an ECS. Basically, you have a lightweight object (the “entity”) and add various components into it. Those components are what provide the features: the text content, flags to say if it is HTML or Markdown, or the path.

While Statiq had a number of these elements built-into the Document call (basically their entity), there were a lot of assumptions that didn't always fit. Likewise, Statiq.Web had some nice opinionated ways of handling it, including a document type (binary, text, etc), but I couldn't find an easy way to extend it.

Gallium

With my ECS, Entity is only a collection plus an integer identifier. Components can be added, removed, and replaced easily using generics to determine the type. Methods are chained together but not pseudo-English fluent (which I'm also not fond of). Entities are immutable, so all the operations return a clone of that entity with the new components added, removed, or otherwise changed. (Thanks to functional programming for some of those ideas.)

Entity entity = new();
Entity entityWithTwoComponents = entity
  .Set(new Uri("https://d.moonfire.us"))
  .Set<FileSystemInfo>(new FileInfo("/bob.txt"));
Entity entityWithReplacedUri = entityWithTwoComponents
  .Set(new Uri("http://path-to-replace-d.moonfire.us/"));
Entity entityWithOnlyUri = entityWithReplacedUri
  .Remove<FileSystemInfo>();

I also really like chained operations, so most of the processing looks like this:

IEnumerable<Entity> input;
var onlyHtmlOutputs = input
  .OrderBy(x => x.Get<FileInfo>().FullPath)
  .ForComponents<Uri>((entity, uri) => this.Uri(entity, uri))
  .WhereComponents<IsHtml>();

The whole idea is ForComponents<T1, T2, T3> will go through the list and for all entities that have T1, T2, and T3, it will do the callback, otherwise it will passs it on. Likewise WhereComponents<T1> is basically a Where that says only the entities with the given components.

Those ideas really simplified a lot of the difficulties I had with CobblestoneJS. Overall, most of the logic “felt” right for me, so I'm really happy with the results. Plus, it is based on a far more stable package ecosystem (NuGet) and in a language I enjoy greatly (C#).

Also, the language uses Autofac as my preferred dependency injection of choice. I really like the library plus NodaTime when coding, so I went with these. It's a bit opinionated, but… only a few people ever used Cobblestone, so I'm going to assume very few are going to use this.

Once I get it cleaned up, I'll probably call the ECS “Gallium” because my original name was “Gallium Nitride” (GaN, because it's a cool name and I like what the molecule does). The static site generator would be named Nitride.

Nitride

Nitride is just a multi-threaded pipeline static generator. It uses pipelines much like Statiq but based on C#'s thread control (ManualReset, ReaderWriterLockSlim).

The rough code looks like this:

List<Entity> list = entities
    .Run(new IdentifyMarkdown())
    .Run(new ParseYamlHeader<PageModel>())
    .ForComponents<PageModel>(this.SetHandlebarsFromPageModel)
    .Run(this.setInstantFromComponent)
    .Run(this.filterOutFutureInstants)
    .Run(this.createCategoryIndexes)
    .Run(this.createTagIndexes)
    .ToList();

Again, DI or direct instantiate of modules, it all works the same and really ties into using Linq and C# generics. All of the pipelines are async but most of the operations (createTagIndexes, ParseYamlHeader) are not. But since the pipelines are, it is easy to make something await without changing signatures.

I really like the pipelines. For my site, I have the following:

PostsPipeline: Load posts and create blog archives
PagesPipeline: Load static pages.
ContentPipeline: Combines PostsPipeline and PagesPipeline, creates categories/tag indexes, handle filtering out future pages.
BareHtmlPipeline: Takes ContentPipeline and converts into bare, unstyled HTML. Also creates the RSS and Atom feeds from the bare HTML.
WebpackPipeline: Runs Webpack.
StyledHtmlPipeline: Takes BareHtmlPipeline and WebpackPipeline and makes it styled using handlebars.
GeminiPipeline: Takes ContentPipeline and turns the Markdown into Gemini, applies some simple styling, and generates Gemtext pages.

That's it, but I'm happy with the result because I've taken lessons learned from my previous attempts to created something that will handle Fedran's massive cross-linking and project pages, MfGames's pulling in of separate Git repositories, and also some of the more complex formatting of my new sci-fi fiction website.

Gemini

I like the idea of Gemini. It is a low-overhead protocol that has almost no extra features, no cookies, and basically focused on presenting content. In my case, I really want to see all of my sites on Gemini because I think it has some significant merits, more so as I want to get away from heavily styled content written by people who like tiny fonts or don't have my color contrast issues.

To do that, I ended up taking inspiration from md2gemini and wrote a C# library that converts Markdown into Gemtext (Gemini's markup format inspired by Markdown). The end result is pretty nice, I think and I'm really happy with the results.

Of course, it meant I had to get a virtual machine to host a Gemini server next to a HTTP one, but that was going to happen sooner or later anyways.

Next Steps

I write a lot libraries that I think are interesting but very few people worry about. They rise up, either I stick with them or I trail off, but they always scratch my itches. On that front, the following things are left to do:

Split out Gallium more formally into MfGames.Gallium or Gallium, not sure.
Clean up the API for Nitride and make sure everything is consistent. Some of the names are a bit… off, but the functionality is good.
Break all the modules into separate Gitlab projects and set up CI for releasing.
Document everything.
Convert five other websites sites to use it to figure out what is missing.

I haven't found the money to get a developer's signing certificate, so I'll probably just put everything up on my public MyGet repository.

Website Update

2020-11-20T06:00:00Z

I haven't been posting here lately. There are a number of reasons, but I'm going to have to go with the simple one: 2020. Lots of things all piling up, adding to to stress, and making just getting things difficult. Along the way, occasionally things slip to make sure other things continue.

Fedran was one of those things. I've been working on maintaining my weekly chapters but it seems like I haven't been doing other than heads-down writing.

One of my favorite methods for managing stress is obsessing about a little change that quickly blossoms into a much larger epic worthy of sailing across the ocean and trying to debate some sirens. Usually this ends up with me redoing one of my websites, rewriting my accounting system, or completely revamping one of my writing tools.

If you can't get, I have a “type” when it comes to reducing stress.

In this case, it was stumbling onto Gemini. This is a stripped-down but still fairly capable protocol based on both Gopher and web browsing. It focused more on information but without stressing about the “pretty” part of the web. There are no themes, font choices, or a ton of links. Well, there can be links but they are one per line and clearly identified as such.

Gemini appeals to me. Almost everything I create is already in written form, this seems like an ideal place to hang up a pole and see what happens. Not to mention, it's just fun to find new places to explore.

This also happened at the time when I had butted up against limitations in Gatsby, which I had just spent three months rewriting Fedran. I had switched to Gatsby because of flaws I encountered in Cobblestone, my own static site generator.

That isn't to say, Gatsby doesn't make a beautiful sites. The trick they use to load the next page quickly was great. But it also is very heavy on the Javascript side of things which was pretty much the opposite of trying out Gemini which has no Javascript.

Since I rarely do things halfway, I want to migrate this site over to Gemini. This means either I stick with Cobblestone (which was already failing on me), convert it to Gatsby and to Gemini, or find something new.

Enter Statiq. It's another static site generator that is one of my preferred languages (C#, Typescript, and apparently Rust) but it also allows for pipelines (what I liked about the Gulp-based Cobblestone) that would let me diverge the site between HTML and Gemini on the final steps while still allowing me the massive cross-referencing I use on Fedran.

I did consider switching Fedran, but after years of spinning wheels on redoing projects, I limit myself to one website rewrite a year. Otherwise, I would just endlessly redo my website and not actually create anything.

If you are reading this, then I've managed to switch over. Fortunately, http://d.moonfire.us/ is one of the more basic websites I have. That means I can build on it as I work toward the final boss, Fedran.