Using Nitride - Pipelines
I found a good way of working on a project is being able to see the end product. That means forging a minimal path from the start to end, and then going back to flesh it out. Interestingly, this is the opposite of how I write novels because, in those, the journey is the important part. Not so much with development projects.
In this case, the goal is to get a website generated and the ability to see it locally.
Series
2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.
Project Setup
Since this mainly a .NET project, we start by creating a new project and solution. This is a fairly simple generator and I don't plan on needing more than one assembly, so I'm going to throw everything into //src/dotnet/ for the website.
$ dotnet new console --name Generator --output src/dotnet
$ dotnet new solution --name Website
$ dotnet sln add src/dotnet/Generator.csproj
And since we want to have the project run, we also update the Justfile:
# //Justfile
# Builds the typewriter.press website
build-typewriter:
dotnet run --project src/dotnet/Generator.csproj --
And that gives us an easy command to run that works anywhere underneath the Git repository (thanks to Just).
$ just build-typewriter
dotnet run --project src/dotnet/Generator.csproj --
Hello, World!
Setting up NuGet
At the moment, the MfGames.* assemblies are not on nuget.org. There are a couple of reasons, mostly self-doubt that anyone would find this useful and the dread of setting up a SSL certificate which appeared to be required. My current intent is to do it once I have a number of sites using Nitride or I get a request to do so. Until then, I use NuGet.config file to hook up my libraries to the website.
<!-- //src/dotnet/NuGet.config >
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<clear />
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
<add key="mfgames.com" value="https://src.mfgames.com/api/packages/mfgames-cil/nuget/index.json" protocolVersion="3" />
</packageSources>
<packageSourceMapping>
<packageSource key="nuget.org">
<package pattern="*" />
</packageSource>
<packageSource key="mfgames.com">
<package pattern="MfGames.*" />
</packageSource>
</packageSourceMapping>
</configuration>
Setting up the Generator
While MfGames.Nitride could use configuration files to do the bulk of the setup, I went with a configuration-as-code approach to it since I almost always end up writing custom code. That way, everything is done in the same manner. This also uses a model roughly based on the generic host for command line.
🛈 I'm in the process of reworking the initial setup to use the generic host more completely. The current implementation was my best approach to write a dependency-injected console application that is configured via the CLI instead of JSON files.
The first step is to include the required package. Nitride is organized in a lot of small, focused packages. This is it avoid pulling in too much for a given project but also to help keep everything organized. For the initial work, we can add the first two required assemblies.
$ cd src/dotnet
$ dotnet add package MfGames.Nitride
$ dotnet add package MfGames.Nitride.IO
$ dotnet add package Autofac
Nitride using the excellent Zio library for abstracting file system access, plus making it easier to write test to make sure everything work. We also use Autofac for a lot of the module registration and processing.
🛈 I'm working on removing the Autofac dependency and using Microsoft.Extensions.DependencyInjection instead but it will take a while. There are a lot of useful features in Autofac that the extensions don't have so it will take some time.
First, we create the website module with a very generic, naive implementation (unless otherwise mentioned, all of these files are in //src/dotnet/):
// WebsiteModule.cs
using Autofac;
namespace Generator;
public class WebsiteModule : Module
{
/// <inheritdoc />
protected override void Load(ContainerBuilder builder)
{
builder
.RegisterAssemblyTypes(GetType().Assembly)
.AsSelf()
.AsImplementedInterfaces()
.SingleInstance();
}
}
With that, we change the initial program to use Nitride:
// Program.cs
using MfGames.Nitride;
using MfGames.Nitride.IO.Setup;
namespace Generator;
public static class Program
{
public static async Task<int> Main(string[] args)
{
var rootDirectory = new DirectoryInfo(Environment.CurrentDirectory);
var builder = new NitrideBuilder(args)
.UseIO(rootDirectory)
.UseModule<WebsiteModule>();
return await builder.RunAsync();
}
}
If we run this, we'll get an error:
$ just build-typewriter
dotnet run --project src/dotnet/Generator.csproj --
Required command was not provided.
Description:
Usage:
MfGames.Nitride [command] [options]
Options:
--log-level <log-level> Controls the verbosity of the output, not case-sensitive and prefixes allowed:
Verbose, Debug, Information, Warning, Error, Fatal [default: Warning]
--log-context-format <log-context-format> Controls the format of the source context for log items, not case-sensitive and
prefixes allowed: None, Class, Full [default: Class]
--log-time-format <log-time-format> Controls the format of the time in the log messages, such as 'HH:mm:ss' (default) or
'yyyy-MM-ddTHH:mm:ss'. Blank means exclude entirely. [default: HH:mm:ss]
--log-level-override <log-level-override> Overrides log levels for certain contexts in the format of either 'Context' or
'Context=Level' (repeat for multiple) [default: Microsoft=Warning]
--version Show version information
-?, -h, --help Show help and usage information
Commands:
build Generate the website
watch Generates the site and then watches for changes
error: Recipe `build-typewriter` failed on line 15 with exit code 1
The watch command doesn't work very well right now, but we want build, so we'll update our Justfile:
# //Justfile
build-typewriter:
dotnet run --project src/dotnet/Generator.csproj -- build
And then try again:
$ just build-typewriter
dotnet run --project src/dotnet/Generator.csproj -- build
[12:27:17 ERR] <PipelineManager> There are no registered pipelines run, use ConfigureContainer to include IPipeline instances
error: Recipe `build-typewriter` failed on line 15 with exit code 1
Pipelines
Now we get to the first big concept of Nitride: pipelines. Some of the ideas are built up from Statiq and also the patterns I've worked out with CobblestoneJS. They basically are larger “units” of processing for the site that are intended to gather a bunch of files and then feed them into another pipeline which then splits out into different ones for output.
It also allows a group of inputs to be handled differently, such as linking the “next” and “previous” to posts which doesn't make for pages. Another case where I've used this heavily is to copy binary assets directly to the styled output, generate CSS through webpack or esbuild, or generate project-specific pages from the input.
+-------+ +-------+
| Pages | | Posts |
+-------+ +-------+
| |
+----+----+
|
v
+-------------+
| Simplified |
| Markdown |
+-------------+
|
+-----------+-----------+
| | |
v v v
+---------+ +---------+ +---------+
| Style | | Atom | | Style |
| HTML | | Feeds | | Gemtext |
+---------+ +---------+ +---------+
| |
v v
+---------+ +---------+
| Output | | Output |
| HTML | | Gemtext |
+---------+ +---------+
Pipelines attempt to run in parallel to reduce the amount of time on the developer's machine. They are also asynchronous at their core since a pipeline could be used to query another system, require long-running processes, or perform some other action that would require using that functionality.
Obviously, you could use a single pipeline. I use to start that way, but the pattern of gathering the information, combining them together to generate tags and categories, and then styling fits better as separate steps for me. Since this is my website, we're going to create a minimum from the pages, generate simplified Markdown, style it, and then write it out.
Since we use dependency injection throughout Nitride, we pass in the dependencies into constructor of a pipeline and then add it to that project. Internally, Nitride runs every pipeline in a separate task when it is building the site.
// ./Pipelines/Inputs/PagesPipeline.cs
using MfGames.Gallium;
using MfGames.Nitride.Pipelines;
namespace Generator.Pipelines.Inputs;
public class PagesPipeline : PipelineBase
{
public override IAsyncEnumerable<Entity> RunAsync(
IEnumerable<Entity> entities,
CancellationToken cancellationToken = default)
{
throw new NotImplementedException();
}
}
// ./Pipelines/Content/SimplifiedMarkdownPipeline.cs
using Generator.Pipelines.Inputs;
using MfGames.Gallium;
using MfGames.Nitride.Pipelines;
namespace Generator.Pipelines.Content;
public class SimplifiedMarkdownPipeline : PipelineBase
{
public SimplifiedMarkdownPipeline(
PagesPipeline pagesPipeline)
{
AddDependency(pagesPipeline);
}
public override IAsyncEnumerable<Entity> RunAsync(
IEnumerable<Entity> entities,
CancellationToken cancellationToken = default)
{
throw new NotImplementedException();
}
}
// ./Pipelines/Html/StyleHtmlPipeline.cs
using Generator.Pipelines.Content;
using MfGames.Gallium;
using MfGames.Nitride.Pipelines;
namespace Generator.Pipelines.Html;
public class StyleHtmlPipeline : PipelineBase
{
public StyleHtmlPipeline(
SimplifiedMarkdownPipeline simplifiedMarkdownPipeline)
{
AddDependency(simplifiedMarkdownPipeline);
}
public override IAsyncEnumerable<Entity> RunAsync(
IEnumerable<Entity> entities,
CancellationToken cancellationToken = default)
{
throw new NotImplementedException();
}
}
// ./Pipelines/Html/OutputHtmlPipeline.cs
using MfGames.Gallium;
using MfGames.Nitride.Pipelines;
namespace Generator.Pipelines.Html;
public class OutputHtmlPipeline : PipelineBase
{
public OutputHtmlPipeline(
StyleHtmlPipeline styleHtmlPipeline)
{
AddDependency(styleHtmlPipeline);
}
public override IAsyncEnumerable<Entity> RunAsync(
IEnumerable<Entity> entities,
CancellationToken cancellationToken = default)
{
throw new NotImplementedException();
}
}
And since we want to avoid exceptions, we'll ignore the entire Entity bit and do a couple of changes that will be explained in the next post.
In PagesPipeline.cs, we want to return an empty list:
public override IAsyncEnumerable<Entity> RunAsync(
IEnumerable<Entity> entities,
CancellationToken cancellationToken = default)
{
return Array.Empty<Entity>().ToAsyncEnumerable();
}
For all the others, we're just going to pass whatever we got in back out:
public override IAsyncEnumerable<Entity> RunAsync(
IEnumerable<Entity> entities,
CancellationToken cancellationToken = default)
{
return entities.ToAsyncEnumerable();
}
I'll admit, I stumbled into the usage of IAsyncEnumerable. It lets me use await calls inside a function without needing to load an entire list into memory just to copy it out again into another list to make the pipelines to work. Given that one of my baseline websites was to process my entire fiction website without killing my machine, I want to reduce the number of copies in memory and reduce the pressure there.
Using IAsyncEnumerable is cumbersome though, so might I might create a convenience method in PipelineBase that allows for a Task<IEnumerable<Entity>> and change it an IAsyncEnumerable but that is a future idea to work out.
When we run this, we get this (after adding ``–log-level debuginto the//Justfile`):
$ just build-typewriter
dotnet run --project src/dotnet/Generator.csproj -- build --log-level debug
[13:22:48 DBG] <BuildCommand> Processing 0 pipeline options
[13:22:48 INF] <BuildCommand> Running pipelines
[13:22:48 INF] <PipelineManager> Completed in 00:00:00.1051617
[13:22:48 INF] <BuildCommand> Command took 00:00:00.1075694
It doesn't really do anything in the 0.1 seconds that it takes to run, but it basically gets the pipeline chain working so things will be generating with the next in this series.
Observations
I'm also aware that this appears to be overkill for most websites. If I was doing an HTML-only website, I would have started with only a single pipeline but I already know that I'm going to generate Atom feeds out of that and they work better with simple HTML instead of fancy chrome. Being able to style the Atom's output separately gives me some flexibility to do different things there. Also there are going to be multiple inputs eventually so experience tells me that this is the “best” approach for generating output given the circumstances of this specific website.
You'll also notice that I'm not doing a site-specific pipelines. I've tried a couple of approached and my current one is to run the entire generation process once per website using some configuration settings, which I'll get after I finish the main website.
Future Plans
The pipelines also are the units that are triggered on the watch command (which doesn't really work). If a pipeline identifies that it needs to be run, it will run but then trigger a re-run of every pipeline that depends on that.
The advantage of that if I'm working on styling CSS, I can have a pipeline feeding into the StyleHtmlPipeline that generates that. When the hypothetical CssHtmlPipeline triggers a rebuild, it runs itself, StyleHtmlPipeline, and OutputHtmlPipeline only because of those dependencies. But changing a page would trigger everything but the CssHtmlPipeline which should keep watches fairly performant.
What's Next
In the next post, I'll explain what Entity is and start make this actually generate something useful.
2025-06-07 Using Nitride - Introduction - I'm going to start a long, erratic journey to update my publisher website. Along the way, I'm going to document how to create a MfGames.Nitride website from the ground up, along with a lot of little asides about reasoning or purpose.
2025-06-08 Using Nitride - Pipelines - The first major concept of Nitride is the idea of a "pipeline" which are the largest units of working in the system. This shows a relatively complex setup of pipelines that will be used to generate the website.
2025-06-09 Using Nitride - Entities - The continued adventures of creating the Typewriter website using MfGames.Nitride, a C# static site generator. The topic of today is the Entity class and how Nitride uses an ECS (Entity-Component-System) to generate pages.
2025-06-10 Using Nitride - Markdown - Examples and explanations of converting Markdown to HTML using MfGames.Nitride and MarkDig.
2025-06-12 Using Nitride - Front Matter - How to use Nitride to take the front matter from pages and add them as a component into the Entity class.
Metadata
Categories:
Tags: