Gallium

Nitride v0.3.4

2022-06-06T05:00:00Z

Last week, I decided to work on Nitride, my static site generator written in C#. One of the outstanding tasks after I wrote it for this site to handle both HTML and Gemtext versions was to look at the various patterns that I had established and then simplify them.

Cleaning up code is good when starting with something that potentially can get large. I had used Cobblestone for almost a decade between all my websites and I fully expected Nitride to last the name. That means I need to be comfortable with the API. Of course, “comfortable with” usually means “I like it until I stop using it for three months and then I want to rewrite it all.” However, I've managed to avoid that with Cobblestone and I'm “mostly” resisting that with MfGames Writing.

Side note: I'm planning on switching MfGames Writing to configuration-as-code instead of declarative files in the future, just not sure when.

The other plan I had was to write unit tests and examples for every option to basically help document the place. This was going to be injected into https://mfgames.com/ as a project, but I realized that I was trying to do too much and I needed to break it down.

This week's effort was just pattern normalization. I've embraced using dependency injection for setting up the various pipelines, so there is no more mix cases of constructor injection and new operations; it's all constructor injection. That way, I can offload the validation logic to FluentValidators and change the signatures, such as to add logging, without breaking existing code.

public PagesMarkdownPipeline(
    ReadFiles readFiles,
    MoveToIndexPath moveToIndexPath,
    ParseYamlHeader<PageModel> parseYamlHeader,
    RemovePathPrefix removePathPrefix,
    IdentifyMarkdownFromPath identifyMarkdownFromPath)
{
    this.readFiles = readFiles.WithPattern("/pages/**/*");
    // ...
}

As you can tell from the example above, I'm also using the fluent pattern a lot more to handle configuration. I think this works since the initial object has mostly sane defaults and validations for the rest of them while the rest can be customized in the constructor.

I also decided to keep the configuration in the constructor of the pipeline. That makes the actual operation of the pipeline much easier to understand.

/// <inheritdoc />
public override Task<IEnumerable<Entity>> RunAsync(
    IEnumerable<Entity> entities)
{
    IEnumerable<Entity> files = this.readFiles.Run()
        .Run(this.removePathPrefix)
        .Run(this.moveToIndexPath)
        .Run(this.identifyMarkdownFromPath)
        .Run(this.parseYamlHeader)
        .ForEachEntity<PageModel>(this.SetInstantFromPageModel);

    return Task.FromResult(files);
}

As you can tell, it also pulls from my history with gulp a lot since I think the chained .Run() methods works. The ForEachEntity is from Gallium, my simply entity component system that I wrote for this (but also want to use in other places).

The other thing in the above example is that the pipelines (but not necessarily the individual operations) are async. I figured this gives a good mix of allowing async operations at the pipeline level but in most cases, it ends up being fairly synchronous.

Thanks to the C#'s enumerations, the actual value sometimes isn't run until after the pipeline “runs”. There are exceptions, like when I need to create a category pages which requires knowing every other page in the website. Those are resolved immediately in the pipeline to process that, and then returned.

Overall, it is moving forward nicely. I still see a lot of shuffling of method names and I want to break up the index creation pages (the annual, monthly, and daily blog listings) into separate components. But, once I get those, I should be ready to start converting the next site, https://moonfire.us/ and then into the next “big” one, https://mfgames.com/. The “end goal,” as it were, is converting https://fedran.com/.

Gallium Nitride and Gemini

2021-07-10T05:00:00Z

Last November, I switched my static site generator from CobblestoneJS (my homebrew Gulp-based one) to Statiq. There were a number of reasons, all of them still good but mainly to support what I want to do with fedran.com and my other sites.

This holiday weekend, I ended up ripping out all of the Statiq and wrote another static site generator, this time in C# and using an Entity-Component-System (ECS). I also migrated my site over to Gemini as part of the effort.

Why?

I liked many of the ideas that Statiq provided:

Treating the website generation as code.
Having pipelines with dependencies for generation.
Immutable objects on the pipeline.

However, I found myself struggling with concepts. It just didn't sit well with me and I would spend two weeks trying to implement a feature and getting stuck. When I realized I had spent a month not fixing something that was bothering me because I didn't want to delve into the code, I knew it was time to change.

That isn't to say Statiq isn't bad. It just isn't for me. That's it.

About once a year, I get 4-7 days of “alone time” to do what I want. This year, I decided to work on a new static site generator that did work the way I work today and that I hoped would carry me over for the next five years or so.

Entity-Component-Systems (ECS)

One thing that Statiq did (but differently) was implemented the system as an ECS. Basically, you have a lightweight object (the “entity”) and add various components into it. Those components are what provide the features: the text content, flags to say if it is HTML or Markdown, or the path.

While Statiq had a number of these elements built-into the Document call (basically their entity), there were a lot of assumptions that didn't always fit. Likewise, Statiq.Web had some nice opinionated ways of handling it, including a document type (binary, text, etc), but I couldn't find an easy way to extend it.

Gallium

With my ECS, Entity is only a collection plus an integer identifier. Components can be added, removed, and replaced easily using generics to determine the type. Methods are chained together but not pseudo-English fluent (which I'm also not fond of). Entities are immutable, so all the operations return a clone of that entity with the new components added, removed, or otherwise changed. (Thanks to functional programming for some of those ideas.)

Entity entity = new();
Entity entityWithTwoComponents = entity
  .Set(new Uri("https://d.moonfire.us"))
  .Set<FileSystemInfo>(new FileInfo("/bob.txt"));
Entity entityWithReplacedUri = entityWithTwoComponents
  .Set(new Uri("http://path-to-replace-d.moonfire.us/"));
Entity entityWithOnlyUri = entityWithReplacedUri
  .Remove<FileSystemInfo>();

I also really like chained operations, so most of the processing looks like this:

IEnumerable<Entity> input;
var onlyHtmlOutputs = input
  .OrderBy(x => x.Get<FileInfo>().FullPath)
  .ForComponents<Uri>((entity, uri) => this.Uri(entity, uri))
  .WhereComponents<IsHtml>();

The whole idea is ForComponents<T1, T2, T3> will go through the list and for all entities that have T1, T2, and T3, it will do the callback, otherwise it will passs it on. Likewise WhereComponents<T1> is basically a Where that says only the entities with the given components.

Those ideas really simplified a lot of the difficulties I had with CobblestoneJS. Overall, most of the logic “felt” right for me, so I'm really happy with the results. Plus, it is based on a far more stable package ecosystem (NuGet) and in a language I enjoy greatly (C#).

Also, the language uses Autofac as my preferred dependency injection of choice. I really like the library plus NodaTime when coding, so I went with these. It's a bit opinionated, but… only a few people ever used Cobblestone, so I'm going to assume very few are going to use this.

Once I get it cleaned up, I'll probably call the ECS “Gallium” because my original name was “Gallium Nitride” (GaN, because it's a cool name and I like what the molecule does). The static site generator would be named Nitride.

Nitride

Nitride is just a multi-threaded pipeline static generator. It uses pipelines much like Statiq but based on C#'s thread control (ManualReset, ReaderWriterLockSlim).

The rough code looks like this:

List<Entity> list = entities
    .Run(new IdentifyMarkdown())
    .Run(new ParseYamlHeader<PageModel>())
    .ForComponents<PageModel>(this.SetHandlebarsFromPageModel)
    .Run(this.setInstantFromComponent)
    .Run(this.filterOutFutureInstants)
    .Run(this.createCategoryIndexes)
    .Run(this.createTagIndexes)
    .ToList();

Again, DI or direct instantiate of modules, it all works the same and really ties into using Linq and C# generics. All of the pipelines are async but most of the operations (createTagIndexes, ParseYamlHeader) are not. But since the pipelines are, it is easy to make something await without changing signatures.

I really like the pipelines. For my site, I have the following:

PostsPipeline: Load posts and create blog archives
PagesPipeline: Load static pages.
ContentPipeline: Combines PostsPipeline and PagesPipeline, creates categories/tag indexes, handle filtering out future pages.
BareHtmlPipeline: Takes ContentPipeline and converts into bare, unstyled HTML. Also creates the RSS and Atom feeds from the bare HTML.
WebpackPipeline: Runs Webpack.
StyledHtmlPipeline: Takes BareHtmlPipeline and WebpackPipeline and makes it styled using handlebars.
GeminiPipeline: Takes ContentPipeline and turns the Markdown into Gemini, applies some simple styling, and generates Gemtext pages.

That's it, but I'm happy with the result because I've taken lessons learned from my previous attempts to created something that will handle Fedran's massive cross-linking and project pages, MfGames's pulling in of separate Git repositories, and also some of the more complex formatting of my new sci-fi fiction website.

Gemini

I like the idea of Gemini. It is a low-overhead protocol that has almost no extra features, no cookies, and basically focused on presenting content. In my case, I really want to see all of my sites on Gemini because I think it has some significant merits, more so as I want to get away from heavily styled content written by people who like tiny fonts or don't have my color contrast issues.

To do that, I ended up taking inspiration from md2gemini and wrote a C# library that converts Markdown into Gemtext (Gemini's markup format inspired by Markdown). The end result is pretty nice, I think and I'm really happy with the results.

Of course, it meant I had to get a virtual machine to host a Gemini server next to a HTTP one, but that was going to happen sooner or later anyways.

Next Steps

I write a lot libraries that I think are interesting but very few people worry about. They rise up, either I stick with them or I trail off, but they always scratch my itches. On that front, the following things are left to do:

Split out Gallium more formally into MfGames.Gallium or Gallium, not sure.
Clean up the API for Nitride and make sure everything is consistent. Some of the names are a bit… off, but the functionality is good.
Break all the modules into separate Gitlab projects and set up CI for releasing.
Document everything.
Convert five other websites sites to use it to figure out what is missing.

I haven't found the money to get a developer's signing certificate, so I'll probably just put everything up on my public MyGet repository.