I'm giving myself a week to focus on something that I've been trying to do for a while, come up with a generic package management library. This of course brings me to the first thought: don't. Actually someone else said it better and with a lot more detail why this is a terrible idea. There are many other blog posts and warnings out there.

And yet I'm still going forward with writing a package management library: Bakfu.

Purpose

The main reason is that I see a need to have package management in a number of projects that are on my plate or I would like to be on my plate. While I could write a dedicated one for each one, I'm loathe to spend that much effort when I can spent far more effort coming up with something generic to separate the various concerns. Ultimate, this is the XKCD's General Problem.

The projects I have in mind for needing a plugin infrastructure are:

• MfGames.Culture: My library for handling arbitrary calendars
• Author Intrusion: My CLI-based tools for working with writing
• Some vague game ideas in my head for mod support

To be honest, Author Intrusion has stalled out years ago, but I keep wanting to write it. The culture library, on the other hand, is on my mind because it is the largest identifiable “rock” causing my writer's block. I haven't been able to edit Second-Hand Dresses because I got stuck on the calendar of events. It's a minor thing, but I really liked it when Fedran had the master timeline and per-character timelines also.

MfGames Culture

My current thoughts for the culture library is to implement the calendars as WASM plugins that let me take the calendar components (e.g., Gregorian) and merge it with the cultural elements (English, French, etc) and allow the conversion between them. If I can, I'd implement each of my calendars as a separate plugin and let me evolve them. (There are a number of more, plus I use the Japanese “small seasons” in my world and a rough time of day also.)

Many of my chapters have a date associated with them. This was to help me identify the dates when certain events happened, but also to make sure a character has enough time to get from one place to the other (given that I don't have a map of the world yet).

---
when:
  start: 1471/3/28 MTR 4::22
  duration: 0::30
---

My original use was to parse the date (1471/3/28 MTR 4::22) into a common format (Julian Days or JD), then use that value to calculate the date in all the calendars I use. This worked out nicely since I could create a tooltip that showed the date on the website.

The original culture library was written in Typescript, which worked fine when I was generating my website with CobblestoneJS, but now that I'm using MfGames.Nitride, which is in C#, that is less useful, hence the rewrite since I dislike opening a pipeline to a CLI to generate websites (so slow…).

But the main part is that I don't want to package all the calendars and cultures in a single file. I want to be able to make one just for Fedran, one that covers US information, or allow someone to pick the minimum set they need for a given project. That leans toward a plugin infrastructure where I can have a file like package.json that lists the calendars needed for that project.

Reevaluation

The eighteen months I've been working on the ideas, mostly trying out a few things and failing a lot. I started a Rust implementation, switch to C# to prototype faster (as a more comfortable language for me). But I want to finish this and move on, so that is why I'm spending this time to hopefully get a working package manager library.

I hadn't reviewed my goals in a while and I was suffering with scope creep.

I realized that this morning when I read the article Distributing WebAssembly components using OCI registries. In specific, I was reading it hoping that the OCI Containers would handle the problem and I could abandon my project entirely (the best goal, to be honest). To be honest, WASM OCI Containers are perfect for my plans for MfGames.Culture, but there is something missing.

That is when I realized I was trying to figure out a container format and also figure out an abstracted dependency system.

Containers

The container format isn't really the important part even though I thought itw as. There are already good formats out there: OCI Containers, zip archives, NuGet packages.

With OCI Containers (specifically Docker images), a developer would curate a set of containers and compose something, but they don't worry too much about updating them or having them move around. Yeah, they could update their tag to :latest, but that is fraught with troubles if my experience updating with the Docker version of Nextcloud was any indication. Too many times having to walk through the updates, one major version at a time with occasionally manually updating.

The composition is also part of the problem. Composing services is awesome for packages, but I would consider a rather technical task. As opposed to someone going into a list of game mods, checking a bunch, and then expecting it to run. Getting that list of mods (which could be OCI containers) isn't provided in a generic manner, including the base libraries (such as UI injection library or a parsing framework) as needed, and then making it work isn't in the scope of OCI Containers. As far as I can tell, it isn't in the scope of anything at a generic level. Everyone seems to reinvent the wheel, or retrofits an existing wheel like JetBrains did with their module system (from my understanding).

Abstraction

Abstraction is something important to me. Libraries evolve, not just from refactoring code or being built on a different pattern, but how they are managed. Maintainers come and go. Sites come and go, though the development world would be in chaos if GitHub ever went away.

But, at one point, SourceForge was thought to be the same. And now, it is barely there as nothing more than a download repository with excessive advertising.

Also, the project owners change. Sometimes they just throw up their hands and delete everything. Sometimes, they just retire. A good example is me. A long time ago, I used to maintain csharp-mode, the C# mode for Emacs. After a while, I got burned out trying to implement generics and wandered off; someone else offered to take the reins and I happily gave it to them.

Another example is the friction that comes when one part of the organization is moving slower than the other. I think BouncyCastle is an example of that. There are three libraries that do roughly the same thing, but each one is different based on their platform (.NET Framework verses .NET Core), package name (BouncyCastle verses Portable.BouncyCastle), and evolution (BouncyCastle.Cryptography is the latest). But in this case, I can usually manually change all references from BouncyCastle or Portable.BouncyCastle to BouncyCastle.Cryptography and have it work.

In the above case, it would be a lot easier if I could say all references to BouncyCastle or Portable.BouncyCastle could be addressed to BouncyCastle.Cryptography and not have to manually edit older packages that don't know about the new library.

There is also remixes and forks. In Minetest, there are numerous “beds” plugins that don't play with each other but are dependencies of others. In many cases, I just have to pick one of them.

It isn't only just the package names that need to be replaced. Many modern languages allow you to pull in a package from a Git repository (if I'm lucky, it doesn't have to be GitHub). NPM allows it, Rust does, I'm pretty sure Go also does. This is great because a developer can pin a specific hash to the version.

But that is also fragile. What happens if the developer deletes the repository? Or changes their name. Suddendly dmoonfire/some-project doesn't work but the person using that library isn't maintaining their project either, so I have to figure out how to get a second- or three-degree reference to point to somewhere new like my local copy. This is basically the same problem as the BouncyCastle example above.

There is no abstraction, which is what I think is important. I want something like this:

• PackageA depends on PackageB
• PackageC provides/substitutes as PackageB

That way, if “PackageB” in the example goes away, then “PackageC” can takes it place without having to rebuild everything. In a more concrete example:

• MyProject depends on BouncyCastle
• Portable.BouncyCastle substitutes BouncyCastle
• BouncyCastle.Cryptography substitutes as BouncyCastle

For locations, the same thing, I want to be able to say that a specific package can be found “here” or “here” or “here”.

Annotation

Security vulnerabilities are a still a big thing for me. I noticed NuGet is starting to download a list of vulnerabilities and report them in Visual Studio.

I want to expand that concept into something more generic, like Bluesky has with their distributed moderation. This is also something I'd love to see with Fediverse servers.

Being able to subscribe to a service that validates libraries and looks for hacks or does code auditing (and gets paid for it) is something I think this world needs. I could also see it as a local project that prevents certain packages from being viewed (like our problem being unable to move past netstandard2.0 but a minor version jumped to netstandard2.1). Or a curated list of assemblies for a project that is checked at the project root.

I use EasyQuery as an example. I love the library and what it does for my day job, but they use the minor version for breaking changes but there is nothing in NPM that tells me that. So I have a wiki page to developers to tell them, but an annotation service could say “breaking changes on minor versions” (they aren't the only one who doesn't honor sematic versioning).

Or also “here is a local cache of the package.”

Final Thoughts

This is just a rehash of what I've said previously, but for my needs. I need avoid scope creep and focus on what I need if I have any hope of finishing.

Obviously, finding that someone else has solved these problems would be awesome. Then I could abandon this project. Until then, I'm going to plug at it and see if I can get something minimal that works for getting me my calendars back.