Package Management

Package Management - Formats and Registries

2023-11-30T06:00:00Z

Since my mind has been on it, I wanted to work out some of the ideas I had for formats in my packaging system. In this case, I'm going to focus on a single one, NuGet, because I have a fair amount of experience with this and it has some of the complexities that are throwing me.

Series

This is going to be a series of posts, but I have no idea of how fast I'll be writing them out. I want to work out my ideas, maybe have a few conversations, and then start to move to more technical concepts.

2023-02-07 Package Management - Introduction - The beginning of a short series as I write down my ideas of an ideal package system with the future goal of writing a generic system called Bakfu.
2023-02-08 Package Management - Versions - A short discussion on packages and their versions.
2023-02-12 Package Management - Identifiers - A short discussion on how to identify packages.
2023-02-13 Package Management - Dependencies - A review of different styles of package dependencies, including across different ecosystems, and then an attempt to consolidate them into a single unified system.
2023-09-20 Package Management - Identifiers 2 - Using URNs to identify packages and some re-thinking of concepts.
2023-11-30 Package Management - Formats and Registries - Thoughts on setting up formats and layering registries for those formats on top of each other.

Configuration Files

All of the configuration for the system will be in a series of JSON5 (or JSON or whatever formats are officially supported) that are merged together with any conflict producing an error message and stopping the systems.

Assuming $GITDIR is the top-level directory for a Git repository, then the configuration would be in $GITDIR/.config/bakfu. All the files will be gathered together, but a .gitignore could ignore *.user.* which means authentication information could be stored locally but have a common configuration on top of that.

In this case, the files all have the same schema which is a required component because it also identifies the version of the file.

{
    "$schema": "...",
}

Any more details on how we look for configuration files will have to wait for another post.

What is a Format?

Right now, a “format” is a specific format of a package, such as NuGet package or a NPM one. Inspired by SGML catalogs and how I like to see things in Git repositories, a format looks roughly like this JSON5.

{
    // In this file, the various components don't have to be URL encoded.
    "formats": {
        "nuget": {
            "defaults": {
                "authority": "nuget",
            },
            "authorities": {
                "nuget": {},
            },
        },
    },
}

Since we merge files, that means I could create project-specific authority for packages that aren't (and probably never will be) on the official NuGet server. For example, my personal Forgejo instance at https://src.mfgames.com/mfgames-cil.

{
    // In this file, the various components don't have to be URL encoded.
    "formats": {
        "nuget": {
            "authorities": {
                "src.mfgames.com/mfgames-cil": {},
            },
        },
    },
}

Authorities

The authority itself is the complex part of the file because it needs to handle how to search for packages, how to download them, authorization needed, and how verification is done.

// merged formats.nuget.authorities:
"nuget": {
    // "enabled": true, // Implied so it can be disabled
    "registries": {
        "nuget.org": {
            "protocol": "nuget-v3",
            "url": "https://api.nuget.org/v3/index.json",
        },
    ],
}

Another file could merge additional registries in, much like you can have a proxy feed in DevOps.

// merged formats.nuget.authorities:
"nuget": {
    "registries": {
        "nuget.org": {
            "protocol": "nuget-v3",
            "url": "https://api.nuget.org/v3/index.json",
        },
        "example": {
            "protocol": "nuget-v3",
            "url": "https://example.org/proxied-feed/v3/index.json",
        },
    },
}

Controlling Order

In the NuGet.config file, there is also the ability to clear out the list of registries and use only a single set of identified registries. In this case, it would be a combination of disabling the known ones, changing the search for other files (the later post), and using a set of ordering controls.

// merged formats.nuget.authorities:
"nuget": {
    "registries": {
        "nuget.org": {
            "protocol": "nuget-v3",
            "url": "https://api.nuget.org/v3/index.json",
            "enabled": "false",
            // Below implies "search": { "after": ["example"] } }
        },
        "example": {
            "protocol": "nuget-v3",
            "url": "https://example.org/proxied-feed/v3/index.json",
            "search": {
                "before": ["nuget.org"],
            }
        },
    },
}

Additional Packages

In NuGet.config, it is possible to map a set of packages to a given URL, such as all MfGames* can only be found at a specific server. In those cases, that should be treated as a separate authority with its own set of registries (that may be a duplicate if it is also a proxy feed).

In the example below, contoso would be treated as a separate authority than the nuget default.

<!-- NuGet.config -->
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSourceMapping>
    <packageSource key="nuget.org">
      <package pattern="*" />
    </packageSource>
    <packageSource key="contoso.com">
      <package pattern="Contoso.*" />
      <package pattern="NuGet.Common" />
    </packageSource>
  </packageSourceMapping>
</configuration>

Protocol

The protocol determines how the registry is accessed. I could see a number of possibilities:

NuGet V3 protocol, obviously NuGet-centric
NPM access
Directory location
gRPC proxy server

Since this library needs to be implemented across a number of platforms and libraries, I would expect that unknown protocols would be filtered out (maybe with a warning) and then the ones that can be accessed are used. If there are no valid ones, then the system should blow up.

The protocol also determines what additional settings might be required such as authentication, file system layout, or the like. It would be obvious specific to that protocol, so it is thrown into a generic “settings” object to control those things.

"local": {
    "protocol": "file-v1",
    "url": "file:///${env:HOME}/src/other/project",
    "settings": {
        // The layout for "bob" would be "b/bob".
        "package": "${PACKAGE_NAME:0-0}/${PACKAGE_NAME}/${PACKAGE_VERSION}",
    },
},

Search Controls

Another aspect if how to handle searching. NuGet will search all the sources at the same time and the first one that responds gets it. However, in some cases, one might want only certain ones to be searched and then stop if it isn't there (such as a full feed proxy verses a subset proxy feed).

I would see this as controlled by two components, at the authority and for an individual repository.

// merged formats
"nuget": {
    "authorities": {
        "nuget": {
            "search": {
                "concurrent": true,
                "defaultOrder": "Alphabetical",
            },
            "registries": {
                "nuget.org": {
                    "protocol": "nuget-v3",
                    "url": "https://api.nuget.org/v3/index.json",
                    "search": {
                        "notFound": "Stop",
                        "timeout": {
                            "time": "00:01:00",
                            "action": "Retry",
                            "maximumRetries": 3,
                        },
                    },
                },
                "example": {
                    "protocol": "nuget-v3",
                    "url": "https://example.org/proxied-feed/v3/index.json",
                    "search": {
                        "notFound": "Continue",
                    },
                },
            },
        },
    },
}

Conclusion

Well, that's my thoughts on authorities and how to search them for packages. One thing you might notice is that I don't have offline packages in the above examples. I want to treat offline (cached) files as a first-class concepts in the packaging and so that requires its own discussion. Not to mention, the cache is for all formats, not just one.

I also want to eventually introduce the ability to have services provide opinions on packages. This way, I could set up a service that translates CVE alerts into controls of the packages found or allow a project-specific settings that would hide packages that were incompatible with the current project. I just don't know how to call them.

Computer science cannot solve two things: cache invalidation, how to name things, and off-by-one errors.

Package Management - Identifiers 2

2023-09-20T05:00:00Z

It's been a many (seven) months since I started working on the ideas of a package management system. A lot's happened but something got me down the path again and I decided to look at it again. This isn't a straight journey where I write every single post ahead of time but instead its more of ramblings and thoughts.

Or, in the words of one of my favorite puzzle games, The Thirtieth Guest:

Feeling lonely?

Series

2023-02-07 Package Management - Introduction - The beginning of a short series as I write down my ideas of an ideal package system with the future goal of writing a generic system called Bakfu.
2023-02-08 Package Management - Versions - A short discussion on packages and their versions.
2023-02-12 Package Management - Identifiers - A short discussion on how to identify packages.
2023-02-13 Package Management - Dependencies - A review of different styles of package dependencies, including across different ecosystems, and then an attempt to consolidate them into a single unified system.
2023-09-20 Package Management - Identifiers 2 - Using URNs to identify packages and some re-thinking of concepts.
2023-11-30 Package Management - Formats and Registries - Thoughts on setting up formats and layering registries for those formats on top of each other.

Mistakes Were Made

While I was reading the first identifiers post, I realized I made a few mistakes. One was that I was using a URL instead of a URN to identify a package:

In contrast, URNs were conceived as persistent, location-independent identifiers assigned within defined namespaces, typically by an authority responsible for the namespace, so that they are globally unique and persistent over long periods of time, even after the resource which they identify ceases to exist or becomes unavailable.

When it comes to identifying a package, that is exactly what we want. A persistent identifier that doesn't point to a specific location. When we want Markdowny, an important part is that we don't want to mandate where to get it, but enough to identify it.

Rehashing as URN Components

URNs always start with urn: and a registered “code”. We're going to pretend bakfu is the registered code, so that means all the package identifiers would be urn:bakfu: then something.

Likewise, I think it is important to know when something is a package identifier (say pkg:) or a reference to a package which has ranges (ref:).

EDIT: After looking at my notes, I already figured out why I didn't need this so I removed it from the content below.

From earlier posts, I decided on the package format being a component with well-known versions (npm, nuget, deno) and arbitrary ones (domain-based).

urn:bakfu:npm
urn:bakfu:minetest
urn:bakfu:authorintrusion.com/spell-check

In the later examples below, I'm going to cut off the urn:bakfu: as noise so npm.

Authority

I don't have a lot of problems or doubt with the components above. The next part, on the other hand is a bit more complicated and fluid. As I develop more, I think we should have separate package repositories/registeries instead of putting everything at npmjs.com or nuget.org. However, that leads into potential name and identifier conflicts.

Using the example from my life, when I started Nitride, I made all the namespaces “Nitride” and was going to buy a developer SSL to push it up to nuget.org. However, by the time I got to a stable point, there was then a Nitride already there and that didn't work.

(This is also one reason why I don't like identifiers that aren't namespaced.)

At the same time, I want to keep these URNs relatively “simple” for the 99% cases. In those cases, that means I want to aim for something like:

npm:markdowny
npm:@mfgames-writing/format
nuget:Humanizer

But, if there is a non-default location, the URN needs to have some mechanism that identifies the “authority” of a package. This authority doesn't need to be a URL, just a unique key to distinguish between two packages with the same identifier.

Originally I thought about something like npm:///markdowny based on using file:/// to reference the local file system but allow a domain and directory to be used:

npm://mfgames.com/markdown
npm://mfgames.com/@mfgames-writing/epub2
npm://example.org/~user/@example-organization/example-package

The problem with that is the last one. Where does the directory structure end, where does the package begin? If the entire URL is opaque, then it would be easy to leave as-is, but because this has to translate, there needs to be an unequivocal way of splitting them into an authority and a package identifier.

URL encoding to the rescue.

If we treat the optional directory structure (on the optional authority domain) as a single “unit”, then we can keep the slash to separate authority from the identifier but still keep the identifier in its most common format (NPM uses slashes):

npm:markdowny
npm:///markdowny (same as above)
npm://npmjs.com/@mfgames-writing/epub2
npm://npmjs.com%2F~dmoonfire/@mfgames-writing/epub2

I think this would work because we can have a rule that states that the component after the package type has either two slashes for an authority and the next slash ends that component. Everything after that is the full identifier.

Well-Known URLs

I'm fond of the .well-known/ infrastructure that has built up over the years. I could easily envision that this could translate into an actual URL to help identify the location of the packages if not known.

https://npmjs.com/.well-known/bakfu/npm/npmjs.com%2F~dmoonfire/@mfgames-writing/epub2

The resulting JSON file would give common locations where to find it. So going to the @mfgames-writing/epub well-known URL would then give the URLs for the official servers or locations, such as npmjs.com, my local package repository, an IPFS address, or whatever makes sense.

The reason it won't use query strings like webfinger is because query strings don't play well with static sites and I use static sites pretty heavily.

Qualified Identifiers

I think the ideas from the original identifiers post for qualified identifiers still have merit, but without the bakfu: prefix because it ends up just being noise. I think these should be limited and defined ahead of time since there is flexibility on the features.

java:org.example.hyphenated_name
npm:markdowny?version=1.1.0
cargo:serde?version=1.0.152&feature[]=derive&feature[]=rc
cargo:serde?version=1.0.152&platform=x86_64-unknown-linux-gnu

Additional Versions

If the package version (as opposed to the content version) is needed, then &package=1.0.0 can be used. Likewise, if the Bakfu itself needs to be bumped, then &bakfu=1.2.0 can be used.

I thought about making versions arbitrary, but I couldn't imagine a case where a package would have two different versions for two purposes. Those would be two separate packages in that case.

Overriding Packages

One of the reasons of this exercise is how to do a modification to a library that is already releases but the users learn after the fact that it breaks semantic versioning (SlimMessageBus). One constraint to this is that according to the specification, the version of the content cannot change once published.

That is why the package has its own version, to indicate that the package metadata such as the dependencies and requirements, can change independently of the contents. In most cases, package=1.0.0 but a proxy service could add in the modified dependency and call it package=1.0.1-service which would then cause the packaging system to prefer the highest version of the package with the same version.

There is some gaps because if we had a Bakfu-aware packaging system, someone could create a package and then keep bumping the package version higher to override anyone's overrides but I think this is a case where an upstream package modification should be blocked if there is an override given. At least until that new version can be reviewed and accepted.

Conclusion

The main reason to have these package identifiers is just to distinguish a package uniquely across the entire ecosystem. I strongly believe there needs to be a decoupling of the location verses identifier because of the other goals in this project: moving from one host to another, caching packages, being able to provide a curated list, blocking malicious packages, and to add after-the-fact changes.

Overall, I think this fits my need for something that is roughly ascetic (urn:bakfu:npm:markdowny?version=1.0.1), has a most-common use of something simple and readable (urn:bakfu:npm:markdowny), but still allows distributed packages and cases where there are name collisions (urn:bakfu:nuget://mfgames.com/Nitride).

It also can be reduced to a common form based on context such as removing the urn:bakfu: which makes the simplest version npm:markdowny.

Also, it shouldn't be hard to create a normalized rule to turn it into a proper C# or Rust structure for doing the next steps.

Package Management - Dependencies

2023-02-13T06:00:00Z

While packages can be self-contained, usually they have dependencies with other packages or even frameworks to avoid duplicating code. Without dependencies, packages would be forced to contain common logic that other packages share, and that code may become stale or not updated as frequently introducing potential bugs.

Series

2023-02-07 Package Management - Introduction - The beginning of a short series as I write down my ideas of an ideal package system with the future goal of writing a generic system called Bakfu.
2023-02-08 Package Management - Versions - A short discussion on packages and their versions.
2023-02-12 Package Management - Identifiers - A short discussion on how to identify packages.
2023-02-13 Package Management - Dependencies - A review of different styles of package dependencies, including across different ecosystems, and then an attempt to consolidate them into a single unified system.
2023-09-20 Package Management - Identifiers 2 - Using URNs to identify packages and some re-thinking of concepts.
2023-11-30 Package Management - Formats and Registries - Thoughts on setting up formats and layering registries for those formats on top of each other.

Simple Dependencies

For most systems, dependencies are very simple: this package requires these packages in this range. When working with Bakfu, this is almost identical to the package identifier but without the bakfu:version and instead a range of values that are acceptable.

Using MfGames.Nitride?version=0.15.1 as an example, here is a fragment of what that could look like:

{
    // Basically this is "bakfu:nuget:MfGames.Nitride?framework=net6.0"
    type: "nuget",
    id: "MfGames.Nitride",
    attributes: {
        "framework": "net6.0",
    },
    content: {
        version: "0.15.1",
        dependencies: [
            "bakfu:nuget:Autofac?framework=net6.0": "[6.4.0, 7.0.0)",
            "bakfu:nuget:FluentValidation?framework=netstandard2.1": "[11.2.1, 12.0.0)",
            "bakfu:nuget:MfGames.Gallium?framework=net6.0": "[0.4.0, 0.5.0)",
        ],
    },
}

Replacements and Substitutions

While the above works in most cases, there are situations when it doesn't. Probably the easiest one to explain in in Minetest with multiple mods that replace the built-in beds module: TenPlus1's and SorceryKid's. In all three of the cases, they are installed as beds which is used for the dependency, but they are distinguished enough for our package system despite the same name.

bakfu:minetest:beds
bakfu:minetest:TenPlus1/beds
bakfu:minetest:sorcerykid/beds

In the C# world, I've encountered this with the BouncyCastle libraries. In specific, we have:

Both of these libraries have a 1.8.9 version and both of them have exactly the same DLL once it is build, BouncyCastle.dll. In the past, this has caused us a lot of problem because while the two versions are API compatible (they have the same application binary interface, or ABI), they also overlap each other and have different packaging systems (one is only for .NET Framework, portable is more universally usable and currently maintained).

Likewise, if someone abandoned a package in a fit of anger, say the leftpad drama in the JavaScript, a strict package identifier means a replacement cannot be find. The same thing happens if a project maintainer dies without planning for a successor, there is a schism in the community causing a soft- or even hard-fork, or even someone using a Git repository or local copy to add some customization they need. Regardless, there are reasonable reasons for having two packages that proclaim to be the same version.

To handle this, packages should list the aliases of what they provide (including themselves if needed) and that is what dependent packages would use. Using the BouncyCastle for an example:

// BouncyCastle
{
    content: {
        id: "BouncyCastle",
        version: "1.8.9",
        attributes: {
            "framework": "net",
        },
        provides: [ "bakfu:nuget:BouncyCastle?bakfu:version=1.8.9&framework=net" ],
    },
}

// Portable.BouncyCastle
{
    content: {
        id: "Portable.BouncyCastle",
        version: "1.8.9",
        attributes: {
            "framework": "net40",
        },
        provides: [
            "bakfu:nuget:Portable.BouncyCastle?bakfu:version=1.8.9&framework=net40",
            "bakfu:nuget:BouncyCastle?bakfu:version=1.8.9&framework=net",
        ],
    },
}

// A third package
{
    content: {
        id: "Something.Else",
        version: "1.0.0",
        requires: {
            "bakfu:nuget:BouncyCastle?framework=net": "[1.8.9,2.0.0)",
        },
        // A missing provides would just produce the identity package
        // in this case, it would imply:
        // provides: { "bakfu.nuget:Something.Else?bakfu:version=1.0.0" }
    },
}

In the above example, if a module doesn't have a provides element, then we could assume that it is just the identity version. “Something.Else” version 1.0.0 becomes bakfu.nuget:Something.Else?version=1.0.0.

Publish and Subscribe

The above example doesn't handle optional dependencies. For example, Minetest modules can check for the presence of other modules and then hook up to them. Atom (one of my favorite editors that was sadly killed) had the idea of services where a package could provide a service (that had a version) and other packages could “consume” them. Using a fragment of my own spell-check, it looked like this:

// from spell-check
"consumedServices": {
  "spell-check": {
    "versions": {
      "^1.0.0": "consumeSpellCheckers"
    }
  }
},

// from spell-check-project
"providedServices": {
  "spell-check": {
    "versions": {
      "1.0.0": "provideSpellCheck"
    }
  }
},

In the above case, the editor would call use the callback method for every package that “provided” a service which would return an opaque object, which would then be handed to spell-check consumed callback (consumeSpellCheckers) as an array. That allowed optional dependencies by publishing a service and then having other packages subscribe to them.

Even if the publisher (spell-check in the above example) would get every subscriber, then doesn't mean that the publisher then could call something back into every subscriber that would let you wire the two together to allow for a reverse- or even bi-directional flow of data.

Since this also has system-specific elements, we use the same attributes construct to give flexibility to extend outside of Bakfu.

{
    content: {
        id: "spell-check",
        version: "0.77.1",
        publish: {
            "bakfu:atom-service:spell-check": {
                accept: "[1.0.0,2.0.0)",
                attributes: {
                    callback: "consumeSpellCheckers",
                },
            },
        },
    },
}

{
    content: {
        id: "spell-check-project",
        version: "0.7.2",
        subscribe: [
            "bakfu:atom-service:spell-check": {
                accept: "[1.0.0,2.0.0)",
                attributes: {
                    callback: "provideSpellCheck",
                },
            },
        ],
    },
}

How the application handles those services isn't in the domain of Bakfu, just the gathering and providing the information that links the two. It would also be based on the application if there can be multiple conflicting publishers (can two packages both publish the same service and version?) or not.

Categorizing Dependencies

The last major topic to cover for this post is how to categorize dependencies. NPM and C# both have the concept of development and production dependencies. NPM is a little easier to work with:

// package.json
{
    dependencies: {
        "package-1": "1.2.3",
    },
    devDependencies: {
        "package-2": "4.5.6",
    },
}

Rust is more complicated because enabling a feature also changes the dependencies of the package. Many crates don't include serialization, but if you add the serde feature, it would include serde and any other requirements. In effect, this is an optional dependency but determined at build item.

It also changes how we organize dependencies above by adding an extra layer inside the requires element that provides an application-specific set of keys. In Rust, this may also include every feature as a separate element of dependencies.

(I could have started with this, but I wanted to build the blocks and work out the ideas on a single dependency before getting into the more complex situations.)

{
    content: {
        id: "Something.Else",
        version: "1.0.0",
        requires: {
            production: {
                "bakfu:nuget:BouncyCastle?framework=net": "[1.8.9,2.0.0)",
            },
            development: {
                "bakfu:nuget:DoesNotValidateInput": "[1.0.0,2.0.0)",
            },
        },
    },
}

Another reason to have this categorization is to allow us to identify what packages are vulnerable in an intelligent mannner instead of just assuming any and all vulnerabilities are important. In other words, if you trust your developers not to use a DDoSing regular expression in their code, you don't have to patch that code.

Expanding Dependency References

One of the areas I'm not sure about is now to describe dependences and provides. In the above examples, I'm using the URI format but my gut feeling is that it should be the more verbose form to allow for extensions.

The previous example would then look like this:

{
    content: {
        id: "Something.Else",
        version: "1.0.0",
        requires: {
            production: [
                {
                    type: "nuget",
                    id: "BouncyCastle",
                    accept: "[1.8.9,2.0.0)",
                    attributes: {
                        framework: "net",
                    },
                },
            ],
            development: [
                {
                    type: "nuget",
                    id: "DoesNotValidateInput",
                    accept: "[1.0.0,2.0.0)",
                },
            },
        },
    },
}

I'm just not entirely sure about which one is the “best” but I'm also trying to work out ideas for a complete system, so I will probably use the URI unless the expanded makes sense. Ultimately, I really won't know until I've gotten through all the analysis and ideas before I can say one way or the other.

(Though, I do lean toward more strict schema which means that the expanded form will probably be the final one.)

I'm also not sure if the Bakfu format really needs to have bakfu: in front of identifiers. I mean, it is inside the “bakfu” file.

Package Management - Identifiers

2023-02-12T06:00:00Z

The other really big topic when it comes to package is how to identify them. This is a lot more complicated because I view packages in a polyglot manner instead of a single foreign ecosystem.

Series

2023-02-07 Package Management - Introduction - The beginning of a short series as I write down my ideas of an ideal package system with the future goal of writing a generic system called Bakfu.
2023-02-08 Package Management - Versions - A short discussion on packages and their versions.
2023-02-12 Package Management - Identifiers - A short discussion on how to identify packages.
2023-02-13 Package Management - Dependencies - A review of different styles of package dependencies, including across different ecosystems, and then an attempt to consolidate them into a single unified system.
2023-09-20 Package Management - Identifiers 2 - Using URNs to identify packages and some re-thinking of concepts.
2023-11-30 Package Management - Formats and Registries - Thoughts on setting up formats and layering registries for those formats on top of each other.

Previous Updates

While I got to thinking (this is a living series), I didn't like the upgrade format in the versions post of this series. In there, I suggested this for Bakfu:

{
    content: {
        version: "1.10.0",
        majorUpdate: "[1.10.0, 1.15.0)",
        minorUpdate: "[1.10.0, 1.11.0)",
    },
    package: {
        version: "2.0.0",
    }
}

I think putting the update rules in their own category makes sense since they related to each other but also to handle adding other rules as needed in there. I thought about putting everything in a version, but then I would have {version: { current: "1.2.3" }} which didn't feel right.

We also should have the package format version.

{
    bakfu: {
        version: "0.0.1",
    },
    content: {
        version: "1.10.0",
        update: {
            major: "[1.10.0, 1.15.0)",
            minor: "[1.10.0, 1.11.0)",
        },
    },
    package: {
        version: "2.0.0",
    }
}

Identifier Schemas

A package identifier is one of those things that varies greatly between different languages:

NPM: markdowny, @mfgames-writing/format
NuGet: MfGames.Nitride, System.Collections.Generic
Minetest: beds, moreores
Java: com.mfgames.ruby (an old game of mine)
Go: github.com/callicoder/packer/numbers
Cargo: serde
Atom: spell-check

That gives a pretty good overview of most languages that use packages. I don't think there are many other ways of referencing them. In addition to the ones above, I want to be able to allow anyone to create an arbitrary package ecosystem because I don't need to have a game using my calendar's packages (unless it wants to) but they are distinct.

Package Groups and Organizations

I will be up front, I don't like flat package systems such as Node and Cargo. They have a significant problem with name squatting (where someone “reserves” a package name that is ideal) and also because many of the packages are generic (Minetest's beds or Node's time) even if they are as capable.

If you notice, almost all of my packages start with mfgames or MfGames. I do that to avoid conflicts with other packages. I thought I wouldn't for Nitride but when I finally created the packages, someone had put one up on nuget.org and I promptly turned it back to MfGames.Nitride. At this point, if someone is using MfGames. in their packages, it's intentional (and probably malicious).

Scoped packages are better, more so if there is a way of controlling who can use that scope. On nuget.org, you can reserve a prefix which is how they prevent someone from uploading a Microsoft.HackYourComputer package. In theory, I could do the same with MfGames but it requires some dependencies that I don't have lying around (cash).

That said, I don't like NPM's @organization because they also use @1.3.2 for versions. So, then you have package and version identifiers as @mfgames-writing/format@3.4.0. It isn't “pretty” and I am frequently driven by what I think looks good. As such, I'm inclined to avoid the @ whenever possible.

Polyglot Identifiers

pol·y·glot /ˈpälēˌɡlät/ -> adj. knowing or using several languages.

When working with code, not only do we use different packages, we use packages that require entirely different systems. Probably one of the most common one for me is the bane of my JavaScript and TypeScript development: node-gyp. Node-gyp requires a C compiler and Python to run, but it is for Node packages. So, that means that if we really want to identify a package (and its dependencies, a later post), we need to be able to address all of those dependencies in the system addressing scheme.

Uniform Resource Identifiers (URIs)

Since I want to be able to identify packages by something that jumps languages, it seems like a perfect use of a URI. In short, a URI looks like this:

URI = scheme “:” ["//" authority] path ["?" query] ["#" fragment]

The most common one we see are URLs:

https://fedran.com/flight-of-the-scions/

In the above example, https is the scheme, //fedran.com is the authority, and flight-of-the-scions/ is the path. If we take the same idea, we could create a pseudo version of package identifiers that use the same thing.

(Side note, Humanizer is an awesome package for C# and everything Charm.sh is very pretty. Markdowny is my own tool for working with Markdown + YAML files, so I think it is fantastic.)

This works for well-known package systems, but what about arbitrary ones? Say I create something for Author Intrusion, I could use authorintrusion:spell-check but that would require registering a URI if I want to avoid conflict. A more efficient way (that I also will help with some future ideas) is to prefix them with bakfu: instead and then allow domain or scoped registrations for non-well-known ecosystems.

bakfu:npm:markdown
bakfu:go:github.com/charmbracelet/lipgloss
bakfu:authorintrusion.com/spell-check (I really should do something about that website.)

Well-Known Systems and Aliases

Sadly, this does assume a centralized repository for the well-known (e.g., npm, go, nuget) packages. It also leans into DNS instead of using other addressable schemas or pet names, I could also use dotnet.microsoft.com for nuget (since that is the general packaging system), nodejs.org, and go.dev for those languages.

If I do go with the well-known, I would also have to establish a extension process that would allow communities to introduce new aliases or “register” their own to avoid conflict. While that isn't something I'd want to do, if this entire idea goes beyond me, it would be the first thing I would establish because I like to play well with others.

I would also call it BEEP - Bakfu Emerging Extension Process so then I could have BEEP-0000 be the first one.

Qualified Identifiers

Using the URI, we can use URL-style query parameters to create a key/value pair for additional details about the package such as Cargo's features, .NET's frameworks, or whatever variants are important to include.

For ones used by Bakfu, using a XML-style prefix would be more clear than either having a name collision (version= could be common) or an arbitrary character prefix (_). In these cases, bakfu:version would be useful and allow for further extensions.

For keys that need multiple, such as crate's features, the key would be suffixed by [] to indicate an array (“Ruby style” is what I've heard this called).

Rust has both platforms and featuers:

bakfu:cargo:serde?bakfu:version=1.0.152&feature[]=derive&feature[]=rc
bakfu:cargo:serde?bakfu:version=1.0.152&platform=x86_64-unknown-linux-gnu

NuGet has frameworks:

bakfu:nuget:Autofac?bakfu:version=6.5.0&framework=netstandard2.0
bakfu:nuget:Autofac?bakfu:version=6.5.0&framework=net6.0

Given that, and the future discussion on dependencies, I think it would make sense to make it a query parameter instead of using the # fragment. Also, some foreign ecosystems don't have versions (Minetest, at least one of the Python systems) which means even a version is optional.

Expanded Form

To simplify coding, the above URL could easily be broken down into a standard JSON structure for less condensed format with the “bakfu:” attributes pulled up a level.

{
    // bakfu:cargo:serde?bakfu:version=1.0.152&feature[]=derive&feature[]=rc&platform=x86_64-unknown-linux-gnu

    type: "cargo",
    id: "serde",
    version: "1.0.152",
    attributes: {
        "platform": "x86_64-unknown-linux-gnu",
        "features": ["derive", "rc"],
    },
}

At the moment, I'm going to use both formats but I lean toward the more expanded form in actual files with the condensed URI form when trying to explain things. I'm sure I will bounce back and forth between something that feels “right”. You may also notice that I spell out most things because I don't like abbreviations while communicating concepts.

Distributed Packages

This is for a future topic, but I am planning on addressing distributed packages that don't encourage using only a single source for the data. So, the URI above only addresses the framework-level components that identify a distinct package irregardless if you find it on nuget.org, npmjs.com, gitlab.com, or even src.mfgames.com.

Package Management - Versions

2023-02-08T06:00:00Z

The start of almost every package journey starts with versions. This is how one distinguishes between different copies and hopefully makes sense of when to get the latest version verses only updating a few or not updating at all.

Series

2023-02-07 Package Management - Introduction - The beginning of a short series as I write down my ideas of an ideal package system with the future goal of writing a generic system called Bakfu.
2023-02-08 Package Management - Versions - A short discussion on packages and their versions.
2023-02-12 Package Management - Identifiers - A short discussion on how to identify packages.
2023-02-13 Package Management - Dependencies - A review of different styles of package dependencies, including across different ecosystems, and then an attempt to consolidate them into a single unified system.
2023-09-20 Package Management - Identifiers 2 - Using URNs to identify packages and some re-thinking of concepts.
2023-11-30 Package Management - Formats and Registries - Thoughts on setting up formats and layering registries for those formats on top of each other.

The Semantic Dream

These days, almost every package system proclaims that they use semantic versions and write the infrastructure as such. I love semantic versions with a passion, including putting them in my novels and stories. They are relatively clear and concise, and easy for me to understand: breaking-things.adding-things.fixing-things.

The problem is, semantic versions are hard to manage when it comes to packages and dependencies, not to mention the business practices.

Romantic Versions

The biggest one are the ones who won't use it. One of the library I use during my day job is EasyQuery because they have a .NET and a JS library and give me a lot of flexibility in creating dynamic queries. They also are responsive to bug issues. What they don't do is follow semantic versions. They use romantic versions with every x.y version being a breaking change from the previous one. This applies to their code on NPM and NuGet.

It is easy to manage, but you have to know that you can't go from 7.0.2 to 7.2.0 without major work, though semantic version says that it should have been a safe change.

My day job product is also romantically versioned, so I deal with this on a quarterly basis.

Dependency Changes

Another example is SlimMessageBus.Host. In version 1.14.1, it requires “.netstandard2.0”. In 1.15.0, it is “.netstandard2.1”. While this seems like safe change, it was a minor version bump that made it impossible for us to update because our product is still bound to “.netstandard2.0” until we can get off WebForms.

Version Ranges

Since it is important, there are also a number of ways of communicating version ranges. Node uses the ^ for semantic versions and ~ for “only patch updates” (basically what EasyQuery uses).

NuGet uses a different system such as [1.0.0, 1.15.0) with the [] meaning inclusive and () meaning exclusive. This is my preferred system, mainly because it hearkens back to my computer science days and allows a mid-version break such as SlimMessageBus. In that regard, it is more flexible because ^ and ~ can translate directly into the verison range.

Being Explicit

I haven't gotten to dependencies yet, but the developer knows when something is safe update a package verses when it is not. EasyQuery knows that a patch is the breaking point, it would be better that they be able to communicate that as part of their metadata instead of requiring a customer support call. In effect, using the first two semantic version components (major and minor), EasyQuery 7.0.0 would be:

{
    version: "7.0.0",
    majorUpdate: "[7.0.0, 7.1.0)",
    minorUpdate: "[7.0.0, 7.1.1)", // The third number is their minor versions
}

On the other hand, SlimMessageBus would use (using an older version for illustrative purposes):

{
    version: "1.10.0",
    majorUpdate: "[1.10.0, 1.15.0)",
    minorUpdate: "[1.10.0, 1.11.0)",
}

I'm assuming that anything more precise is a patch and is safe to update. I'm also think Bakfu should lean heavily on well-defined undefined values. So if majorUpdate, then assume semantic version. If minorUpdate is missing, then use majorUpdate (which may be assumed).

Package Versions

The last bit is one that I've encountered a number of times while packaging books, but also while I was doing Debian packages. The version of the package should be independent of the content. Using the above example, if SlimMessageBus realized they had made a breaking change, they should be able to update the 1.14.0 version of their package to indicate that 1.15.0 is a major breaking change.

Today, their only choice would be to remove the 1.15.0 (and any later) version and create a 2.0.0 version. This can be a lot of work because most likely someone realized it after there were a number of updates already out in the field.

Instead, if the package version was decoupled from the content version, then it would be possible to retroactively update critical information such as dependencies and safe upgrade ranges. Using SlimMessageBus again:

{
    package: {
        version: "1.0.0",
    },
    content: {
        version: "1.10.0",
    },
}

After they realize the breaking change:

{
    package: {
        version: "2.0.0",
    },
    content: {
        version: "1.10.0",
        majorUpdate: "[1.10.0, 1.15.0)",
        minorUpdate: "[1.10.0, 1.11.0)",
    },
}

Package Management - Introduction

2023-02-07T06:00:00Z

As I'm working through one project, I'm frequently planning the next. I noticed that a number of my future ideas, fantasies as you will, all have one thing in common: packages. It doesn't if it is a game idea or working on programming calendars or Author Intrusion, they all need some form of package management.

Given that and the rule of three, it makes sense to sit back and working that problem independent of the individual projects. Some people call this procrastinations. I like to call it composition. After a few months of rolling it around in my head, plus a bit of thoughts from Drew DeVault's various posts on the topic, I figured I'd codify my ideas and see what I can do to create something I'm happy with in the future.

Series

This is going to be a series of posts, but I have no idea of how fast I'll be writing them out or if it will pan out, but this is a list of the related posts. I want to work out my ideas, maybe have a few conversations, and then start to move to more technical concepts.

2023-02-07 Package Management - Introduction - The beginning of a short series as I write down my ideas of an ideal package system with the future goal of writing a generic system called Bakfu.
2023-02-08 Package Management - Versions - A short discussion on packages and their versions.
2023-02-12 Package Management - Identifiers - A short discussion on how to identify packages.
2023-02-13 Package Management - Dependencies - A review of different styles of package dependencies, including across different ecosystems, and then an attempt to consolidate them into a single unified system.
2023-09-20 Package Management - Identifiers 2 - Using URNs to identify packages and some re-thinking of concepts.
2023-11-30 Package Management - Formats and Registries - Thoughts on setting up formats and layering registries for those formats on top of each other.

Package Systems

Packages are awesome because they allow over developers provide components, they split apart logic into smaller chunks which make it easier to understand, and they can have different update paths instead of needing a single monolithic deployment. This also means, I'm more likely to be able to focus on a small part instead of having to “swap in” an entire project. That is one thing that has made Nitride a lot more enjoyable to work with.

At their core idea, a package is just a “blob” of something along with metadata. For purposes of this series, I don't care what is in the blob. It could be C# code, a WASM binary, or Rust source. The thing I've gotten hung up is that we have a ton of different package systems (e.g., LISP modules, Python packages, NPM, Minetest, Debian packages) and each one was written for a specific use and has different rules. Some are nicer on the disk, others support signed code, and some have methods to avoid squatters. Every community wants to write their own for their specific uses, but to me it feels like we are reinventing the wheel so many times when we could be focusing on more fun things (unless packaging systems is your idea of fun, it is mine).

(Yes, I'm aware that I want to write my own for my own specific purpose, but that's going to happen anyways. It's just a matter of what framework I write it in.)

Terminology

Since I'm thinking about doing a series of posts, here are some terms I plan on using:

Application - Whatever needs a package, be it a game, library, or utility.
Bakfu - My hypothetical system for managing packages. This is the Lojban word for “package” and seems like a decent handle for what I'm working for.
Foreign - Another packaging system.
Package Ecosystem - All the servers, tools, and the packages themselves for a given type of packaging. This includes bakfu and foreign packaging ecosystems.
Package Server - The servers used to query and distribute packages.
Package - A specific type of package. This will have bakfu, application, or foreign as an adjective when necessary.
Package Content - The “thing” being packaged. This can be a set of files, a word list, assets for a game, or even a foreign package. For purposes of these posts, this is a generic “blob” of data that is specific to an application.

Bakfu

I want something generic. More specifically, I want to find what I consider are the “best” parts of the various systems I've worked with, solve problems that I've experienced, and maybe come up with a set of libraries and utilities that let me implement a packaging ecosystem that only requires me to override the application-specific components while delegating the rest of the ecosystem to bakfu.

I have no intent on this being a “standard” because everyone likes to reinvent the wheel. Everyone has their own idea of what is the “right” way of handling packages, even when there is a defacto standard already implemented.

That isn't to say this is only going to be for me. Like most of my work, I'm hoping to create something generic enough that others can use it for any purpose and in the languages of their choice and still get some of the benefits I envision.