Project Layout

A Week of Dependencies plus Nor Curse Be Found 13

2023-01-29T06:00:00Z

I'm trying to get get back to the one-week iterations/sprints for my personal projects. As one of my goals for the year, I want to write at least one short story or a chapter a month. “At least” being the goal since I want to write more, but writing Fedran is still a struggle because of burnout, entanglement, and just life.

Markdowny

So I dedicated this week to writing chapter thirteen of Nor Curse Be Found, but in the process of setting it up, I finally hit the point where bugs in Markdowny were too much for me. The main one is that I used the wrong shebang in the scripts but didn't realize it until I switched to NixOs. The key part is that I need to use /usr/bin/env bash for all my shebangs instead of assuming the path to bash.

# !/usr/bin/env bash

So that meant I needed to go into that project to fix it (and add a few features I've been wanting). But that was so old that I spent a few days bringing it up to fix my new project layout so I had a consistent environment. While doing that, I ended up on yet another tangent to create a script that could create my script files. (Not sure where to store that, to be honest.)

But then I finished the script to get the layout working to get Markdown working. After another day or so to coerce markdowny into Woodpecker CI, I was finally ready to actual write a feature I've been needing for a while.

New Feature

To submit entries to my writing group, I need to post a “events so far” document because it could be weeks or months between submissions. I use markdowny to do that with the list command:

$ markdowny list chapters/*.md | head -n 5
1. Always Moving: While Linsan waits for her mother to come home, she bounces around on the furniture and talks to her father. She announces that she has named a violin her father is making Palisis and learns that the violin is for her father's first wife who got married to her mother's best friend.

2. Early Lessons: Years later, Linsan is learning how to play the violin from her father. The lesson is interrupted when Dukan, her father's best friend and manager for the business, visits in a panic to tell him that the family's workshop in the valley is on fire.

3. Home Early: Unable to visit the burnt remains of the family's workshop, Linsan comes home to find her father work in depression. He had given up working on instruments and switched to writing articles about music. She goes into the attic to put some books away and finds Palisis in a corner, returned after Marin's death. She plays it, but then finds out that no one had ever played it before.

I needed a bit more details, so I decided to expand it to use Handlebars to give me a template for the chapters and to access all of the components inside the YAML header.

$ markdowny list ~/src/fedran/sources/allegro/chapters/*.md --template='- Chapter {{_number}} - {{{title}}} => {{{summary}}}' --trim-whitespace | head -n 5
- Chapter 1 - Always Moving => While Linsan waits for her mother to come home, she bounces around on the furniture and talks to her father. She announces that she has named a violin her father is making Palisis and learns that the violin is for her father's first wife who got married to her mother's best friend.
- Chapter 2 - Early Lessons => Years later, Linsan is learning how to play the violin from her father. The lesson is interrupted when Dukan, her father's best friend and manager for the business, visits in a panic to tell him that the family's workshop in the valley is on fire.
- Chapter 3 - Home Early => Unable to visit the burnt remains of the family's workshop, Linsan comes home to find her father work in depression. He had given up working on instruments and switched to writing articles about music. She goes into the attic to put some books away and finds Palisis in a corner, returned after Marin's death. She plays it, but then finds out that no one had ever played it before.
- Chapter 4 - Solace in Memories => As Linsan frequently did, she visited the family's ruins after school. The spot gave her peace despite everything they had lost. However, a bully from school, Dukan's daughter Brook, follows after her and they fight. During the brawl, they both manifest their powers: Linsan with music and Brook with concussion powers.
- Chapter 5 - Bitter Partings => Linsan comes limping home after her fight with Brook. Her parents are surprised she is there, but then Dukan and Brook show up. Dukan has his daughter apologizes and then offers to send money to the Sterlig's. Linsan's father tries to refuse it, but Dukan phrases it as helping Linsan and they accept.

It might not be much, but it was something I've been looking for. It also means that I can include things like POV, when it happened, or other details in the list to help work timelines, locations, and the like. Or use it as a simple index page generator for HTML and Gemini pages, if someone found a use for that.

Tables

Of course, there is also a table approach to the same thing:

markdowny table chapters/*.md --fields _basename title when.start locations.primary | head -n 5

_basename	title	when.start	locations.primary
chapter-01.md	Rutejìmo	1471/3/28 MTR 4::22	Shimusogo Valley
chapter-02.md	Confession	1471/3/28 MTR 4::75	Shimusogo Valley
chapter-03.md	Morning	1471/3/28 MTR 11::71	Shimusogo Valley

Nor Curse Be Found

Despite all that, I got another chapter of Nor Curse Be Found written this afternoon. It isn't polished as I hope, I'm still struggling with the “feel” for the chapter and how the prince responds to things, but that is also something I have to finish to really understand and go back to edit.

Overall, it was a good day (it took me three days to write the chapter).

Semantic Release and Woodpecker CI

2022-08-07T05:00:00Z

With the recent drama of GitLab, both with the CI/CD changes and then more recent possible threat of deleting old repositories, I continue my migration to a local Gitea instance, https://src.mfgames.com/ for the bulk of my code and writing.

For the most part, migrating is just a matter of shuffling data. I have a lot of repositories, both active and inactive, and it will take me months to move them over. Plus I haven't decided if I'm going to purge them from my GitLab account so there is a single source of truth or just mirror back to them.

Currently, the most difficult task was figuring out how to handle the build processing. I've mentioned previously that I use Conventional Commits and Semantic Release fairly heavily. I've branched out a little from there using Lefthook and my project layout.

Currently, the CI does the following:

Build the project
Test various conditions including valid commit messages
If the commits indicate a new build:
1. Tag it
2. Build the release version
3. Create a release on Gitea

This changes over time, but it is the basic pattern.

Tags and Git Depth

Woodpecker does not automatically download the needed tags for semantic-release (and GitVersion). This means that the .woodpecker.yml file needs to include tags.

clone:
    git:
        image: woodpeckerci/plugin-git
        settings:
            tags: true
pipeline:
    # The pipeline elements

Unike GitLab, which only limits to the last ten commits, it appears that Woodpecker downloads the full repository by default which is also needed by GitVersion because it calculates every version. Not entirely sure about semantic-release logging indicates it doesn't need the full repository, just enough back to find a version.

Building and Testing

To support task branches, I have a basic build and test code that runs on pushes and pull requests. This lets me identify bugs earlier and catch typos with my commits.

build:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - nix develop --command scripts/build.sh
    when:
        # We need both "tag" for the next section.
        event: [push, pull_request, tag]
        tag: v*

test:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - nix develop --command scripts/test.sh
    when:
        event: [push, pull_request]

From the build tasks, you can see that I'm using my current project layout which uses scripts in the scripts/ folder instead of npm run or dotnet run. This is to make it easier to work with polyglot plus works around the issue that I need to use nix develop to get into my reproducible environment since the Docker image doesn't automatically do that. This is because both Gitlab and Woodpecker use the image which bypasses initialization files and I couldn't have it run direnv allow automatically to set up environment variables.

One thing that is missing is that Woodpecker doesn't have a clean mechanism for temporary build artifacts. I can't upload the build files and then download them so I can see the final results. Instead, I have to script it out or use a S3 plugin.

Building on Versions

With most cases, I build the release version of the project when the conventional commits indicate that there is a new version (feat and fix). This is an additional pipeline that comes after the test: line.

release-main:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - export DRONE="true" # Required to convince `env-ci`
        # semantic-release needs this locally
        - git branch $DRONE_BRANCH origin/$DRONE_BRANCH
        - nix develop --command scripts/release.sh
    secrets:
        - gitea_token
        - git_credentials
    when:
        event: push
        branch: main

There are a number of things in this block that took me a while. The first is the event and branch. We only do releases on the main (I'm still moving away from master as racist language).

The second is the export DRONE line. At the time I set this up, env-ci wasn't aware of Woodpecker, but it was recently added thanks to 6543 on the Woodpecker Matrix channel, #woodpecker-ci:matrix.org. I don't know when the latest semantic-release will have it, but it shouldn't be needed soon, if not already.

The third is the git branch line in the above script. Woodpecker creates a detached head, as does Gitlab. But when it doesn't do is also create a local branch for the one being created. This causes a problem because the release process appears to “jump” to the branch to figure out the changes between the detached head (the commit being built) and the actual branch.

Finally, we have the secrets. semantic-release automatically picks up $GITEA_TOKEN for the release process but also needs $GIT_CREDENTIALS to verify Git access.

The token is easy, that is what given by Gitea for the user.

GIT_CREDENTIALS is slightly harder, it is a colon-separate tuple of the user name and the Gitea access token. From observations, the Gitea is basically just a bunch of URL-safe characters, so the URL-escaping isn't needed in my case (you need to URL-escape the left and right of the colon but not the colon itself).

$ export GITEA_TOKEN=9fc6d72c72e4b149f07491a0b2d3ec9215d57caf
$ export GIT_CREDENTIALS="dmoonfire:$GITEA_TOKEN"

These need to be set on a per-project basis since Woodpecker, unlike GitLab and sourcehut, there doesn't appear to be a good way of having a shared set of secrets for projects (GitLab has organization/group level secrets, sourcehut has the secret storage). This means I have to set the same GITEA_TOKEN and GIT_CREDENTIALS for all 80+ of my Fedran repositories†.

† Woodpecker has a CLI, woodpecker-cli which will let me automate that. I will use that.

Create Release on Tag

One thing I'm moving toward is creating a release entry on the forge. Since this happens after the build process and Woodpecker uses a different Docker image, I need it to be a post-release event so I hang it off the tagging process instead of the push to main.

6543 came to my rescue again with this one, so that is why there are those two “tag” elements in the script above. I also have a new stanza for the release process:

release-gitea:
    image: plugins/gitea-release
    settings:
        base_url: https://src.mfgames.com
        files:
            - "*.pdf"
            - "*.epub"
        api_key:
            from_secret: gitea_token
    when:
        event: tag
        tag: v*

If I didn't have the event: and tag: in the build: stanza, it wasn't working for the tags. This caused me some difficulties because I usually treat hashes as being unordered, but Woodpecker uses file order for processing pipelines. So, I needed to have the build: target build the file (with the correct version because it was tagged) and then release-gitea: to use that output for the release process. The test: and release-main: are skipped because they don't have those events listed.

In addition, secrets are handled differently when done as a parameter for a plugin. That is why I have the from_secret: element in the above script. This inconsistency threw me for a few days.

Putting it Together

If you want to see the final version, check out this example which has my current version as a single file.

Conclusion

I'm happy to move over to Woodpecker (you know, except for the cost of hosting) both because of the control and the challenge. I also don't have a need for speed, so if it takes a while to get through the queue, I'm okay.

Author Intrusion v0.9.0

2018-06-26T05:00:00Z

After a few weeks of work, the current rewrite of Author Intrusion got to a stopping point. This has the barest minimum functionality to detect echo words but it has a long way to go. Depressingly long way, but I need to let it settle a little before I jump back into it.

I also want a little encouragement so I'm going to toot my own horn and show some progress.

Starting Over Again

I think this is the ninth attempt I've had to write Author Intrusion. Each time, I've encountered various walls where my ideas didn't have the performance or couldn't conceptually move beyond the proof of concept. Over the last eight years, I've learned a lot about writing this and each time I hope “this is it”.

The current goal is to write a command-line interface (CLI) inspired by compilers (like GCC and TypeScript tsc) and Git. Basically, have AI have a core set of function but don't worry about re-implementing a text editor (which was, in all honesty, one of my more common mistakes for previous versions).

I spent a lot of time trying to get a solid, cross-platform GUI for this. That includes various Gtk# and Electron implementations before I decided to switch to CLIs.

For reference, this is some of my efforts five years ago:

The “mmm” was a search and replace on one of my larger commissions to test performance. In this case, it was a 100k commissioned novel.

There were a lot of attempts in there and I spent a lot of time trying to create an editor. While I was “fairly” successful, I think it made the project too big for one person to do. Each major iteration, I've been removing features trying to get to a core functionality while still honoring my core goals of helping me write.

Current Implementation

After the attempt to write it in Typescript, I decided this version is going to use .NET Core. The folks at Microsoft have done an amazing job of writing something that is fast and capable while running on both Windows and Linux (one of my requirements). It also is my core language, so I'm not struggling with learning and the tools like I was with Typescript. My primary environment is Visual Studio 2017 with ReSharper, Rider, or Visual Studio Code. While I use Atom, global configuration options pretty much means my Atom setup is specifically for writing, not coding.

NuGet

However, after the Typescript implementation and later code, I realize that the approach npm uses to manage packages is perfect for what I'm looking for. My struggles with various incarnations of MfGames Writing showed me that bitrot is a major problem while writing over years. The earlier incarnations of the build framework would evolve to handle new novels but then it would break older generation in the process. With npm, I can have a specific version in one project, then the library could continue to evolve while still providing the ability to stay at an older version for those older works.

NuGet has a number of C# libraries for writing a client that would give me the same thing. Various utilities, analyzers, and libraries can be packaged up as NuGet packages and then installed. If they evolve, the older version can remain behind and still work.

This version has the basics of this in, I can install packages and have various functionality available for processing.

This part is actually one of the neatest parts, I think. I'm using Autofac, a bit of reflection, and the NuGet libraries to install packages and then load assemblies from those packages without needing to pull them into a central location.

As soon as the plugins load, the system rebuilds the plugins and injects functions. This can be various plugins, new XSLT functions, or anything else.

Layout Plugins

The above screenshot is an example of the layout plugin. It tags files in the project as to their purpose. I made a mistake earlier on this when I started doing grammar checking on notes. In this case, a plugin can indicate something as “content” (e.g., the novel or story) or “lookup” (notes) or something else.

Right now, I'm only implementing my “standard” project layout that I've used for the last few projects. Eventually I'll write more but I'm trying to get end-to-end before fleshing out ideas as needed.

A layout plugin is responsible for gathering metadata about a file so decisions can be made. Eventually this will go into the YAML header for the file. This means I'll be able to identify files that have a specific point of view with something like:

---
title: Chapter 1
pov: Dylan
swain: scene
---

It was a bright and depressingly sunny day....

The idea for this is to be able to list all chapters of a given POV and then arrange them chronologically while listing the location. Or tag a file as being a scene or sequel so they are interspersed correctly.

Structure Plugins

A structure plugin basically figures out the structure of a file, such as figuring out if it is broken into paragraphs, sentences, and words. I didn't want to hard-code this because sentence splitting is hard and expensive plus my fantasy novels all have epigraphs. Those who like Scrivner may want to arrange it into scenes.

Structure plugins are boring but critical.

Xpath

An important part of the structure is not applying a structure to certain files. Lookup files don't need to know the individual words or paragraphs. To limit it, I'm using a scope variable which is an Xpath into the project.

<project>
  <file path="/chapters/chapter-01.md" class="content" />
  <file path="/characters/dylan.md" class="lookup" />
</project>

This means, using a path of /content[is-content()] will select only the content files but not the lookup. Originally, I implemented this as a CSS-like library which I eventually realized I was going down a rabbit hole (I still broke apart the library for later if I need it).

Again, C# has the ability to have defined XSLT functions so I wrote is-content() (injected via Autofac) that does custom logic.

As the various structure plugins operate (defined by the author-intrusion.aipy file), it will extend the XML structure used for selections.

<project>
  <file path="/chapters/chapter-01.md" class="content">
    <para start="0" length="10" />
    <para start="11" length="21" />
  </file>
  <file path="/characters/dylan.md" class="lookup" />
</project>

<project>
  <file path="/chapters/chapter-01.md" class="content">
    <para start="0" length="10">
      <token start="0" length="3" />
      <token start="5" length="4" />
      <token start="9" length="1" />
    </para>
    <para start="11" length="21">
      <token start="11" length="3" />
      <token start="16" length="4" />
      <token start="20" length="1" />
    </para>
  </file>
  <file path="/characters/dylan.md" class="lookup" />
</project>

This is because of another previous mistake (yeah, I made a lot). Various implementations tried to normalize contents while writing. This meant it would correct double spaces after periods or adjust the text.

That didn't work.

I also couldn't break it down into a simple tree structure because English didn't fit well. So, this XML just goes into the original file to get the text.

Analysis Plugins

The entire reason to break apart a document into a structure is for the analysis. An analysis plugin, such as echo detection, uses the XML structure to gather information.

All plugins are configured in the author-intrusion.aipy (project file in YAML format).

plugins:
  analysis:
  - plugin: EchoDetection
    key: echoes

    scope: content
    select: //token[length() > 3]

    compare: text()

    within: 20
    warning: 2
    error: 5

The plugin attribute is the class to use the plugin. The key is just a label for Atom's linter in case someone uses multiple echo detections in a file.

The second section is to figure out what is being detected. The scope uses content which is a shorthand for /file[is-content()] or /file[has-class("content")]. This breaks apart the search process. Using / for the path would do echo detection across chapters (and require more memory and would be slower) while the file-level ones makes it more efficient by only comparing a single file against itself.

For every scope, the select figures out what is going to be compared. In this example, for every file, we select every word over three characters long (//token[length() > 3]). If we had three chapters of a thousand words each, this is a different of a single list of three thousand items (scope: /) verses three lists of a thousand each (scope: //file[is-content()]), or three hundred paragraphs of ten words each (scope: //file[is-content()]/para).

The third section is the compare. This basically figures out what to compare. The text() means the raw text of the file. However, functions to handle case-insensitivity, stemming (base words so I jumped over the jumper would have two echoes), Soundex (to find similar-sounding words near each other), or whatever else I need.

Finally, the echo detection has the rules for what is a detection. It basically counts how many identically entries (as determined by compare) in each selected tag inside the scope within within entries. If that number is equal to or greater than error, then it marks that selected element as an error.

In the above example, the echo detection says “for every word in a file, look at the twenty surrounding words. If there are five or more, it's an error, otherwise if there is two more then it's a warning”.

Functionality

This isn't even remotely polished at this point. The project is self contained in that building it will generate the correct data, it just isn't… pretty.

dotnet run -v:q --no-build --no-restore --project src/AuthorIntrusion.Cli -- file-list -p "Examples/Sand and Blood"

Eventually, it should be something like:

aicli file lists

There is also a lot of missing functionality, using it probably requires me to understand, though I'd like to think it is pretty simple. Then again, I wrote it, of course it's simple.

Complexity

You may have noticed that the author-intrusion.aipy file is somewhat complicated. This is actually intentional. There are a lot of tools for writers. I'm looking for something very specifically to help with flaws I'm aware of in my fiction, not a generic “one size fits all” application. Because of that, I need it to work for me instead of trying to inflict

This desire to customize to the author is purely inspired by James White's Fast Trip. If you have a chance, consider reading it. It talks about modifying the environment the way you need to work, not the other way around.

However, flexibility comes at a price: simplicity. I considered trying to make it easy, but I'm looking for something that looks for overuse of adverbs (technically correct) or gerunds. I want to be able to make sure a character only speaks in past tense or doesn't use a pronoun to identify themselves. One of the earlier ones I'm going to get done is looking for present tense outside of a quote. These are pie in the sky items, but I think possible.

Later, if this gains traction (e.g., users besides me), someone may come up with a fancy GUI or configuration wizard to add common settings.

Development

Author Intrusion is currently being managed via its Gitlab project. I'm not sure if it would be worthwhile for anyone to consider joining, but if you want to watch it, this would be the place.

If you have questions, please don't hesitate to poke me on any social network I'm on. I always love to bounce ideas or talk about future place. The more I do, the more I can make it useful for everyone, not just myself.