Fast Trip

generator-mfgames-writing v0.3.2

2021-07-31T05:00:00Z

Related to process of writing from a few days ago, I've been working on another tool that I find myself using fairly often: Yeoman. This is a scaffolding tool, which basically means it asks a few questions and then sets up a project. In my case, it does a lot of the drudge work of creating a writing project and getting it ready for Gitlab's CI which I also use heavily (and why I wrote commitlint-gitlab-ci).

Like many of the writing tools, this is based on Javascript (but not Typescript this time).

$ sudo npm install -g yo generator-mfgames-writing

The first thing it does is ask a number of questions about the project. I'll use Nor Curse Be Found, my Beauty and the Beast sequel, as an example.

$ mkdir nor-curse-be-found
$ cd nor-curse-be-found
$ yo mfgames-writing
? The title for your novel or story: (Nor Curse Be Found)

All of my writing projects have a unique “slug” to describe them. This is used for the website's URL, such as https://fedran.com/nor-curse-be-found, and also the name of the output (dmoonfire-nor-curse-be-found-0.0.1.epub). I use a library to pull out the current directory's name and try to make a reasonable title out of it. I can change the title to something more accurate, maybe with apostrophes or possessives, but generally I keep the slug pretty even. In this case, the parenthetical item is the suggested output, so I can just hit return.

? The title for your novel or story: Nor Curse Be Found
? The slug based on the title: (nor-curse-be-found)

Just in case the directory doesn't match the directory, I use a library to take that given title and create a slug out of it. One of the biggest things, my Fast Trip rule, is that I don't want to tell someone how to work. That is why I called this semi-opinionated in that there are a lot of options but the default is how I work.

There is also a question about bylines, but then we start to get into the technical bits.

? The title for your novel or story: Nor Curse Be Found
? The slug based on the title: nor-curse-be-found
? Author's name or byline: D. Moonfire
? Do you want to create the Git repository? (Y/n)

This came from some years ago, but I do one story per repository, more so if I think the story has a possibility of being a bigger piece (which many of mine are), I want to track the versions independently, or I might be sending to an editor or submitting it somewhere (not likely these days, I self-censor a lot there). Since this is so common, I put it in as part of the questions.

I have a slowly evolving set of practices when it comes to writing processes. One of the first ones is setting up a Git repository. This always starts with configuring husky, semantic-release, conventional-commits, and commitlint. (I'm working on updating my Yeoman generator to make this a lot easier, but that will be later this week.)

? Do you want to create the Git repository? Yes
? Any command to run after `git init`? set-moonfire-git-user

I also write for a number of different projects (and bylines) which means I don't always use the same Gitlab login or even email. To faciliate this, I do not have a global user.name and user.email set up in Git so it blows up if I try to make a commit without setting a local one. Since that is tedious, I write a bunch of scripts that set the user information and SSH keys (set-moonfire-git-user for writing, set-mfgames-git-user for programming for example) and I want to call them before I do that first commit. If I just hit return to have a blank, then nothing extra will happen.

? Any command to run after `git init`? set-moonfire-git-user
? Do you want to create the initial commit? Yes
? Do you want to create the initial Git tag? Yes
? What version do you want to start? 0.0.1

Now, the initial commit is a messy one and a philosophical difference between me and the Semantic Release folks. They have their reasons to hard-code the value to “1.0.0” but I'm specifically start this process before a release, so I feel it should be “0.0.1” because I want that tracking as soon as possible.

Since I don't have the ability to set it via a configuration file, I get around it by creating the initial commit and then allowing it to be tagged so it can be pushed up and get the starting version I want.

This is also because I tag versions I give to editors, beta readers, and even my writing group (when I went to it). The tagged version also shows up on the various Fedran pages and mainly it shows progression as I write.

So, Fast Trip so I jump through a few hoops. It should also be clear why I needed that command to run after git init to set my user, otherwise this part would blow up with a “user not set” message.

? What version do you want to start? 0.0.1
? Which package manager do you use? (Use arrow keys)
❯ NPM
  Yarn

Moving away from Git and into the build process, the first question is package management. I have a love/hate relationships with npm. The command works great, but I've been struggling with it (more on than off lately) so I've been using yarn fairly heavily in the last year or so. But, with Yeoman I can give both so the user can use up or down keys to select and hit return to choose. Once they do, the menu goes away and is replaced with the choice.

? Which package manager do you use? Yarn
? Do you want to use semantic releases? Yes
? Do you want to use conventional commits? Yes
? Do you want to use Husky? Yes

The next three are to set up the common functionality I use with releases. They set up semantic-release, conventional commits, commitlint, and Husky which I use to make sure all the commit messages are clean. These four program have been a stable in most of my development lately, for coding and writing, for Typescript and even C#.

? Do you want to use Husky? Yes
? Do you want to use CI/CD? Yes
? Do you use Gitlab for your CI/CD? Yes

I can't say enough for Gitlab's CI. Whenever I push up code to the server, it runs the entire publication process. Assuming it works, it then produces a zip archive with EPUB, MOBI, PDF, and other formats without me doing anything else. Every single time. With sematic releases and the rest, each one has a unique version so I don't have “final”, “final-2”, “final-2a”, etc. Just a clean version number that shows up everywhere, starting at 0.0.1.

? Do you use Gitlab for your CI/CD? Yes
? Do you use asdf? No

This is another tool I use, asdf. This was also something that pretty much started last year, but one of the goals of mfgames-writing was to be able to produce consistent output of novels even years later. Well, what I didn't realize is how much the Node ecosystem would change and Node 15 doesn't exactly run Node 8 packages very well due to deprecations and changing libraries.

This is where asdf comes in. It puts a small file (.tool-versions) which says which version of Node, Yarn, C#, Python, whatever and will automatically change to those versions when I cd into the directory. It basically rolls in all the functionality of nvm, virtualenv, and a bunch of other localized version libraries into one. Since I write and work across many languages, having one tool to manage that is easier for me. So, saying “yes” to the question copies that .tool-versions into the resulting directory.

? Do you use asdf? Yes
? Which formats do you want to use? (Press <space> to select, <a> to toggle all, <i> to invert selection)
❯◉ PDF
 ◉ EPUB
 ◯ DOCX
 ◯ HTML

With this next question, we are out of the build section and into the formats. Basically, this just queries which types of files will be written out. The directions are pretty clear, but basically this will determine which of the format libraries are included.

? Which formats do you want to use? PDF, EPUB, DOCX, HTML
? The name of the theme package to use: (@mfgames-writing/clean-theme)

I only have three themes right now, two are “generic” and one I prefer just for Fedran stories (but obviously don't prevent anyone from using it). Later, I hope to have more themes, but basically the choices are @mfgames-writing/clean-theme and @mfgames-writing/greekil-theme (a Gentium-based theme).

? The name of the theme package to use: @mfgames-writing/clean-theme
Creating initial Git repository
Initialized empty Git repository in fedran/nor-curse-be-found/.git/
Running custom Git command


I'm all done. Running npm install for you to install the required dependencies. If this fails, try running the command yourself.


   create package.json
    force ../../../.yo-rc-global.json
    force .yo-rc.json
   create release.config.js
   create commitlint.config.js
   create .husky/commit-msg
   create .gitlab-ci.yml
   create .tool-versions
   create publication.yaml
   create yarn.lock
   create README.md
   create .gitignore
   create chapters/chapter-01.md

Changes to package.json were detected.

Running yarn install for you to install the required dependencies.

With that, it just producing a wall of tell as it creates all the files, hooks up everything, and basically gets my entire project ready, including a sample first chapter, a really basic README.md, and a Git repository ready to push.

$ ls -a
.                     .git            node_modules       README.md
..                    .gitignore      package.json       release.config.js
chapters              .gitlab-ci.yml  package-lock.json  .yo-rc.json
commitlint.config.js  .husky          publication.yaml

Right off the bat, I can build the files and see the output.

$ yarn build:pdf
... lots of output
$ ls *.pdf
nor-curse-be-found-0.0.1.pdf
$ yarn build
... huge amounts of output
$ ls nor-curse-be-found*
nor-curse-be-found-0.0.1.docx  nor-curse-be-found-0.0.1.epub
nor-curse-be-found-0.0.1.html  nor-curse-be-found-0.0.1.pdf

I'm also ready to push up to Gitlab.

$ git remote add origin git@gitlab.com:fedran/nor-curse-be-found.git
$ git push --tags
$ git push -u origin main

Hopping over to Gitlab, I can see it building immediately.

As soon as it is done, it uploads the output as an artifact.

I can then click on the browse (or download) button on the right.

Browsing lets me see a list of all the outputs that I've added.

And I can even click on the PDF to preview it.

The End After Three Hundred Weeks

2021-03-17T05:00:00Z

Just under two years ago, I posted about how I had been posting weekly chapters for two hundred consecutive weeks. Since then, I've managed to do it for another hundred weeks which puts me up to six years of posting (mostly) consistently. There were two instances where I missed a week but I caught up and returned quickly. Overall, I can say three hundred weeks of posting with a straight face and still feel like I wasn't lying.

I had a lot of hopes in 2019. Things were looking positive and I was writing strong. I found a new community and they were throwing thumbs up at me. Then 2020 hit and things started to get a little less rosy. I don't need to say what happened, but my household has been in relative isolation since March and then we got hit by the derecho in August. They still haven't finished fixing the roof or repairing the hole in the side of my house before the snow came.

(Unlike my normal rule of limiting negative posts, this one talks about depression.)

Skill and Perceptions

Somewhere in September, I started to get a feeling that everything I was writing was “wrong”. I can't tell how it was wrong, only that I wasn't capable of fixing it because I couldn't tell what I was missing. It didn't take long to realized was in a slump probably related to the skill/perception curve (source).

I'm just left of frowning face, a point where I know something is wrong but my skills and abilities aren't enough to identify the problem. I've had this feeling before, where I know there is something wrong, but I've never had a situation where I had no clue how to improve my writing.

This discomfort with my writing continued to grow, sapping my creativity.

Flight of the Scions

In October 2020, I also had Flight of the Scions edited. When I didn't get the enthusiasm I was hoping for the piece, that pretty much sent me down a spiral and the growing sense that I was missing something turned into a full-blown loathing. I haven't integrated the edits for that novel because of it… because I feel like I can never do the piece justice.

Plague

In November, I got sick.

End of the Year

In mid-December, I still wasn't able to identify what was wrong with my writing. I was writing. I was still passionate about Allegro but every word that got down on the page felt alien, twisted and foul.

That was also the point I realized that I can't just push myself past it. There weren't enough words because I just spent months not liking my words but unable to find the epiphany that would help me through the roadblock.

NaNoWriMo

Something has to change if I want to keep getting better at my writing. I've written a lot, I've grown as a writer and as a person. More importantly, I want to graduate from an “okay” writer to a “good” writer. This requires me to figure out what I'm missing.

While I thought about it, I looked at what writing weekly was giving me. At the beginning, much like NaNoWriMo, it was about “can I do it”. Six years of writing says I can. I also will admit, my writing has significantly improved in the six years of steady writing and I'm a much stronger creator than ever.

But I don't think I've improved much in the last year. I think I have reached the top of this mesa with my skill by just forcing out more words. I need something more deliberate, which also means less grinding out words and more of trying things to shuffle things up and expand in a new direction.

I said the same thing about NaNoWriMo. I was able to do it, but the month of writing wasn't really the challenge. For me, it was more about getting the word count, but I already know I can write fifty thousand words in four days. I've done it, more than once. I even published the results of one of those experiences.

NaNoWriMo came down to word count, which is a metric that I love to use but it also a useless one for me. It slides under the Goodhart's Law:

When a measure becomes a target, it ceases to be a good measure.

Word counts, including minimum words a day or reaching a certain number, isn't helping me because I just work toward the target. With NaNoWriMo, I find that I can't easily write past 50k words even if the story needs it. A word count in a day becomes a chore.

Same with writing time.

Using likes and reviews is also difficult because I'm an okay writer, not a good one. The volume of what I write means folks stop reading it, or they realize I'm not a good writer. I'm not sure without feedback, so I'm going to say they don't work because I can't measure anything.

Burnout

To be honest, much of this feels like burnout also. I've gone through two major burnouts in my life and I think the weekly chapters are beginning to strip those screws bare again. That just gives more credence to the idea of stopping my weekly chapters.

Burnout also leads to depression and a feeling of worthlessness that never goes away. That's been hanging around me since I was seventeen but it got really bad since August. Every time I stumbled into the “never write out of your lane,” I realized I had created a world that would require me to venture out and I shouldn't have. Every time I got a rejection, I knew I wasn't that good of a writer. Every time I tried to convince someone to read my work and they just… gave up, it just reinforced that.

So the logical response, to avoid upsetting everyone, was to consider giving up. Not because of a “if I can't write that, then I should stop writing” but more of a realization that I'm not really that important and there are far better people out there that need the space more than me.

I've been struggling with this for months. I think one of the reasons I have have only made a single submission in the last four years is because of that. (I was rejected, of course.) There are “own voices” that need to be said more than what I do. I can't do much beyond boosting others, but not trying to draw attention to myself is one of those. This is something I've been considering for months.

Because of that, I'm having trouble wanting to draw attention to myself. My weekly chapters were a single post around six in the morning once a week and I consider that excessive. When I finished my last book, it was a second posting in a week and nothing more. I don't know if I can, or want, to go beyond that. I haven't asked for a table at ICON for the same reason. I haven't paid for advertisement either.

Mr. Tanner

I'm not going to stop writing. I don't think I can stop writing. It brings me so much joy and I love creating stories. I just don't know who I'm writing for anymore. In many ways, the final lyrics of Harry Chaplin's “Mr. Tanner” have been speaking to me.

Music was his life, it was not his livelihood
And it made him feel so happy and it made him feel so good
And he sang from his heart and he sang from his soul
He did not know how well he sang; It just made him whole

Much like James White's Fast Trip, “Mr. Tanner” has become a sign of the last year or so.

What's Next

I don't know.

Right now, I'm focusing on finishing projects and not writing anything new. That includes a side piece, Flight of the Scions (hopefully for August), and Second-Hand Dresses for November.

If I do write something new, I want to finish Allegro because I'm almost near the end. I also had a vague idea of sending it to the inspiration of the novel, but then I made Linsan a demisexual with lesbian tendencies and I probably overstepped my bounds. I will release the chapters for public reading as I go because I started that way and I feel that is a social contract I'm not willing to

I honestly don't know about new stories or novels. I like the occasional up-vote on various channels. To be honest, very few folks are actually reading it. When they do, it's in a surge and then they wander off. I have two fans on Patron but only one speaks up; I can't thank either of them enough for the encouragement.

If this is burnout, then I need time to recover. I need adventures and to work on other things.

There is always a fear that I just started the process of giving up on writing, but I can't honestly tell at this moment. It's possible, but that is life. It changes, it evolves. I've been writing fiction, in some form, for at least forty years. I can't imagine just stopping cold, but then again, I wasn't planning on getting married, having children, and having a hole punched in the side of my house.

Author Intrusion - I've Been Here Before

2019-04-30T05:00:00Z

When it comes to projects that I probably will never finish, Author Intrusion is probably right up there. Most of the time, it is because I keep restarting because I learned significant lessons. This last week was one of those cases.

Author Intrusion is also one of those projects that I keep coming back because I know it can help me but it requires a lot of effort to turn what I think I need into something that is useful. One might say that is obvious from the sheer number of times I restarted development but it is a moving target even from the beginning.

There are a lot of reasons for the various restarts:

Can't handle my 25k word penultimate chapter on a project:
- ... in a reasonable time.
- ... or with reasonable memory concerns.
- ... or without grinding my entire system to a halt.
Can't handle my 600k word epic:
- ... in a reasonable time, overloading my memory, and stopping my machine.
Couldn't figure out how to determine parts of speech correctly.
I tried to:
- ... write an entire text editor.
- ... okay, just the text control.
- ... okay, just load single chapters.
- ... okay, just a CLI to parse the files.

This is over eight years of struggling with this tool.

Expectations

Part of the problem is I'm thinking too grandly. I always had that problem when I can't code, silence is probably the worse things for my plots and my projects. I think I've managed to winnow down the list to a more obtainable set of goals:

Work with YAML + Markdown, my preferred format.
Analyze the chapters of the book at:
- Identify problem words.
- Identify clustered echo words.
Get me a good count of words for the chapters:
- ... for the entire book.
- ... for a range of chapters.
Identify sentences:
- ... that start with the same pattern.
- ... have the same number of words.
Identify concurrent paragraphs that start with the same word.
Identify overuses of gerunds (a category of -ing words).
Identify overuses/clusters of “to be” verbs.
Allow me to insert a chapter easily.
Identify present tense outside of dialog.
Integrate with Atom as a language service.
Work on Windows and Linux.
Handle custom spelling.

Okay… that isn't that much of a shorter list. Even so, it will take a while to even get close to it despite these features being 90% of what I use tools to do today.

The Last Failure

The last version used XPath and XSLT. I figured it was a good way of framing the ideas, however I started getting bogged down in making it flexible because I hate the idea of opinionated systems and didn't want to force others to work the way I work (blame James White's Fast Trip for that).

Also, it struggled with the large files and projects.

When I encounter a problem, sometimes it is good to think of it in a different way. This might require retooling it, starting over (I did that with RPG games a lot as a kid too), or trying to come around the problem a different way.

Rust Language

I decided to spend a week trying another iteration of Author Intrusion. This time, I used it as a learning experience for an entirely different (and new to me) language: Rust. Rust is a pretty low-level language but has much of the packaging and niceties of C# (my primary language). Since I knew the initial versions of Author Intrusion so well (I have restarted over ten times), it was old territory that would make it easier to map the language into something I already knew.

Overall, Rust is fairly pleasant to use but there are places where I struggled a lot with it conceptually. I've worked in object-oriented languages for the last twenty years, Rust has some OO-like support but I had to change how I was working to use it. Also I still haven't groked the “borrow” concepts enough, so sometimes I just revert to randomly adding and removing & and * in hopes of figuring it out.

In other words, I beat on it like a black obelisk.

The tooling with Rust is considerably poorer than C#. Not being able to refactor easily definitely ate up significant time in the last week. Also my laptop hates working with Rust, it gets burning hot after only an hour of coding… which was good because I put it down until everything cooled off.

That said, I've been pretty happy with the results so far so I'm hoping to keep it going with this variant instead of the previous ones, at least to see if I can get closer to my end goals.

Running the Tool

So, if you are interested in checking it out with this post, head over the the GitLab repository and clone it. I'm using rustup for my installation with the nightly builds, so the following should work:

$ git clone https://gitlab.com/author-intrusion/author-intrusion-rust
$ cd author-intrusion-rust
$ ./run-000 chapter list

On Linux, I create an alias to run it from anywhere:

$ alias author-intrusion="cargo run --quiet --"

Unless noted, I'll also run all of these examples from the ./examples/000-simple-project directory.

Project Files

Like before, everything hangs off a author-intrusion.aipry project file. This tells the system where the root of the project is located (I love being able to use npm and gulp from sub-directories) and how the project is organized.

The project file is a YAML file. There are two examples in the ./examples/ folder in the source code.

name: First Example Project
content:
  pattern: chapters/chapter-??.md

The name is just there because I like to have something.

The content chapter says how the project is organized for the actual content. Now, in my case, I almost universally use chapter for my files but I've been switching from chapter-01.markdown to chapter-01.md. Someone might prefer a different approach (like the src directory or eventually a chapter/scene approach).

File List

With that, we have the basic ability to list chapters.

$ author-intrusion chapter list
chapters/chapter-01.md 1 Chapter One  9
chapters/chapter-02.md 2 Chapter Two 11
$

This shows off some of the cool things it can do already. The first column is the relative path in project. This is the same as adding -f file.rel_path to the command.

The second column is the chapter number which is parsed from the file (chapter-01.md -> 01 -> 1). This is the same as -f file.num.

The third is pulled from the file itself in the front matter (the section between the --- lines). This uses JMESPath for the queries that lets you do some really tool things. In this case, it is just a simple -f title which pulls the title: line from the files.

Finally, the last column is the number of words inside the file or -f count.words.

The reason I have these individual fields is because maybe I only want the word count in each chapter.

$ author-intrusion chapter list -f count.words
 9
11
$

Using JMESPath

The JMESPath also lets me pull out more detailed information from the query. As I've shown in sample chapters, I put a lot of information in the front matter to help me keep track of the book.

A simple case is chapter 1 of the example file:

---
title: Chapter One
locations:
  - Location A
  - Location B
---

This is chapter 1 for Anton.

So, if I want to pull out the locations with the tool as a comma separated list, I can do the following.

$ author-intrusion chapter list -f file.rel_path -f "locations[] | sort(@) | join(', ', @)"
chapters/chapter-01.md Location A, Location B
chapters/chapter-02.md
$

It looks more impressive with the second example, which is the first nineteen chapters of Sand and Blood. I can use that to list all the characters who are present in each chapter besides the main character:

$ author-intrusion chapter list -f file.rel_path -f "characters.secondary[] | sort(@) | join(', ', @)" | head -n 3
chapters/chapter-01.markdown Hyonèku
chapters/chapter-02.markdown Gemènyo, Somiryòki, Tejíko
chapters/chapter-03.markdown Desòchu, Gemènyo, Hyonèku, Mapábyo, Opōgyo, Panédo
$

This is also Unicode-friendly, which is important to me since I'm fond of constructed languages and have a number of them for my world.

Defining Field Aliases

Now, that can get really wordy, so I added the ability to save those fields in the project file:

name: Sand and Blood by D. Moonfire
content:
  pattern: chapters/chapter-??.markdown
query:
  fields:
    characters.secondary:
      jmes: characters.secondary[] | sort(@) | join(', ', @)

That way, they can be easily used as an aliased field:

$ author-intrusion chapter list -f file.rel_path -f characters.secondary
chapters/chapter-01.markdown Hyonèku
chapters/chapter-02.markdown Gemènyo, Somiryòki, Tejíko
chapters/chapter-03.markdown Desòchu, Gemènyo, Hyonèku, Mapábyo, Opōgyo, Panédo
$

Analysis

The other part I managed to get done this week is the basic version of checking. Using Sand and Blood as an example, I have a problem with an overuse of “sigh” and “sighed”. I can use the tool to highlight when I start to use it too much.

To do that, I first define a check in the project file:

name: Sand and Blood by D. Moonfire
content:
  pattern: chapters/chapter-??.markdown
checks:
  - overused_pattern:
      # The `(?i)` makes this case insensitive.
      pattern: "^(?i)(sigh|sighed)$"
      err: 0.002
      warn: 0.001

The err and warn are ratios in the file. So, 0.002 means if more than 0.2% of the chapter is sighing, then just flag them as errors.

$ author-intrusion check
chapters/chapter-01.markdown:100:17: warning: found overused word "sighed"
chapters/chapter-01.markdown:140:11: warning: found overused word "sighed"
chapters/chapter-02.markdown:92:230: warning: found overused word "sighed"
chapters/chapter-02.markdown:120:48: warning: found overused word "sigh"
chapters/chapter-02.markdown:214:8: warning: found overused word "sigh"
$

The message is formatted so most tools like Atom or Emacs will move you the line and first character of the word being reported. And they can highlight the lines as yellow or red depending on the warning or error.

Additional Problems

The “sighing” problem was also mirrored by a “nodding” one. So, once I identify that word as a problem, I can add it to the file:

checks:
  - overused_pattern:
      # The `(?i)` makes this case insensitive.
      pattern: "^(?i)(sigh|sighed)$"
      err: 0.002
      warn: 0.001
  - overused_pattern:
      pattern: "^(?i)(nod|nodded)$"
      err: 0.002
      warn: 0.001

Then it highlights each one as either a warning (under 0.2%) or error (0.2% or higher).

Side note, I'm probably going to rename it to overused_word.

Speed

The biggest test is how fast this thing is. I don't have a lot done, but I had it process almost three million words in all the chapters I've written in the last twenty years in about twelve seconds. Doing actual English checking (overused words, patterns) will take longer, but twelve seconds is a promising start.

Obviously, it will get slower once I put the various checks in, but this is better than the previous iteration at an hour to handle two million words.

Conclusion

So that is how much I got going in a week with a language I just started learning a week ago Saturday. I'm pretty happy with the results and in the process, I found a lot of places that I was over-engineering the code. While I don't know if I'll ever be “done” with this, I think this is currently my best viable approach and it already has something I can use for this coming week when I work on Raging Alone.

Author Intrusion v0.9.0

2018-06-26T05:00:00Z

After a few weeks of work, the current rewrite of Author Intrusion got to a stopping point. This has the barest minimum functionality to detect echo words but it has a long way to go. Depressingly long way, but I need to let it settle a little before I jump back into it.

I also want a little encouragement so I'm going to toot my own horn and show some progress.

Starting Over Again

I think this is the ninth attempt I've had to write Author Intrusion. Each time, I've encountered various walls where my ideas didn't have the performance or couldn't conceptually move beyond the proof of concept. Over the last eight years, I've learned a lot about writing this and each time I hope “this is it”.

The current goal is to write a command-line interface (CLI) inspired by compilers (like GCC and TypeScript tsc) and Git. Basically, have AI have a core set of function but don't worry about re-implementing a text editor (which was, in all honesty, one of my more common mistakes for previous versions).

I spent a lot of time trying to get a solid, cross-platform GUI for this. That includes various Gtk# and Electron implementations before I decided to switch to CLIs.

For reference, this is some of my efforts five years ago:

The “mmm” was a search and replace on one of my larger commissions to test performance. In this case, it was a 100k commissioned novel.

There were a lot of attempts in there and I spent a lot of time trying to create an editor. While I was “fairly” successful, I think it made the project too big for one person to do. Each major iteration, I've been removing features trying to get to a core functionality while still honoring my core goals of helping me write.

Current Implementation

After the attempt to write it in Typescript, I decided this version is going to use .NET Core. The folks at Microsoft have done an amazing job of writing something that is fast and capable while running on both Windows and Linux (one of my requirements). It also is my core language, so I'm not struggling with learning and the tools like I was with Typescript. My primary environment is Visual Studio 2017 with ReSharper, Rider, or Visual Studio Code. While I use Atom, global configuration options pretty much means my Atom setup is specifically for writing, not coding.

NuGet

However, after the Typescript implementation and later code, I realize that the approach npm uses to manage packages is perfect for what I'm looking for. My struggles with various incarnations of MfGames Writing showed me that bitrot is a major problem while writing over years. The earlier incarnations of the build framework would evolve to handle new novels but then it would break older generation in the process. With npm, I can have a specific version in one project, then the library could continue to evolve while still providing the ability to stay at an older version for those older works.

NuGet has a number of C# libraries for writing a client that would give me the same thing. Various utilities, analyzers, and libraries can be packaged up as NuGet packages and then installed. If they evolve, the older version can remain behind and still work.

This version has the basics of this in, I can install packages and have various functionality available for processing.

This part is actually one of the neatest parts, I think. I'm using Autofac, a bit of reflection, and the NuGet libraries to install packages and then load assemblies from those packages without needing to pull them into a central location.

As soon as the plugins load, the system rebuilds the plugins and injects functions. This can be various plugins, new XSLT functions, or anything else.

Layout Plugins

The above screenshot is an example of the layout plugin. It tags files in the project as to their purpose. I made a mistake earlier on this when I started doing grammar checking on notes. In this case, a plugin can indicate something as “content” (e.g., the novel or story) or “lookup” (notes) or something else.

Right now, I'm only implementing my “standard” project layout that I've used for the last few projects. Eventually I'll write more but I'm trying to get end-to-end before fleshing out ideas as needed.

A layout plugin is responsible for gathering metadata about a file so decisions can be made. Eventually this will go into the YAML header for the file. This means I'll be able to identify files that have a specific point of view with something like:

---
title: Chapter 1
pov: Dylan
swain: scene
---

It was a bright and depressingly sunny day....

The idea for this is to be able to list all chapters of a given POV and then arrange them chronologically while listing the location. Or tag a file as being a scene or sequel so they are interspersed correctly.

Structure Plugins

A structure plugin basically figures out the structure of a file, such as figuring out if it is broken into paragraphs, sentences, and words. I didn't want to hard-code this because sentence splitting is hard and expensive plus my fantasy novels all have epigraphs. Those who like Scrivner may want to arrange it into scenes.

Structure plugins are boring but critical.

Xpath

An important part of the structure is not applying a structure to certain files. Lookup files don't need to know the individual words or paragraphs. To limit it, I'm using a scope variable which is an Xpath into the project.

<project>
  <file path="/chapters/chapter-01.md" class="content" />
  <file path="/characters/dylan.md" class="lookup" />
</project>

This means, using a path of /content[is-content()] will select only the content files but not the lookup. Originally, I implemented this as a CSS-like library which I eventually realized I was going down a rabbit hole (I still broke apart the library for later if I need it).

Again, C# has the ability to have defined XSLT functions so I wrote is-content() (injected via Autofac) that does custom logic.

As the various structure plugins operate (defined by the author-intrusion.aipy file), it will extend the XML structure used for selections.

<project>
  <file path="/chapters/chapter-01.md" class="content">
    <para start="0" length="10" />
    <para start="11" length="21" />
  </file>
  <file path="/characters/dylan.md" class="lookup" />
</project>

<project>
  <file path="/chapters/chapter-01.md" class="content">
    <para start="0" length="10">
      <token start="0" length="3" />
      <token start="5" length="4" />
      <token start="9" length="1" />
    </para>
    <para start="11" length="21">
      <token start="11" length="3" />
      <token start="16" length="4" />
      <token start="20" length="1" />
    </para>
  </file>
  <file path="/characters/dylan.md" class="lookup" />
</project>

This is because of another previous mistake (yeah, I made a lot). Various implementations tried to normalize contents while writing. This meant it would correct double spaces after periods or adjust the text.

That didn't work.

I also couldn't break it down into a simple tree structure because English didn't fit well. So, this XML just goes into the original file to get the text.

Analysis Plugins

The entire reason to break apart a document into a structure is for the analysis. An analysis plugin, such as echo detection, uses the XML structure to gather information.

All plugins are configured in the author-intrusion.aipy (project file in YAML format).

plugins:
  analysis:
  - plugin: EchoDetection
    key: echoes

    scope: content
    select: //token[length() > 3]

    compare: text()

    within: 20
    warning: 2
    error: 5

The plugin attribute is the class to use the plugin. The key is just a label for Atom's linter in case someone uses multiple echo detections in a file.

The second section is to figure out what is being detected. The scope uses content which is a shorthand for /file[is-content()] or /file[has-class("content")]. This breaks apart the search process. Using / for the path would do echo detection across chapters (and require more memory and would be slower) while the file-level ones makes it more efficient by only comparing a single file against itself.

For every scope, the select figures out what is going to be compared. In this example, for every file, we select every word over three characters long (//token[length() > 3]). If we had three chapters of a thousand words each, this is a different of a single list of three thousand items (scope: /) verses three lists of a thousand each (scope: //file[is-content()]), or three hundred paragraphs of ten words each (scope: //file[is-content()]/para).

The third section is the compare. This basically figures out what to compare. The text() means the raw text of the file. However, functions to handle case-insensitivity, stemming (base words so I jumped over the jumper would have two echoes), Soundex (to find similar-sounding words near each other), or whatever else I need.

Finally, the echo detection has the rules for what is a detection. It basically counts how many identically entries (as determined by compare) in each selected tag inside the scope within within entries. If that number is equal to or greater than error, then it marks that selected element as an error.

In the above example, the echo detection says “for every word in a file, look at the twenty surrounding words. If there are five or more, it's an error, otherwise if there is two more then it's a warning”.

Functionality

This isn't even remotely polished at this point. The project is self contained in that building it will generate the correct data, it just isn't… pretty.

dotnet run -v:q --no-build --no-restore --project src/AuthorIntrusion.Cli -- file-list -p "Examples/Sand and Blood"

Eventually, it should be something like:

aicli file lists

There is also a lot of missing functionality, using it probably requires me to understand, though I'd like to think it is pretty simple. Then again, I wrote it, of course it's simple.

Complexity

You may have noticed that the author-intrusion.aipy file is somewhat complicated. This is actually intentional. There are a lot of tools for writers. I'm looking for something very specifically to help with flaws I'm aware of in my fiction, not a generic “one size fits all” application. Because of that, I need it to work for me instead of trying to inflict

This desire to customize to the author is purely inspired by James White's Fast Trip. If you have a chance, consider reading it. It talks about modifying the environment the way you need to work, not the other way around.

However, flexibility comes at a price: simplicity. I considered trying to make it easy, but I'm looking for something that looks for overuse of adverbs (technically correct) or gerunds. I want to be able to make sure a character only speaks in past tense or doesn't use a pronoun to identify themselves. One of the earlier ones I'm going to get done is looking for present tense outside of a quote. These are pie in the sky items, but I think possible.

Later, if this gains traction (e.g., users besides me), someone may come up with a fancy GUI or configuration wizard to add common settings.

Development

Author Intrusion is currently being managed via its Gitlab project. I'm not sure if it would be worthwhile for anyone to consider joining, but if you want to watch it, this would be the place.

If you have questions, please don't hesitate to poke me on any social network I'm on. I always love to bounce ideas or talk about future place. The more I do, the more I can make it useful for everyone, not just myself.

Giveaways, Patrons, Sorrow Rides a Fast Horse, and Sand and Bone 32

2017-05-23T05:00:00Z

A busy week as I reach the end of Sand and Bone being serialzied on my website. There are giveaway, decisions, and a call for patrons. Not to mention a short chapter of Rutejìmo night time run.

Goodreads and Second Run Reviews

I've started up a Goodreads giveaway for Sand and Bone. There is also one on Second Run Reviews for my spotlight there last week.

Patrons and Future Decisions

I'm at a crux in a few short weeks. I have two novels that are “almost” done and one of them is going to be the one that I'm going to be serializing after this one. This is one of the few benefits of being a patron, the ability to vote on my next project so here is the choice:

Flight of the Scions: A YA start for a four-book series about Kaneko, a young girl without any magic who is trying to prove herself to her earth knight father who dismisses her after he learns that she will never follow in his footsteps. During this time, a mercenary group attempts to kidnap her and only her ingenuity and creativity, along with her new friends, can save her from becoming a pawn. This has an adorable dog girl with elemental magic and a telepathic boy with raw information in his mind. It also has an eighty foot telepathic toad and the mercenaries' air ship. The end of this series dovetails into Sand and Bone. The above link has the first two chapters as sample.

Second-Hand Dresses: A Regency-inspired high fantasy romance about a young woman who lost her chance at marriage after her debutante ball was interrupted by a duel going wrong. Nine years later, she is on the cusp of becoming a spinster when both men enter her life and throw it into chaos once again. This is a polyamorous and bisexual piece that is naughtier than my usual fare. This will be a stand-alone novel set in the same universe. The link has three sample chapters.

The poll is both at the Patreon site and my forum. As a reminder, patrons (even the $1/month) can read all my novels including the WIP of the two above.

Sorrow Rides A Fast Horse

I've talked about Fast Trip by James White as one of the major stories that has influenced my life. This chapter is inspired by another one, Sorrow Rides a Fast Horse by Dorothy Gilman Butters. I remember reading this story in the Writing Workshop at Prospect High School back when I was a junior. For some reason, the events in the story really resonated by me.

It is a shorter story, one that doesn't have action, but the looming despair of “Sorrow” chasing after the mother was a beautiful imagery. If I get around to it, you'll see that Sorrow is a horseman in one of potential future stories. It is the same images I saw all those years ago.

How does this relate to the story? Well, death is riding after our protagonist and he knows it.

Sand and Bone 32: Limits

With poison flowing through his veins and death chasing after him, Rutejìmo is desperate to make it home before darkness when he loses all power. As the slowest of the runners, he knows he won't make it but he can't stop, not even for the night.

Read the chapter at https://fedran.com/sand-and-bone/chapter-32/. If you like it, please become a patron or review one of my previous books. Subscribers get access to all my novels, including the first book of my next series and my high-society romance novel.

Author Intrusion status for May 1, 2017

2017-05-01T05:00:00Z

After a rough start of the previous week, I got a chance to really focus on Author Intrusion (AI). Despite the rather significant amount of check-ins and coding, I wasn't quite able to get to a really good “show off” point. Instead, this week ended up being a black triangle (significant progress but nothing visible).

Introduction

I haven't really talked about AI for a while. The entire idea started in 2009 or so when I realized I had some major blindspots in my writing. At the time, it was an overuse of gerunds that was hanging over me but as my skill improves, the problems areas also shift around. My writing group has been a great help in tracking these down but I felt a lot of them were things that could be detected ahead of time; basically something that would let me file off the rough edges before someone spent the effort to correct them.

To my surprise, the current offering of grammar checkers doesn't actually look for the same thing. They seem to review things sentence-by-sentence but I wanted something that looked at the entire document and looked for trouble areas. For example, using the same word repeatedly in a few paragraphs (echo words).

At the same time, I'm a programmer. I use a wonderful program called ReSharper which has a lot of refactoring and analysis. There are times when I wanted to see every time a character shows up (name outside of dialog) or is talked about. I also rename characters (frequently) and suffer from the occasional search-and-replace bugs.

That all lead to me want to create an IDE for writers, basically a Visual Studio and ReSharper that was geared toward authors.

One of my major influences is a short story by James White called Fast Trip. I love that story because it comes down to “change the environment to succeed.” Well, that meant a couple of things for AI, it had to be flexible to work the way the author wants and it needed to use the paradigm that I was comfortable with.

Iterations

I've worked on off-and-on for about eight years. I've tried a lot of iterations, gotten to a point, and then hit some conceptal problem that threw everything into disarray. Some of them were understandings on my part (you can't reformat text while writing), getting caught working on the wrong thing (I should reinvent the word processor), and a ton of other dead ends.

This means that it isn't done until it is stable. So if you are not interested in alpha software, this might not be what you are looking for at this point.

The current design goals is to create a separate, independent program that can be called by text editors. Inspired by OmniSharp, I figured it was better to create a program that did one thing well (analysis and refactor novels) and then create hooks that can talk to other editors like Emacs and Atom.

I'm also using Grunt and NPM from my own experiences. Some of the earlier implementations of AI were self-contained with all the plugins installed there. However, as I've been working on my Javascript publishing framework, versioning is very important. I can't break the analysis of novels I wrote ten years ago just because the code has improved to handle the novels of today. NPM has a great way of handling that with the package.json file that lists specific packages and versions, which can be installed and used. So, AI is done the same way, the system uses specific versions of packages to ensure that it always produces the same data. If you upgrade the underlying packages, then you can deal with the changes and check in the results knowing it won't change tomorrow.

I've written AI in a number of languages (Javascript, Typescript, C#, Pythong) along with different UIs. I've gotten lost in a lot of them. At the moment, my skills in C# are considerably more advanced than the others, but more importantly, the tools (ReSharper) make me a lot more effective. I don't want this to be a learning experience at this point because I have a lot of novels I need to write and I have to focus on either making fun tools (AI) or finish books.

I don't want this to be an argument of language. I like C# and I'm good at it, so I'm using it.

Overall Status

It isn't there yet. I'm working for an end-to-end which means setting up the files and being able to run a program (aicli) and have it produce an output that shows echo words (the first analysis I want to write and my current blindspot). What does work requires a relatively specific setup (not that bad, everything is contained inside the repository) and a bit of hand-holding.

This is still in the “black triangle” point since the technical framework is set in place, I'm still combining them to produce something that looks cool.

All of the examples are based on a subset of Sand and Blood used for testing. It can be found here.

Configuration

Like Grunt, there is a single file that controls how AI works with a project. I took inspiration from Visual Studio Code and Atom in that every file isn't explictly added to the project. Instead, AI assumes that all files underneath the directory containing the author-intrusion.yaml file is part of the project. If you add a chapter, the YAML file won't change but the system will pick it up.

I used YAML because it is easy to use, doesn't require a lot of noise, and pretty much handles the relatively simplistic data. Also, unlike JSON, YAML can let you copy a section from one place to another which I consider to be pretty useful.

I am trying to avoid changes a lot of files whenever you add a chapter, that's just noise in most cases. Related to that, AI will create a .ai directory for its internal caches, that shouldn't be checked into source control at all.

Eventually, aicli will search for author-intrusion.yaml. That way, like Grunt, it can be called anywhere inside the directory tree and it will “do the right thing” to produce consistent results. This is the same thing git does among other CLI implementations.

The basic author-intrusion.yaml file is pretty simple:

file:
    - plugin: AddClassFromPath
      match: chapters/*.markdown
      class: chapter
data:
	- plugin: AddClassFromData
	  select: file.chapter
	  class: pov-{pointOfView}
layout:
    - plugin: SplitLines
    - plugin: SplitParagraphs
    - plugin: OpenNlpSplitTokens
      select: para
    - plugin: WordTokenClassifier
analysis:
    - plugin: Echoes
      select: file.chapter token.word
      scope: token.word:within(10)
      threshold: { error: 5, warning: 2 }

In the example file from the link above, there is a lot more in the file either for documentation purposes or just to work out some concept or idea.

There are four sections: file, data, layout, and analysis. These are various operations that are performed on each file to classify (file and data), organize (layout), and analyze (analysis) the files.

Plugins

Most of the processes are based on the idea of a “plugin”. A plugin is a discrete class (typically in its own NuGet package) that is versioned and specific to the current project. They are identified by the plugin property inside the list. A plugin can be used more than once.

analysis:
    - plugin: Echoes
	  id: Words Within 10
      select: file.chapter token.word
      scope: token.word:within(10)
      threshold: { error: 5, warning: 2 }
    - plugin: Echoes
	  id: Sounds Alike
      select: file.chapter token.word
      scope: token.word:within(5)
	  compare: :root:attr(soundex)
      threshold: { error: 5, warning: 2 }

In each case, the plugin property of each plugin identifies which plugin to use. This is usually the base class but it means I'll have to have a registry to list which plugins do what. The reason for doing this is because if someone else writes a plugin (extends AI), they can push it up to NuGet (about that later) and everyone can use it. It doesn't require a release of the main code (assuming AuthorIntrusion.Contracts remains the same).

The rest of the properties are based on that plugin. There is a bit of interesting complexity in making this work (e.g., hack but I wrote it) but everything is type safe when it comes to coding.

Selectors

In the above examples, select and scope are based on CSS selectors. I couldn't find a library that did it generically, so I wrote one that just creates an abstract syntax tree of CSS selectors which is used by this library. Like most of the plugins, I'll break it into a separate project once AI gets stable.

The advantage of using CSS, such as file.chapter token.word or :root is that many developers understand how CSS selectors works. Except for the specifics (pseudo-classes for example), how they chain together, how you combine them, those are known enough that I don't have to force someone to learn something new.

I'm setting up the layout of a file (using the layout plugins) to look like HTML. Those plugins create the tag/element structure that CSS uses.

Right now, the following are implemented:

:first-child
:last-child
:nth-child
:root is the top-level item being selected (usually the file but can be something else)

There is also scoped pseudo-classes. These are used to get elements based on their relationship to another element. In every case so far, the scope property is based on each element found in the select of the plugin.

:before(x) finds the elements before the scope. It can be used like token.word:before(5) to get the five words before the current one.
:after(x)
:within(x) which is basically :before(x) combined with :after(x).
:self- versions of the three above.
:parent matches an item that is the parent of the scope.

These scoped variables are used so we can control what we are looking for. For example, the echoes plugin may look to see if the same word has been used somewhere within ten words of the current one:

analysis:
    - plugin: Echoes
      select: token.word
      scope: token.word:within(10)

It can also be used to make sure the same word isn't used at the beginning of the surrounding paragraphs:

analysis:
    - plugin: Echoes
      select: para token.word:first-child
      scope: para:within(1) token.word:first-child

There is a third category used for comparisons (used by the Echoes). These are where I hack the CSS system but basically let me define operations.

:lower get a lowercase version of the text.
:attr(attribute-name) uses the value of the attribute instead of the text.

This lets us analyze for something more than just the text. For example, words that sound the same:

analysis:
    - plugin: Echoes
      select: token.word
      scope: token.word:within(10)
	  compare: :root:attr(soundex) # :attr() isn't done yet

Or ones that have the same base word:

analysis:
    - plugin: Echoes
      select: token.word
      scope: token.word:within(10)
	  compare: :root:attr(stem)

A third example is if we are looking for too many sentences that start with the same pattern of parts of speech. This is to find where “bob did this”, “bob did that”, “mary did something”.

layout:
    - plugin: ClassifyElementRange # Not done yet
	  select: sent token.word:first-child
	  scope: sent:parent token.word:self-or-after(2)
	  tag: sent-leading
analysis:
    - plugin: Echoes
      select: sent-leading
      scope: sent-leading:within(5)
	  compare: :root:attr(pos)

File Plugins

Now, how all those topics are used. The first set of plugins are simply based on the structure of the project. Every file has one automatic element, the file which is the html of the file. We can add classes and identifiers to that to limit how the rest of the code works.

The biggest one is AddClassFromPath. You can use that to add the chapter class to files in the chapters directory. That way, the later selectors can use file.chapter or .chapter to limit processing (don't need to do grammar on your notes).

file:
    - plugin: AddClassFromPath
      match: chapters/*.markdown
      class: chapter

Data Plugins

I write using Markdown with a YAML header. My chapters look like this:

---
availability: public
when: 1471/3/28 MTR
duration: 25 gm
date: 2012-02-18
title: Rutejìmo
locations:
  primary:
    - Shimusogo Valley
characters:
  primary:
    - Shimusogo Rutejìmo
  secondary:
    - Shimusogo Hyonèku
  referenced:
    - Funikogo Ganósho
    - Shimusogo Gemènyo
    - Shimusogo Chimípu
    - Shimusogo Yutsupazéso
concepts:
  referenced:
    - The Wait in the Valleys
purpose:
  - Introduce Rutejìmo
  - Introduce Hyonèku
  - Introduce naming conventions
  - Introduce formality rules
  - Introduce the basic rules of politeness
summary: >
  Rutejìmo was on top of the clan's shrine roof trying to sneak in and steal his grandfather's ashes. It was a teenage game, but also one to prove that he was capable of becoming an adult. He ended up falling off the roof.

  The shrine guard, Hyonèku, caught him before he hurt himself. After a few humiliating comments, he gave Rutejìmo a choice: tell the clan elder or tell his grandmother. Neither choice was good, but Rutejìmo decided to tell his grandmother.
---

> When a child is waiting to become an adult, they are subtly encouraged to prove themselves ready for the rites of passage. In public, however, they are to remain patient and respectful. --- Funikogo Ganóshyo, *The Wait in the Valleys*

Rutejìmo's heart slammed against his ribs as he held himself still. The cool desert wind blew across his face, teasing his short, dark hair. In the night, his brown skin was lost to the shadows, but he would be exposed if anyone shone a lantern toward the top of the small building. Fortunately, the shrine house was at the southern end of the Shimusogo Valley, the clan's ancestral home, and very few of the clan went there except for meetings and prayers.

Given that, I would use a plugin to let me add identifiers and classes to files based on the YAML header (the bit between the --- lines). For example, the following would add a pov-ShimusogoRutejìmo class to the file.

data:
	- plugin: AddClassFromData # Not done
	  select: file.chapter
	  class: pov-{characters.primary}

There will be ways of manipulating it, but basically it lets you tag the chapter with “scene” or “sequel” if you follow the Techniques of the Selling Writer or want to identify if a chapter is combat or talking. It doesn't matter how you want to tag it, you can use any element in the data to filter or make decisions later. This will also be used for the querying to let you say “what chapters are from Rutejìmo's point of view” or “what scenes happen at night”. Eventually, I'll tie it into my culture library so you can also say “show me the chapters in chronological order”.

Layout Plugins

The bulk of my effort in the last two weeks has been in the layout. This is what carves up the contents of the file into something that can be selected via the CSS system. The default is only to have the file element which contains everything.

layout:
    - plugin: SplitLines
	  tag: line # implied

The above splits everything into <line> elements. We need that for reporting line number errrors. Below, we have a plugin that splits things into paragraphs based on Markdown rules (a blank line separates a paragraph). Using the above examples, this is what lets us use the para .word:first-child for selectors.

layout:
    - plugin: SplitParagraphs
	  tag: para # implied

It gets complicated when we start adding tokens. A token is a word or puncutation. I'm using OpenNLP to break apart the words at the moment. This splits up the contents of the paragraph (because of the select: para) into tokens.

layout:
    - plugin: OpenNlpSplitTokens
      select: para

Once I have tokens, I can add the word class to the actual words.

layout:
    - plugin: WordTokenClassifier
	  class: word # implied

Eventually, there will be a OpenNlpSplitSentences plugin where you would split the paragraph into sentences and then split the sentences into tokens. That isn't done but we don't need it for the base. I will also eventually create a ParagraphSentenceWord plugin that does most of this and makes it easier to add it as a single line.

The end result is that we have an abstract tree that represents the file. Eventually there may be a lot more such as dialog identification, blockquotes and epigraphs, and whatever else makes sense.

The main reason for the layout plugins is to make the selectors work for the analysis. It is also a complex bit of code that has to run in order unlike analysis which can be multi-threaded for a given file.

Analysis Plugins

Finally, the last bit. I'm not done with anything else, but the analysis plugin is the bit that finds errors and warnings. It is the parts that “does” something. In other words, the entire reason I'm writing this.

analysis:
    - plugin: Echoes
      select: file.chapter token.word
      scope: token.word:within(10)
      threshold: { error: 5, warning: 2 }

I'm working on this next, but I hope to have aicli check produce an error like gcc or a complier that gives a specific file and line number (via the SplitLines plugin) that lists the errors. It will be the same thing Atom uses to highlight problems.

Since analysis plugins only add/remove errors, they don't change the structure. This means they can use all the CPUs using worker processes to make it as efficient as possible. I'm also going to eventually have some optimizations put in (most elements are hashed so they can be cached).

Status and Plans

This last week, I've gotten the folllowing:

CSS selectors are working.
CSS pseudo-classes working for scope to get ranges of words.
CSS for doing transformations to make text lowercase before comparison.
The basic layout plugins needed to break a file into lines, paragraphs, tokens, and words.
A search path for plugins can find assemblies in a different directory.
The starting code for using NuGet for packages is there.
Echoes can find the words, just not report them.

The goal is to get a simple echo words analysis done, basically looking for duplicates within ten words.

Get TextQuery to compare text between the select and the scope.
Add warnings and errors (notices) to elements.
Report the notices to the console for the aicli check command.

I really want to get this to the point it can be used. I think it will be benefical even with the basics for finding problem spots with my personal projects and can benefit others. I also think that once I get the end-to-end, additional functionality will be easily added for other needs.

Development

Development is currently on the drem-0.0.0 branch on Gitlab. I'm adding issue of secondary items while tagging them with expected complexity. While I don't expect anyone to contribute, if someone does get inspired the simple and trivial items are good starting points.

I plan on adding notes for contributing “soon”.

So far, everything is MIT licensed but there will be other licenses already involved. OpenNLP is Apache licensed so I'll need to figure out how that interacts with MIT. Once I split the packages into individual files, it will probably be easier but for the time being, life is faster if I keep it as a single repository.

Author Intrusion 0.1.0 "James White"

2013-07-01T05:00:00Z

After five weeks, I finally got Author Intrusion to a stopping point enough to make its first release. This is version 0.1.0 and is dedicated to James White since his story, Fast Trip, is actually what inspired me to create AI in the first place.

This release is neither pretty nor smooth. It only has five menu items (all hard-coded of course), limited keyboard shortcuts, and configuration is done by editing the XML file and reloading (which it also doesn't support).

The important part is that it works and I think it is stable enough to write a novella with. So, in the next few weeks, I'm going to write long piece with this and figure out what needs to be done. Along the way, I'm going to figure out what really doesn't work. The ones I cannot live without, I'll fix for next month, otherwise I'll start getting them over time.

Why release?

The biggest question I had was my decision to call this a release. In the end, it came down to building up a rhythm and also accountability. I've been working on Author Intrusion for three years now and never got anywhere. I would get to a certain point and just stop to work on other things. If I put it out as a release, I have something out there to help me focus on its development and still get some writing done.

The other reason is that even a novel starts with a single word. First drafts are terrible things as is this. I never had the raw talent to be a decent writer or a programmer. Yes, I had a few advantages on the programming side, but mostly I got to where I am by steadily moving forward. If I want AI to work, I have to do the same thing.

Just keep swimming. -- Dori, Finding Nemo

Links

I decided to create a SourceForge project for now, mainly for file distribution. I'll put it up on my own site also, but that's running a bit behind while I work on a few other things. Hopefully by the end of the week, I'll have more than just a plain page on the Moonfire Games site.

There is only a MSI installer for Windows. For everything else, you'll need source code, a lot of luck, and probably a wide variety of curses to interpret what I've done. I went with WiX for the installer and it is built with every release which means I should hopefully be able to catch errors early and often.

Though, if you are looking for something remotely more capable than EDIT on DOS 3.1, you probably don't want to use this.

The final week

This last week was cleaning up code. I ended up doing a lot of reorganization and refactoring to smooth over some of the more heinous parts of the code. Projects that I thought should be combined got merged, then split, and then re-merged together. Right now, I think I have a good layout of projects for moving forward.

I did end up creating a placeholder package for Gtk#, Mono.Cairo, and Mono.Posix because I needed to move those dependencies out of my code.

Everything went smoothly right up to Saturday night. On Saturday, I realized I hadn't tested on Linux (one of my tenants of this project is to run on both Linux and Windows and eventually a Mac). It didn't go well. Libraries that I was pretty sure I tested ended up not working. I also did this project with the 4.5 framework instead of 3.5 which I tested earlier; this meant I needed the latest version of Mono (thanks Meebey).

The biggest thing that didn't work was the inversion of control system I was using. Ninject. It didn't work with any version of Mono. Well, the main one may have worked but the "by conventions" extension I was using didn't and I gave up. Since I already knew StructureMap worked, I switched over to that (only a single class) which worked beautifully. I spent a few more hours trying to get Ninject working but gave up.

The other was the spell-checking system, NHunspell. This one I didn't check on Linux and it didn't work. It might be a minor thing, it may not. But, I tried to make the spelling framework simply enough I can drop in something else at a later date.

What's next?

There are quite a few things for the next month. The biggest is that I'm going to write something with it. Actually use the program and figure out what are the worse pain points. Those that block me, I'll fix in July. The others will be thrown on the issues list so I don't lose them.

I also have a few more blog posts that I think would be helpful to write. Mostly thoughts I had in the last few weeks, some decisions I made, and general informational things that may help a Google search in the future.