Programming2024-03-28T17:39:17Zhttps://d.moonfire.us/categories/programming/D. MoonfireCreative Commons Attribution-NonCommercial-ShareAlike 4.0 InternationalMfGames.Markdown.Gemtext v1.2.12022-02-17T06:00:00Zhttps://d.moonfire.us/blog/2022/02/17/mfgames-markdown-gemtext-1.2.1/New features for MfGames.Markdown.Gemtext, tables!
<p>As part of yesterday's post, I wanted to create a table in Gemtext that I could also produce in my Markdown to HTML pipeline. However, that was one step I skipped because it was overwhelming when I first wrote. However, the need to rant and warm pushed me over the board so I implemented tables.</p>
<h1>What is MfGames.Markdown.Gemtext?</h1>
<p>Let's start with the biggest question. This is a library that uses <a href="https://github.com/xoofx/markdig">Markdig</a> to parse Markdown and output <a href="https://gemini.circumlunar.space/docs/gemtext.gmi">gemtext</a> instead of HTML. This is the limited markup format used by Gemini browsers which is kind of a low-overhead and simplified browsing.</p>
<p>The limitations of Gemtext are a big one. There are a few I struggle with, specifically a lack of italics and bolds, but it also doesn't let you have more than one link per paragraph, no Javascript, and generally is “what you see is exactly what you get”. For a writer, it fits nicely with that especially when used with a lovely client such as <a href="https://github.com/skyjake/lagrange">Lagrange</a>.</p>
<p>I wrote the library inspired by <a href="https://github.com/makeworld-the-better-one/md2gemini">md2gemini</a> which creates a “reasonable” output from Markdown which is my native writing format for novels and posts.</p>
<p>At the moment, I don't have a SSL certificate to sign the library so I can't post it on NuGet, so I have my libraries at <a href="https://www.myget.org/feed/Packages/mfgames">MyGet</a>. In specific, <a href="https://www.myget.org/feed/mfgames/package/nuget/MfGames.Markdown.Gemtext">MfGames.Markdown.Gemtext</a> gives the information for downloading and using it.</p>
<p>Note, if you want want to multiple NuGet sources, the following <code>NuGet.config</code> works well for my packages:</p>
<pre><code><?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
<add key="mfgames" value="https://www.myget.org/F/mfgames/api/v3/index.json" protocolVersion="3" />
</packageSources>
</configuration>
</code></pre>
<p>I have this checked into any project that uses them. I have… opinions about single source repositories but getting a SSL certificate isn't quite in my cards for at least half a year.</p>
<p>The source is located on <a href="https://gitlab.com/mfgames-cil/mfgames-markdown-gemtext-cil">Gitlab</a> along with ticket-tracking and the usual slew of tools.</p>
<h1>Using MfGames.Markdown.Gemtext</h1>
<p>The bulk of the code can be seen in the unit tests, but I want to work on better documentation. It's just waiting for <a href="/tags/nitride/">Nitride</a>. Until then, here are some basics.</p>
<pre><code>// using MfGames.Markdown.Gemtext;
string input = "This is input.";
string expected = "This is input.";
string actual = MarkdownGemtext.ToGemtext(input);
Assert.Equal(expected, actual);
</code></pre>
<p>This is pretty much how the base Markdig processing is done. I had to jump through a few extra hoops because of hard-coding, but overall, this is it.</p>
<p>The biggest thing is converting a paragraph with links into something Gemtext. There are a couple of formats, but my preferred (and the one I use on my <a href="gemini://d.moonfire.us/">Gemini version of this site</a>) is a table of links after the paragraph.</p>
<pre><code>var input = string.Join(
"\n",
"This is a paragraph with an [inline](https://example.com) link.",
"Here is [another](https://example.org/) link, part of the same paragraph.",
"",
"This is a second paragraph, with a different [link](https://duck.com) in it.");
string actual = MarkdownGemtext.ToGemtext(
input,
new MarkdownPipelineBuilder()
.Use(new SetBlockLinkHandling(BlockLinkHandling.ParagraphEnd))
.Build());
Console.WriteLine(actual)
/* Output is:
This is a paragraph with an inline[10] link. Here is another[11] link, part of the same paragraph.
=> https://example.com 10: https://example.com
=> https://example.org/ 11: https://example.org/
This is a second paragraph, with a different link[12] in it.
=> https://duck.com 12: https://duck.com
*/
</code></pre>
<h1>Tables</h1>
<p>Tables are not part of Gemtext. Before this version, if you passed in a table such as:</p>
<pre><code>aaa|bbb|ccc
--:|---|:-:
1|2|3
4|5|6
</code></pre>
<p>You would get everything on a single line.</p>
<pre><code>aaa|bbb|ccc --:|---|:-: 1|2|3 4|5|6
</code></pre>
<p>With this version, you can pass a table parsing that produces a normalized table based on formatting using the excellent <a href="https://www.nuget.org/packages/ConsoleTableExt">ConsoleTableEx</a> library.</p>
<pre><code>//using MfGames.Markdown.Gemtext;
//using MfGames.Markdown.Gemtext.Extensions;
MarkdownPipeline pipeline = new MarkdownPipelineBuilder()
.Use(
new GemtextPipeTableExtension(
new GemtextPipeTableOptions()
{
OmitPreformatLines = true,
ConfigureTableBuilder = (x) => x.WithFormat(ConsoleTableBuilderFormat.MarkDown),
}))
.Build();
string input = string.Join(
"\n",
"aaa|bbb|ccc",
"--:|---|:-:",
"1|2|3",
"4|5|6");
string expected = string.Join(
"\n",
"| aaa | bbb | ccc |",
"|-----|-----|-----|",
"| 1 | 2 | 3 |",
"| 4 | 5 | 6 |",
"");
string actual = MarkdownGemtext.ToGemtext(input, pipeline);
Assert.Equal(expected, actual);
</code></pre>
<p>I went with the library because I like how <code>ConsoleTableEx</code> will make sure everything is lined up. Plus, you can change the formatting to change the borders around and between the cells, different layouts, and the like. Basically, the options configured in the <code>ConfigureTableBuilder</code> lets the developer set the format instead of me deciding on a One True Way™️.</p>
<h1>What's Next</h1>
<p>I have a couple minor little things with this library, but a lot of it is based on getting Nitride cleaned up and usable. This site uses Nitride and it runs nicely, but it isn't quite ready for prime time. So, I'm taking lessons learned, refactoring that library, and hopefully get it ready for the next big site (<a href="https://mfgames.com/">https://mfgames.com/</a>).</p>
fedran-cli2022-01-16T06:00:00Zhttps://d.moonfire.us/blog/2022/01/16/fedran-cli/I started working on a small tool, `fedran-cli`, to help me manage and work with the growing cluster of stories and novel Git repositories and automated processes.
<p>Due to some random happenstance, I was presented the opportunity to get a sensitivity reader for <a href="/tags/flight-of-the-scions/">Flight of the Scions</a>. I've been thinking about doing this for a few years, ever since I realized I rewrote the novel for no-so-great reasons but had the misfortune of hanging much of the phase-1 stories off of events in that book.</p>
<p>This forced me to integrate the edits I've been avoiding for the last six months and get them into the site. So, the version currently on my website would be the latest, but at the moment, there is a bug preventing me from actually showing them.</p>
<p>I know that <em>Flight</em> has been hanging around for decades (it was supposed to be my first published novel) but this is a step that I've been feeling like I need to do. Of course, my anxiety helpfully suggests that the reader is going to find the novel complete and utter trash so there is that. I'm hoping it isn't but I've never had a sensitivity reader before and I'm not sure what to expect.</p>
<p>To distract myself from the impending complete rejection of my project, I needed to distract myself. I have a writing commission that I needed to finish, but that was limited to off-hours because of the concentration required and I couldn't do it with the kids.</p>
<p>To do something, and hopefully get a dopamine hit, I dug up a project that I had started a few times but failed: a program to help me work with my <a href="/tags/fedran/">Fedran</a> projects. In specific, something that would let me update and add <a href="https://fedran.com/sources/">characters and volumes sources</a> easily.</p>
<p>Along the way, I figured I'd include a few other items that have been bothering me. One of the biggest, which is needed for the fedran.com rewrite was to be able to check out the Git repositories in a single shot. That requires me to identify my stories, which is going to get more complicated since I recently (last year) decided that I was going to be purely one repository per story to handle versioning better.</p>
<p>According to my sources page, I have exactly eighty stories and novels. That means I need a central place to keep track of eighty repositories. I could maintain them in the website folder, but I thought tying it into my <code>identifiers.json</code> (formerly YAML) would be a good way of centralizing all the information about the sources in a single place.</p>
<p>Hence, I'm working on <code>fedran-cli</code> which is a CLI for working with the Fedran ecosystem. Since I recently started playing with <a href="/tags/nix/">Nix</a>, I wanted to figure out a way of integrating <code>fedran-cli</code> into that whole mess without me needing to do some formal release process. Originally, I tried a <span class="missing-link" data-path="/tags/csharp/">C#</span> version because I can't get C# to play with my new libraries in Nix. However, I could make it work in <a href="/tags/rust/">Rust</a> so I banged up a basic program that lets me choose the existing character and volume for a given project.</p>
<p>(I was going to upload a GIF to this post, but I helpfully used the <code>rm</code> command instead of <code>mv</code>, so here is a <a href="https://octodon.social/@dmoonfire/107616954866735580">link to the post with a video including alt text</a> on my social page).</p>
<p>For a non-trivial Rust application, I was pretty happy with the results. I have a number of other commands I need to write for basic functionality and what I consider minimum viable product.</p>
<ul>
<li><code>fedran-cli characters add</code> to add new characters.</li>
<li><code>fedran-cli volumes add</code> to add new volumes.</li>
<li><code>fedran-cli project sync</code> to upload a subset of information to the identifiers file including the Git repository.</li>
<li><code>fedran-cli git clone --all</code> to basically clone the eighty sources into the current directory based on the identifier file.</li>
</ul>
<p>This is kind of an untethered project for me. The CLI doesn't really have an end, just an “end for now”. But, I think if I focus on getting to the <code>git clone --all</code>, I should have something that can help me.</p>
generator-mfgames-nix-project v1.4.1 and generator-mfgames-writing 1.0.02021-12-19T06:00:00Zhttps://d.moonfire.us/blog/2021/12/19/yeoman-generators/A new Yeoman generator and a refresh on a second one.
<p>I create a lot of projects. I think between Github and Gitlab, I have a few hundred now. Much if it is because I started pushing toward strictly being one story per repository (OSPR). This means I have a lot of common patterns that not only do I have to set up but also maintain as my skills and knowledge evolve. The problem comes when I figure out a new pattern, I have to update a couple hundred repositories if I want to keep them consistent. Needless to say, this is one drawback of a monorepo but there are a lot of advantages of OSPR.</p>
<p>A lot of my inspiration comes from Larry Wall's Three Virtues of a Great Programmer:</p>
<ol>
<li>Laziness: The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful and document what you wrote so you don't have to answer so many questions about it.</li>
<li>Impatience: The anger you feel when the computer is being lazy. This makes you write programs that don't just react to your needs, but actually anticipate them. Or at least pretend to.</li>
<li>Hubris: The quality that makes you write (and maintain) programs that other people won't want to say bad things about.</li>
</ol>
<p>This means, instead of manually doing it, I want to find a nice automated way (virtue one) that I can script and have it run across everything at once (virtue two).</p>
<p>Previously, I had used <a href="https://yeoman.io/">Yeoman</a> but I was wondering if I needed to break out of that to work on a “me-specific” CLI tool. That line of thought came down because so many of the tools I'm using all package their own custom CLIs. While I do enjoy writing a good CLI, I don't want to write one if I can't.</p>
<p>In the process of finding an alternative, I finally understood how to compose Yeoman generators together. That means I could create a generic project generator and then layer the <a href="/tags/mfgames-writing/">writing</a> one on top of it.</p>
<p>At the same time, I could migrate away from <a href="/tags/asdf/">asdf</a> and onto my preferred infrastructure du jour, <a href="/tags/nix/">Nix</a>.</p>
<p><em>Side Note: Let me be honest, I'm constantly evolving my structures and they change as I learn new tools and my processes change with me. I won't be on Nix forever, but right now, Nix works out very nicely for what I want it to do.</em></p>
<p>This weekend I pulled something together and actually got a working version by tonight. This involves three projects:</p>
<ul>
<li><a href="https://gitlab.com/mfgames-writing/mfgames-writing-js/">mfgames-writing</a>: My publishing framework for Markdown and YAML files.</li>
<li><a href="https://gitlab.com/mfgames-js/generator-mfgames-nix-project">generator-mfgames-nix-project</a>: A Yeoman generator for creating a basic project layout.</li>
<li><a href="https://gitlab.com/mfgames-writing/generator-mfgames-writing-js">generator-mfgames-writing</a>: A Yeoman generator for setting up a writing project.</li>
</ul>
<h1>generator-mfgames-nix-project v1.4.1</h1>
<p>I pulled out the basic project stuff from <code>generator-mfgames-writing</code> and made it a dedicated generator while improving it. It still asks a bunch of questions, but mainly it does the following:</p>
<ul>
<li>Set up <a href="/tags/semantic-release/">semantic-release</a> to handle non-romantic releases. “Non-romantic” meaning automated and every time I push up to <code>main</code>.</li>
<li>Set up <a href="/tags/conventional-commits/">Conventional Commits</a> to drive the semantic release process. This is why my check-ins start with <code>feat:</code>, <code>fix:</code>, or something other tag. That way, I can just focus on a given commit being a new feature, fixing something, or breaking everything.</li>
<li>Set up <a href="/tags/commitlint/">commitlint</a> to make sure the commits are properly handled for the semantic releasing.</li>
<li>Set up <a href="/tags/husky/">Husky</a> to make sure commitlint is automatically handled.</li>
<li>Configure <a href="/tags/prettier/">Prettier</a> to make everything pretty and normalized. This mostly helps if someone ever contributes, but I like things being consistent and neat. While I disagree with Prettier on many things, it does a fantastic job.</li>
<li>Set up a <a href="/tags/nix/">Nix flake</a> to represent all the build tools needed to build the project.</li>
<li>Set up <a href="/tags/direnv/">direnv</a> to automatically ‘enter’ the environment when you go into the directory.</li>
<li>Write out the files needed to run <a href="/tags/gitlab/">Gitlab's CI/CD</a> on push and generate all the files and automatically release.</li>
<li>Set up <a href="/tags/editorconfig/">EditorConfig</a> for normalized formatting.</li>
</ul>
<p>That's a lot of work and it was getting tedious as I migrated/created new projects over the last month or so as I figured out the pattern. I like it, though it is opinionated. Previously, I hated that word, but this is honestly how <em>I</em> work and I think it's a reasonable pattern for any writing or coding project.</p>
<p>Running it is pretty simple:</p>
<pre><code>$ npm install -g generator-mfgames-nix-project yo
$ mkdir name-of-project
$ cd name-of-project
$ yo mfgames-nix-project
... lots of prompting
</code></pre>
<p>Overall, I like the resulting project and I'm going to be testing it fairly heavily in the new few weeks.</p>
<h1>generator-mfgames-writing v1.0.0</h1>
<p>Most of the intial work for <code>generator-mfgames-nix-project</code> came from <code>generator-mfgames-writing</code>, but I ended up also expanding that over the weekend to ask better questions about theme, author information, and the release process. It works the same as the project generator.</p>
<pre><code>$ npm install -g generator-mfgames-writing yo
$ mkdir name-of-project
$ cd name-of-project
$ yo mfgames-writing
... a different set of questions
$ npm run build
... lots of noise
$ ls *.epub *.pdf
name-of-project-0.0.1.epub
name-of-project-0.0.1.pdf
</code></pre>
<p>This one bypasses some questions that the project one does (build, test, and release questions) and adds some other ones (theme, configuration file format, title of the work), but otherwise works the same as the other. The end result is something you should be able to build and get the output immediately.</p>
<p>I honestly think that is really cool.</p>
<h1>mfgames-writing</h1>
<p>There were a bunch of versions with this, but I needed to tweak some processes because I found some bugs in these tools while writing the generators. Sadly, a “simple” fix of moving <code>rimraf</code> from a developer dependency to a normal dependency required me to upgrade <a href="/tags/typescript/">TypeScript</a> to the last version and accept a wall of warnings because some of the linting tools don't like the latest version.</p>
Using Nix2021-11-30T06:00:00Zhttps://d.moonfire.us/blog/2021/11/30/nix/Today, I finished putting the final touches on additional features for `generator-mfgames-writing`, my scaffolding generator for Yeoman that lets me set up writing projects quickly.
<p>Apparently, it's been a few months since I've blogged about anything. That isn't to say I wasn't thinking about it. I would find myself working on a little bit of code and think “this would be something interesting to talk about” or come up on a post about elitism that I want to respond. But then… it didn't happen.</p>
<p>The last four month have been probably one of the hardest I've had in a long time, even over the struggles of 2020's derecho and pandemic. It started with just a bit of stress at work, but then something got dumped on top of it. And then something on top of that, then on that, and then on that. At one point, for every time I took off my task list, two more were added.</p>
<p>A good example is while I was fixing Partner's toilet that wasn't flushing (just a simple stalk replacement) and our youngest comes up the stairs because the downstairs toilet basically exploded (water pouring out of the bottom, ruptured wax seal, no longer flushing, soaking the carpet). I've been trying to get around to it for almost two months and <em>just</em> removed the carpet today to find out how much drywall damage was done.</p>
<p>There was a lot more, but going through them just turns into a story like <a href="https://www.imdb.com/title/tt0310357/">Willard (2003)</a> where the main character starts off in the dump, for a brief moment sees hope, and then plunges even further into the hole. Since I try to keep my blog posts positive, let's just go with that.</p>
<p>One of those bright points was getting <a href="https://weirdauthor.com/merger">Merger of Evil</a> published. Shannon has always been a fantastic author and I'm glad I got to see the sequel to <a href="https://weirdauthor.com/evil">Minion of Evil</a> in print. Plus, the cover is fantastic (even if I didn't do it).</p>
<p>My brother says my life is frequently a “that's good, that's bad” game. while I was able to do my part in getting <em>Merger of Evil</em> out, I was unable to finish <a href="/tags/second-hand-dresses/">Second-Hand Dresses</a> or <a href="/tags/flight-of-the-scions/">Flight of the Scions</a> out this year, which I wanted to. The reasons are worth of a different post, but basically I had to scoop things off my pile to avoid drowning.</p>
<p>Working on <em>Merger of Evil</em> hit a couple of technical annoyances. A few months ago, I started using <a href="/tags/asdf/">asdf</a> to let me switch environments more fluidly. The project for <em>Merger</em> seemed like a great place to use that, but somewhere in that time, there were some updates to Node and various packages that turned a minor maintenance task into something far more complicated simply because I couldn't get into the right head space.</p>
<p>As a side note, when I was forced from my apartment in 2008 because of the flood in Cedar Rapids, I had a few months of high stress that also made things difficult to function. Much like the the last few months, minor problems became complicated when there were dozens of other things competition for my time and attention. When I got back to my apartment, I found that my old life didn't quite fit anymore. My daily ritual of going through 20-30 web comics and sites then seemed pointless so I purged my RSS feed, I stopped playing <a href="https://www.kingdomofloathing.com/">Kingdom of Loathing</a>, and basically had to review what “sparked joy” in my life.</p>
<p>Those technical annoyances got me to thinking about improvements. Yeah, I had a bunch of other things demanding my attention, but <em>optimizing a process</em> is one of the ways I reduce stress (along with <em>rewriting problems</em>) and I desperately needed that distraction. As things were, someone on my social feed started talking about migrating from asdf to <a href="https://nixos.org/">Nix</a>. Nix is a lot like the old GNU <a href="https://www.gnu.org/software/stow/">Stow</a> but had a much better package management and, more importantly, allowed me to have seamless transitions to different versions. Nix also is more of a holistic approach since one could use <a href="https://nixos.wiki/wiki/Flakes">flakes</a> to wrap everything including specific Node, Python, and C# packages into a single unified whole.</p>
<p>After a few weeks of using Nix, I can say that it definitely solves the little nits and scratches that asdf had plus handles my need for reproducible/consistent builds. I'm sure there are other problems with it, starting with the complexity of the setup files and the fact the featuers I want are still in beta, but I think I'm going to stay with it and see if it works out.</p>
<p>Things are still in flux and life is still heaping up on my shoulders, which means there is only one thing I can do: break apart the problems as much as I can and then focus on one thing at a time.</p>
semantic-release-nuget v1.1.02021-09-04T05:00:00Zhttps://d.moonfire.us/blog/2021/09/04/semantic-release-nuget/To pair with last's week utility, I finished up the first version of semantic-release-nuget and used it.
<p>Last week, I wrote <a href="/blog/2021/08/29/semantic-release-dotnet/">semantic-release-dotnet</a> which was a <a href="https://semantic-release.gitbook.io/semantic-release/">semantic-release</a> plugin to automatically set the version to the appropriate one before building.</p>
<p>One of the key parts missing from the normal .NET development cycle was also publishing the packages. I decided to break that into a separate plugin because I have a number of places where I don't want to publish but I do want something versioned (internal projects and customized deployments). So, in the essence of the <a href="https://en.wikipedia.org/wiki/Single-responsibility_principle">Single Responsibility Principle</a>, I created a second utility which does one thing: build and publish NuGet packages.</p>
<p>Introducing <a href="https://www.npmjs.com/package/semantic-release-nuget">semantic-release-nuget</a>. It doesn't have a lot of configurations, but the documentation covers all of them. Basically, it does one thing.</p>
<p>I mostly tested with <a href="/tags/mfgames-locking/">MfGames.Locking</a>, my CIL library for some thread-locking patterns, because I'm the process of carving out <a href="/tags/gallium/">Gallium</a> and <a href="/tags/nitride/">Nitride</a> into their own packages. I'm just not sure where to put them, so they are probably going in my <a href="https://gitlab.com/mfgames-cil/">Gitlab</a> organization until I find a “better” organization/home.</p>
semantic-release-dotnet v1.0.02021-08-29T05:00:00Zhttps://d.moonfire.us/blog/2021/08/29/semantic-release-dotnet/I wrote a little utility for `semantic-release` to handle versioning of .NET projects.
<p>I've pretty much <a href="/tags/semantic-release/">embraced</a> <a href="https://semantic-release.gitbook.io/semantic-release/">semantic-release</a> for most of my release processes, both for books and software. It has gone pretty smoothly, but occasionally I find myself hacking it to get something working. One of the more recent ones (and something that is going to affect me in the near future) is the interaction between semantic-release and .NET projects.</p>
<p>These are two very different systems, including packaging, and there are plenty of NuGet packages to do semantic-release as a .NET project, but I use Node for my package management, thanks to <a href="https://typicode.github.io/husky/#/">Husky</a>. So, after a year or so of writing up little ad-hoc programs, I decided to write a formal package.</p>
<p>Introducing <a href="https://gitlab.com/dmoonfire/semantic-release-dotnet/">semantic-release-dotnet</a>. It is basically a “prepare” plugin for <code>semantic-release</code> and writes out the <code><Version/></code> element into <code>Directory.Build.props</code> or a <code>.csproj</code> file. It uses minimatch (via glob) to update the files, so you can edit the project files directly and it will write out certain files if they are missing.</p>
generator-mfgames-writing v0.3.22021-07-31T05:00:00Zhttps://d.moonfire.us/blog/2021/07/31/generator-mfgames-writing/Today, I finished putting the final touches on additional features for `generator-mfgames-writing`, my scaffolding generator for Yeoman that lets me set up writing projects quickly.
<p>Related to process of writing from <a href="/blog/2021/07/27/commitlint-gitlab-ci/">a few days ago</a>, I've been working on another tool that I find myself using fairly often: <a href="https://yeoman.io/">Yeoman</a>. This is a scaffolding tool, which basically means it asks a few questions and then sets up a project. In my case, it does a lot of the drudge work of creating a writing project and getting it ready for <a href="https://gitlab.com/">Gitlab's CI</a> which I also use heavily (and why I wrote <code>commitlint-gitlab-ci</code>).</p>
<p>Like many of the writing tools, this is based on Javascript (but not Typescript this time).</p>
<pre><code class="language-shell">$ sudo npm install -g yo generator-mfgames-writing
</code></pre>
<p>The first thing it does is ask a number of questions about the project. I'll use <a href="https://fedran.com/nor-curse-be-found/">Nor Curse Be Found</a>, my Beauty and the Beast sequel, as an example.</p>
<pre><code class="language-shell">$ mkdir nor-curse-be-found
$ cd nor-curse-be-found
$ yo mfgames-writing
? The title for your novel or story: (Nor Curse Be Found)
</code></pre>
<p>All of my writing projects have a unique “slug” to describe them. This is used for the website's URL, such as <a href="https://fedran.com/nor-curse-be-found">https://fedran.com/nor-curse-be-found</a>, and also the name of the output (<code>dmoonfire-nor-curse-be-found-0.0.1.epub</code>). I use a library to pull out the current directory's name and try to make a reasonable title out of it. I can change the title to something more accurate, maybe with apostrophes or possessives, but generally I keep the slug pretty even. In this case, the parenthetical item is the suggested output, so I can just hit return.</p>
<pre><code>? The title for your novel or story: Nor Curse Be Found
? The slug based on the title: (nor-curse-be-found)
</code></pre>
<p>Just in case the directory doesn't match the directory, I use a library to take that given title and create a slug out of it. One of the biggest things, my <a href="http://www.sectorgeneral.com/shortstories/fasttrip.html">Fast Trip</a> rule, is that I don't want to tell someone how to work. That is why I called this semi-opinionated in that there are a lot of options but the default is how <em>I</em> work.</p>
<p>There is also a question about bylines, but then we start to get into the technical bits.</p>
<pre><code>? The title for your novel or story: Nor Curse Be Found
? The slug based on the title: nor-curse-be-found
? Author's name or byline: D. Moonfire
? Do you want to create the Git repository? (Y/n)
</code></pre>
<p>This came from some years ago, but I do one story per repository, more so if I think the story has a possibility of being a bigger piece (which many of mine are), I want to track the versions independently, or I might be sending to an editor or submitting it somewhere (not likely these days, I self-censor a lot there). Since this is so common, I put it in as part of the questions.</p>
<p>I have a slowly evolving set of practices when it comes to writing processes. One of the first ones is setting up a Git repository. This always starts with configuring <a href="https://typicode.github.io/husky/#/">husky</a>, <a href="https://www.conventionalcommits.org/en/">semantic-release</a>, <a href="https://www.conventionalcommits.org/en/">conventional-commits</a>, and <a href="https://commitlint.js.org/#/">commitlint</a>. (I'm working on updating my <a href="https://www.npmjs.com/package/generator-mfgames-writing">Yeoman generator</a> to make this a lot easier, but that will be later this week.)</p>
<pre><code>? Do you want to create the Git repository? Yes
? Any command to run after `git init`? set-moonfire-git-user
</code></pre>
<p>I also write for a number of different projects (and bylines) which means I don't always use the same Gitlab login or even email. To faciliate this, I do <em>not</em> have a global <code>user.name</code> and <code>user.email</code> set up in Git so it blows up if I try to make a commit without setting a local one. Since that is tedious, I write a bunch of scripts that set the user information and SSH keys (<code>set-moonfire-git-user</code> for writing, <code>set-mfgames-git-user</code> for programming for example) and I want to call them before I do that first commit. If I just hit return to have a blank, then nothing extra will happen.</p>
<pre><code>? Any command to run after `git init`? set-moonfire-git-user
? Do you want to create the initial commit? Yes
? Do you want to create the initial Git tag? Yes
? What version do you want to start? 0.0.1
</code></pre>
<p>Now, the initial commit is a messy one and a philosophical difference between me and the Semantic Release folks. They have <a href="https://github.com/semantic-release/semantic-release/blob/caribou/docs/support/FAQ.md#can-i-set-the-initial-release-version-of-my-package-to-001">their reasons</a> to hard-code the value to “1.0.0” but I'm specifically start this process <em>before</em> a release, so I feel it should be “0.0.1” because I want that tracking as soon as possible.</p>
<p>Since I don't have the ability to set it via a configuration file, I get around it by creating the initial commit and then allowing it to be tagged so it can be pushed up and get the starting version I want.</p>
<p>This is also because I tag versions I give to editors, beta readers, and even my writing group (when I went to it). The tagged version also shows up on the various Fedran pages and mainly it shows progression as I write.</p>
<p>So, <em>Fast Trip</em> so I jump through a few hoops. It should also be clear why I needed that command to run after <code>git init</code> to set my user, otherwise this part would blow up with a “user not set” message.</p>
<pre><code>? What version do you want to start? 0.0.1
? Which package manager do you use? (Use arrow keys)
❯ NPM
Yarn
</code></pre>
<p>Moving away from Git and into the build process, the first question is package management. I have a love/hate relationships with <code>npm</code>. The command works great, but I've been struggling with it (more on than off lately) so I've been using <a href="https://yarnpkg.com/">yarn</a> fairly heavily in the last year or so. But, with Yeoman I can give both so the user can use up or down keys to select and hit return to choose. Once they do, the menu goes away and is replaced with the choice.</p>
<pre><code>? Which package manager do you use? Yarn
? Do you want to use semantic releases? Yes
? Do you want to use conventional commits? Yes
? Do you want to use Husky? Yes
</code></pre>
<p>The next three are to set up the common functionality I use with releases. They set up <a href="https://semantic-release.gitbook.io/semantic-release/">semantic-release</a>, <a href="https://www.conventionalcommits.org/en/">conventional commits</a>, <a href="https://commitlint.js.org/#/">commitlint</a>, and <a href="https://typicode.github.io/husky/#/">Husky</a> which I use to make sure all the commit messages are clean. These four program have been a stable in most of my development lately, for coding and writing, for Typescript and even C#.</p>
<pre><code>? Do you want to use Husky? Yes
? Do you want to use CI/CD? Yes
? Do you use Gitlab for your CI/CD? Yes
</code></pre>
<p>I can't say enough for Gitlab's CI. Whenever I push up code to the server, it runs the entire publication process. Assuming it works, it then produces a zip archive with EPUB, MOBI, PDF, and other formats without me doing anything else. Every single time. With sematic releases and the rest, each one has a unique version so I don't have “final”, “final-2”, “final-2a”, etc. Just a clean version number that shows up everywhere, starting at 0.0.1.</p>
<pre><code>? Do you use Gitlab for your CI/CD? Yes
? Do you use asdf? No
</code></pre>
<p>This is another tool I use, <a href="https://github.com/asdf-vm/asdf">asdf</a>. This was also something that pretty much started last year, but one of the goals of <a href="https://gitlab.com/mfgames-writing/mfgames-writing-js/">mfgames-writing</a> was to be able to produce consistent output of novels even years later. Well, what I didn't realize is how much the Node ecosystem would change and Node 15 doesn't exactly run Node 8 packages very well due to deprecations and changing libraries.</p>
<p>This is where <code>asdf</code> comes in. It puts a small file (<code>.tool-versions</code>) which says which version of Node, Yarn, C#, Python, whatever and will automatically change to those versions when I <code>cd</code> into the directory. It basically rolls in all the functionality of <code>nvm</code>, <code>virtualenv</code>, and a bunch of other localized version libraries into one. Since I write and work across many languages, having one tool to manage that is easier for me. So, saying “yes” to the question copies that <code>.tool-versions</code> into the resulting directory.</p>
<pre><code>? Do you use asdf? Yes
? Which formats do you want to use? (Press <space> to select, <a> to toggle all, <i> to invert selection)
❯◉ PDF
◉ EPUB
◯ DOCX
◯ HTML
</code></pre>
<p>With this next question, we are out of the build section and into the formats. Basically, this just queries which types of files will be written out. The directions are pretty clear, but basically this will determine which of the format libraries are included.</p>
<pre><code>? Which formats do you want to use? PDF, EPUB, DOCX, HTML
? The name of the theme package to use: (@mfgames-writing/clean-theme)
</code></pre>
<p>I only have three themes right now, two are “generic” and one I prefer just for Fedran stories (but obviously don't prevent anyone from using it). Later, I hope to have more themes, but basically the choices are <code>@mfgames-writing/clean-theme</code> and <code>@mfgames-writing/greekil-theme</code> (a Gentium-based theme).</p>
<pre><code>? The name of the theme package to use: @mfgames-writing/clean-theme
Creating initial Git repository
Initialized empty Git repository in fedran/nor-curse-be-found/.git/
Running custom Git command
I'm all done. Running npm install for you to install the required dependencies. If this fails, try running the command yourself.
create package.json
force ../../../.yo-rc-global.json
force .yo-rc.json
create release.config.js
create commitlint.config.js
create .husky/commit-msg
create .gitlab-ci.yml
create .tool-versions
create publication.yaml
create yarn.lock
create README.md
create .gitignore
create chapters/chapter-01.md
Changes to package.json were detected.
Running yarn install for you to install the required dependencies.
</code></pre>
<p>With that, it just producing a wall of tell as it creates all the files, hooks up everything, and basically gets my entire project ready, including a sample first chapter, a really basic <code>README.md</code>, and a Git repository ready to push.</p>
<pre><code class="language-shell">$ ls -a
. .git node_modules README.md
.. .gitignore package.json release.config.js
chapters .gitlab-ci.yml package-lock.json .yo-rc.json
commitlint.config.js .husky publication.yaml
</code></pre>
<p>Right off the bat, I can build the files and see the output.</p>
<pre><code class="language-shell">$ yarn build:pdf
... lots of output
$ ls *.pdf
nor-curse-be-found-0.0.1.pdf
$ yarn build
... huge amounts of output
$ ls nor-curse-be-found*
nor-curse-be-found-0.0.1.docx nor-curse-be-found-0.0.1.epub
nor-curse-be-found-0.0.1.html nor-curse-be-found-0.0.1.pdf
</code></pre>
<p>I'm also ready to push up to Gitlab.</p>
<pre><code class="language-shell">$ git remote add origin git@gitlab.com:fedran/nor-curse-be-found.git
$ git push --tags
$ git push -u origin main
</code></pre>
<p>Hopping over to Gitlab, I can see it building immediately.</p>
<p><img src="./initial-push.png" height="256" style="margin-top:1em;" alt="Initial CI Run" /></p>
<p>As soon as it is done, it uploads the output as an artifact.</p>
<p><img src="./gitlab-log.png" height="256" style="margin-top:1em;" alt="End of Log" /></p>
<p>I can then click on the browse (or download) button on the right.</p>
<p><img src="./gitlab-download.png" height="256" style="margin-top:1em;" alt="Browse and Download Buttons" /></p>
<p>Browsing lets me see a list of all the outputs that I've added.</p>
<p><img src="./gitlab-browse.png" height="256" style="margin-top:1em;" alt="Browse Artifacts" /></p>
<p>And I can even click on the PDF to preview it.</p>
<p><img src="./gitlab-pdf.png" height="256" style="margin-top:1em;" alt="View PDF" /></p>
commitlint-gitlab-ci v0.0.42021-07-27T05:00:00Zhttps://d.moonfire.us/blog/2021/07/27/commitlint-gitlab-ci/Since I'm usually creating a new Git repo (about 3/month) and commitlint is one of the first things I set up, I ended up writing a little NPM utility to solve a bug that was causing me problems on Gitlab.
<p>I have a slowly evolving set of practices when it comes to writing processes. One of the first ones is setting up a Git repository. This always starts with configuring <a href="https://typicode.github.io/husky/#/">husky</a>, <a href="https://semantic-release.gitbook.io/semantic-release/">semantic-release</a>, <a href="https://www.conventionalcommits.org/en/">conventional-commits</a>, and <a href="https://commitlint.js.org/#/">commitlint</a>. (I'm working on updating my <a href="https://www.npmjs.com/package/generator-mfgames-writing">Yeoman generator</a> to make this a lot easier, but that will be later this week.)</p>
<p>Sadly, my Git host of choice, <a href="https://gitlab.com/">Gitlab</a>, has a little problem with <code>commitlint</code>. It blows up on the first few commits which lead to me reporting <a href="https://github.com/conventional-changelog/commitlint/issues/885">this</a> issue.</p>
<p>Since I'm usually creating a number of new Git repositories in a month and <code>commitlint</code> is one of the first things I set up, I ended up writing <a href="https://www.npmjs.com/package/commitlint-gitlab-ci">a little NPM utility</a> to handle the bug for me since the original package (rightfully, I feel) don't want to add Gitlab-specific code:</p>
<pre><code>npx commitlint-gitlab-ci -x @commitlint/config-conventional
</code></pre>
<p>It seems to work fairly well for me, I've been messing with it for a little while and haven't had too many complaints.</p>
Gallium Nitride and Gemini2021-07-10T05:00:00Zhttps://d.moonfire.us/blog/2021/07/10/gallium-nitride-and-gemini/Last November, I switched my static site generator from CobblestoneJS (my homebrew Gulp-based one) to Statiq. There were a number of reasons, all of them still good but mainly to support what I want to do with fedran.com and my other sites. This weekend, I changed it to a new static site generator that I have written and hosted the site also on Gemini.
<p>Last <a href="/blog/2020/11/20/website-update/">November</a>, I switched my static site generator from CobblestoneJS (my homebrew Gulp-based one) to <a href="https://statiq.dev/">Statiq</a>. There were a number of reasons, all of them still good but mainly to support what I want to do with <a href="https://fedran.com/">fedran.com</a> and my other sites.</p>
<p>This holiday weekend, I ended up ripping out all of the Statiq and wrote another static site generator, this time in C# and using an <a href="https://en.m.wikipedia.org/wiki/Entity_component_system">Entity-Component-System</a> (ECS). I also migrated my site over to <a href="gemini://d.moonfire.us/">Gemini</a> as part of the effort.</p>
<h1>Why?</h1>
<p>I liked many of the ideas that Statiq provided:</p>
<ul>
<li>Treating the website generation as code.</li>
<li>Having pipelines with dependencies for generation.</li>
<li>Immutable objects on the pipeline.</li>
</ul>
<p>However, I found myself struggling with concepts. It just didn't sit well with me and I would spend two weeks trying to implement a feature and getting stuck. When I realized I had spent a month not fixing something that was bothering me because I didn't want to delve into the code, I knew it was time to change.</p>
<p><em>That isn't to say Statiq isn't bad. It just isn't for me. That's it.</em></p>
<p>About once a year, I get 4-7 days of “alone time” to do what I want. This year, I decided to work on a new static site generator that did work the way I work today and that I hoped would carry me over for the next five years or so.</p>
<h1>Entity-Component-Systems (ECS)</h1>
<p>One thing that Statiq did (but differently) was implemented the system as an ECS. Basically, you have a lightweight object (the “entity”) and add various components into it. Those components are what provide the features: the text content, flags to say if it is HTML or Markdown, or the path.</p>
<p>While Statiq had a number of these elements built-into the <code>Document</code> call (basically their entity), there were a lot of assumptions that didn't always fit. Likewise, <code>Statiq.Web</code> had some nice opinionated ways of handling it, including a document type (binary, text, etc), but I couldn't find an easy way to extend it.</p>
<h1>Gallium</h1>
<p>With my ECS, <code>Entity</code> is only a collection plus an integer identifier. Components can be added, removed, and replaced easily using generics to determine the type. Methods are chained together but not pseudo-English fluent (which I'm also not fond of). Entities are immutable, so all the operations return a clone of that entity with the new components added, removed, or otherwise changed. (Thanks to functional programming for some of those ideas.)</p>
<pre><code class="language-csharp">Entity entity = new();
Entity entityWithTwoComponents = entity
.Set(new Uri("https://d.moonfire.us"))
.Set<FileSystemInfo>(new FileInfo("/bob.txt"));
Entity entityWithReplacedUri = entityWithTwoComponents
.Set(new Uri("http://path-to-replace-d.moonfire.us/"));
Entity entityWithOnlyUri = entityWithReplacedUri
.Remove<FileSystemInfo>();
</code></pre>
<p>I also really like chained operations, so most of the processing looks like this:</p>
<pre><code class="language-csharp">IEnumerable<Entity> input;
var onlyHtmlOutputs = input
.OrderBy(x => x.Get<FileInfo>().FullPath)
.ForComponents<Uri>((entity, uri) => this.Uri(entity, uri))
.WhereComponents<IsHtml>();
</code></pre>
<p>The whole idea is <code>ForComponents<T1, T2, T3></code> will go through the list and for all entities that have <code>T1</code>, <code>T2</code>, and <code>T3</code>, it will do the callback, otherwise it will passs it on. Likewise <code>WhereComponents<T1></code> is basically a <code>Where</code> that says only the entities with the given components.</p>
<p>Those ideas really simplified a lot of the difficulties I had with CobblestoneJS. Overall, most of the logic “felt” right for me, so I'm really happy with the results. Plus, it is based on a far more stable package ecosystem (NuGet) and in a language I enjoy greatly (C#).</p>
<p>Also, the language uses <a href="https://autofac.org/">Autofac</a> as my preferred dependency injection of choice. I really like the library plus <a href="https://nodatime.org/">NodaTime</a> when coding, so I went with these. It's a bit opinionated, but… only a few people ever used Cobblestone, so I'm going to assume very few are going to use this.</p>
<p>Once I get it cleaned up, I'll probably call the ECS “Gallium” because my original name was “Gallium Nitride” (GaN, because it's a cool name and I like what the molecule does). The static site generator would be named Nitride.</p>
<h1>Nitride</h1>
<p>Nitride is just a multi-threaded pipeline static generator. It uses pipelines much like Statiq but based on C#'s thread control (<code>ManualReset</code>, <code>ReaderWriterLockSlim</code>).</p>
<p>The rough code looks like this:</p>
<pre><code class="language-csharp">List<Entity> list = entities
.Run(new IdentifyMarkdown())
.Run(new ParseYamlHeader<PageModel>())
.ForComponents<PageModel>(this.SetHandlebarsFromPageModel)
.Run(this.setInstantFromComponent)
.Run(this.filterOutFutureInstants)
.Run(this.createCategoryIndexes)
.Run(this.createTagIndexes)
.ToList();
</code></pre>
<p>Again, DI or direct instantiate of modules, it all works the same and really ties into using Linq and C# generics. All of the pipelines are <code>async</code> but most of the operations (<code>createTagIndexes</code>, <code>ParseYamlHeader</code>) are not. But since the pipelines are, it is easy to make something <code>await</code> without changing signatures.</p>
<p>I really like the pipelines. For my site, I have the following:</p>
<ul>
<li><code>PostsPipeline</code>: Load posts and create blog archives</li>
<li><code>PagesPipeline</code>: Load static pages.</li>
<li><code>ContentPipeline</code>: Combines <code>PostsPipeline</code> and <code>PagesPipeline</code>, creates categories/tag indexes, handle filtering out future pages.</li>
<li><code>BareHtmlPipeline</code>: Takes <code>ContentPipeline</code> and converts into bare, unstyled HTML. Also creates the RSS and Atom feeds from the bare HTML.</li>
<li><code>WebpackPipeline</code>: Runs Webpack.</li>
<li><code>StyledHtmlPipeline</code>: Takes <code>BareHtmlPipeline</code> and <code>WebpackPipeline</code> and makes it styled using handlebars.</li>
<li><code>GeminiPipeline</code>: Takes <code>ContentPipeline</code> and turns the Markdown into Gemini, applies some simple styling, and generates Gemtext pages.</li>
</ul>
<p>That's it, but I'm happy with the result because I've taken lessons learned from my previous attempts to created something that will handle Fedran's massive cross-linking and project pages, MfGames's pulling in of separate Git repositories, and also some of the more complex formatting of my new sci-fi fiction website.</p>
<h1>Gemini</h1>
<p>I like the idea of <a href="//gemini.circumlunar.space/">Gemini</a>. It is a low-overhead protocol that has almost no extra features, no cookies, and basically focused on presenting content. In my case, I really want to see all of my sites on Gemini because I think it has some significant merits, more so as I want to get away from heavily styled content written by people who like tiny fonts or don't have my color contrast issues.</p>
<p>To do that, I ended up taking inspiration from <a href="https://pypi.org/project/md2gemini/">md2gemini</a> and wrote a C# library that converts Markdown into Gemtext (Gemini's markup format inspired by Markdown). The end result is <a href="gemini://d.moonfire.us/">pretty nice</a>, I think and I'm really happy with the results.</p>
<p>Of course, it meant I had to get a virtual machine to host a Gemini server next to a HTTP one, but that was going to happen sooner or later anyways.</p>
<h1>Next Steps</h1>
<p>I write a lot libraries that I think are interesting but very few people worry about. They rise up, either I stick with them or I trail off, but they always scratch my itches. On that front, the following things are left to do:</p>
<ul>
<li>Split out Gallium more formally into <code>MfGames.Gallium</code> or <code>Gallium</code>, not sure.</li>
<li>Clean up the API for <code>Nitride</code> and make sure everything is consistent. Some of the names are a bit… off, but the functionality is good.</li>
<li>Break all the modules into separate Gitlab projects and set up CI for releasing.</li>
<li>Document everything.</li>
<li>Convert five other websites sites to use it to figure out what is missing.</li>
</ul>
<p>I haven't found the money to get a developer's signing certificate, so I'll probably just put everything up on my public <a href="https://www.myget.org/">MyGet repository</a>.</p>
MfGames Writing Upkeep2021-02-05T06:00:00Zhttps://d.moonfire.us/blog/2021/02/05/mfgames-writing-upkeep/As usual between major projects, I try to spend a little time working on maintenance and upkeep of the multitude of projects that I've written or contributed too.
<p>As usual between major projects, I try to spend a little time working on maintenance and upkeep of the multitude of projects that I've written or contributed too. It is somewhat random, but I had a task to move two of my Node packages, <a href="https://www.npmjs.com/package/@mfgames-writing/ncx">@mfgames-writing/ncx</a> and <a href="https://www.npmjs.com/package/@mfgames-writing/opf">@mfgames-writing/opf</a> into the <a href="https://gitlab.com/mfgames-writing/mfgames-writing-js">mfgames-writing monorepo</a> when I had done the rest of the projects.</p>
<p>The decision to convert to a single repository for a multiple of projects took me quite a while. Mostly it came from the problem I have with package-based arrangements, the “rail car”. When I have to make a change to a library but that change has to be propagated through a number of packages until the one that needed the change. It looks like a chain and it's tedious when I'm doing rapid sets of changes trying to figure something out.</p>
<p>Now, I could just do everything in a single monolithic project, basically have a <code>mfgames-writing</code> package that does everything, but I still follow the Unix Philosophy: Do One Thing Well. The individual packages handle EPUB, PDF, and HTML generation. It feels “wrong” to force all of them to be combined together.</p>
<p>In the end, I want separate packages but I dislike the process of working with them. With Node packages, I found <a href="https://lerna.js.org/">Lerna</a> which handles a lot of that forwarding, plus plays well with <a href="https://semantic-release.gitbook.io/semantic-release/">semantic-release</a> and <a href="https://www.conventionalcommits.org/">conventional commits</a>. I've been happy, but it was a learning process and I missed two packages.</p>
<p>I also reformatting the code and switched testing over to Jest. It bumped up a few versions, but that always seem to happen when I'm learning things.</p>
<p>While that cleans up most of the packages, there are a few others related packages I still need to convert, namely the <a href="https://hub.docker.com/r/dmoonfire/mfgames-writing-js/">Docker image</a> and the <a href="https://www.npmjs.com/package/generator-mfgames-writing">Yoman generator</a>. Those are completely different than the core packages, so they shouldn't be in the monorepo in my opinion.</p>
<p>But those can wait until next around.</p>
Website Update2020-11-20T06:00:00Zhttps://d.moonfire.us/blog/2020/11/20/website-update/I've switched the generator used to create this website from Cobblestone to Statiq, a much different type of static site generator. With this change came a revamp of the site's appearance, one that is far less graphical than before, but it will allow for some future plans.
<p>I haven't been posting here lately. There are a number of reasons, but I'm going to have to go with the simple one: 2020. Lots of things all piling up, adding to to stress, and making just getting things difficult. Along the way, occasionally things slip to make sure other things continue.</p>
<p><a href="https://fedran.com/">Fedran</a> was one of those things. I've been working on maintaining my weekly chapters but it seems like I haven't been doing other than heads-down writing.</p>
<p>One of my favorite methods for managing stress is obsessing about a little change that quickly blossoms into a much larger epic worthy of sailing across the ocean and trying to debate some sirens. Usually this ends up with me redoing one of my websites, rewriting my accounting system, or completely revamping one of my writing tools.</p>
<p>If you can't get, I have a “type” when it comes to reducing stress.</p>
<p>In this case, it was stumbling onto <a href="https://gemini.circumlunar.space/">Gemini</a>. This is a stripped-down but still fairly capable protocol based on both Gopher and web browsing. It focused more on information but without stressing about the “pretty” part of the web. There are no themes, font choices, or a ton of links. Well, there can be links but they are one per line and clearly identified as such.</p>
<p>Gemini appeals to me. Almost everything I create is already in written form, this seems like an ideal place to hang up a pole and see what happens. Not to mention, it's just fun to find new places to explore.</p>
<p>This also happened at the time when I had butted up against limitations in <a href="https://www.gatsbyjs.com/">Gatsby</a>, which I had just spent three months rewriting Fedran. I had switched to Gatsby because of flaws I encountered in Cobblestone, my own static site generator.</p>
<p>That isn't to say, Gatsby doesn't make a beautiful sites. The trick they use to load the next page quickly was great. But it also is very heavy on the Javascript side of things which was pretty much the opposite of trying out Gemini which has no Javascript.</p>
<p>Since I rarely do things halfway, I want to migrate this site over to Gemini. This means either I stick with Cobblestone (which was already failing on me), convert it to Gatsby <em>and</em> to Gemini, or find something new.</p>
<p>Enter <a href="https://statiq.dev/">Statiq</a>. It's another static site generator that is one of my preferred languages (C#, Typescript, and apparently Rust) but it also allows for pipelines (what I liked about the Gulp-based Cobblestone) that would let me diverge the site between HTML and Gemini on the final steps while still allowing me the massive cross-referencing I use on Fedran.</p>
<p>I did consider switching Fedran, but after years of spinning wheels on redoing projects, I limit myself to one website rewrite a year. Otherwise, I would just endlessly redo my website and not actually create anything.</p>
<p>If you are reading this, then I've managed to switch over. Fortunately, <a href="http://d.moonfire.us/">http://d.moonfire.us/</a> is one of the more basic websites I have. That means I can build on it as I work toward the final boss, Fedran.</p>
Author Intrusion and Naming Languages2019-05-02T05:00:00Zhttps://d.moonfire.us/blog/2019/05/02/author-intrusion/This week I submitted chapters three and four of [Raging Alone](https://fedran.com/raging-alone/) to the writing group. There were some good points made but one of the ones that the entire table brought up was the names. I figured that I could use the tool I just wrote to find out how bad it was.
<p>This week I submitted chapters three and four of <a href="https://fedran.com/raging-alone/">Raging Alone</a> to the writing group. There were some good points made but one of the ones that the entire table brought up was the names.</p>
<p>Like <a href="https://fedran.com/sand-and-blood/">Sand and Blood</a>, <em>Raging Alone</em> is focused around a small, isolated clan in the desert. The language and the names were inspired by people I've worked with over the years with a healthy borrowing from French and Japanese. However, they appear not to be names that are easy for English readers to <a href="/blog/2012/10/25/introduction-to-miwafu/">understand</a>.</p>
<p>Once I figured that out, I made a point of having <em>Sand and Blood</em> start off slowly to introduce characters and conventions gradually instead of throwing a lot of names quickly at the reader. In some ways, it ended up coloring that entire piece; more than once I was told it started off too slowly which dragged out Rutejìmo's flaws.</p>
<p>With <em>Raging Alone</em>, though I got much of the same thing. However, since I just finished a round of <a href="/tags/author-intrusion/">Author Intrusion</a>, I realized I could quantify my efforts using what I just finished writing <a href="/blog/2019/04/30/author-intrusion/">wrote</a>.</p>
<h1>Metadata</h1>
<p>I keep information about the chapter in the top of each file in a YAML “front matter”.</p>
<pre><code>---
title: Sulking
characters:
primary: [Desòchu]
secondary: [Kiramíro]
referenced: [Gemènyo, Hyonèku, Kiríshi, Chyojímo, Hikòru]
---
> There is no more graceless being than a teenage boy coming into his age. --- Nagufiga Makinàfu, *Sexuality Among the Sands*
Like most typical days in Shimusogo Valley, the hot midday blew winds...
</code></pre>
<p>The key part is the <code>characters</code> section which lists the point of view (<code>primary</code>), characters in the chapter (<code>secondary</code>), and those who are talked about but not present (<code>referenced</code>).</p>
<p><em>This is also what is used to populate the footer of the chapters such as <a href="https://fedran.com/raging-alone/chapter-01/">this one</a> at the bottom.</em></p>
<h1>Author Intrusion</h1>
<p>Using the <a href="/blog/2019/04/30/author-intrusion/">last post</a> for setup, I can use <a href="http://jmespath.org/">JMESPath</a> to create a comma-separated list of characters that are in the chapter.</p>
<p>If I just wanted a list of secondary characters, I would use: <code>characters.secondary[] | sort(@) | join(', ', @)</code> which says get all the items from <code>characters.secondary</code>, then sort them, then combine them together.</p>
<p>If I want all the characters from all three categories (<code>primary</code>, <code>secondary</code>, and <code>referenced</code>), I can just replace <code>secondary</code> with <code>*</code> to get <code>characters.*[] | sort(@) | join(', ', @)</code>.</p>
<p>Related to that, I can also get the count of character in the chapter with: <code>characters.*[] | length(@)</code>.</p>
<h1>Analysis</h1>
<p>Running the tool on the first five chapters of <em>Sand and Blood</em>, I get:</p>
<pre><code>$ author-intrusion chapter list -f file.num -f title -f "characters.*[] | length(@)" -f "characters.*[] | sort(@) | join(', ', @)"
</code></pre>
<table>
<thead>
<tr>
<th style="text-align: right;">Num</th>
<th>Title</th>
<th style="text-align: right;">Count</th>
<th>Characters</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right;">1</td>
<td>Rutejìmo</td>
<td style="text-align: right;">7</td>
<td>Chimípu, Ganósho, Gemènyo, Hyonèku, Rutejìmo, Tejíko, Yutsupazéso</td>
</tr>
<tr>
<td style="text-align: right;">2</td>
<td>Confession</td>
<td style="text-align: right;">10</td>
<td>Byodenóre, Chimípu, Desòchu, Gemènyo, Hyonèku, Rador, Rutejìmo, Shimusògo, Somiryòki, Tejíko</td>
</tr>
<tr>
<td style="text-align: right;">3</td>
<td>Morning</td>
<td style="text-align: right;">17</td>
<td>Chimípu, Desòchu, Gemènyo, Hyonèku, Kizúchi, Laminar, Mapábyo, Mifuníko, Nidohána, Ojinkomàsu, Opōgyo, Panédo, Rutejìmo, Tachìra, Tachìra, Tejíko, Yutsupazéso</td>
</tr>
<tr>
<td style="text-align: right;">4</td>
<td>Rivals</td>
<td style="text-align: right;">6</td>
<td>Desòchu, Karawàbi, Palasaid, Rutejìmo, Tejíko, Tsubàyo</td>
</tr>
<tr>
<td style="text-align: right;">5</td>
<td>Decisions</td>
<td style="text-align: right;">14</td>
<td>Chimípu, Desòchu, Gemènyo, Goryápe, Karawàbi, Mapábyo, Mifuníko, Pidòhu, Rutejìmo, Shimusògo, Somiryòki, Tejíko, Tsubàyo, Yutsupazéso</td>
</tr>
</tbody>
</table>
<p>Well, seen this way, I didn't do a very good job of “gradually” introducing character names at all. One might said I did an absolutely terrible job of introducing the constructed language. I guess that points out that a subjective belief verses objective analysis.</p>
<p>Now, I don't distinguish characters who are there but not named so the list is smaller. Below is a quick test of the seven names in the first chapter. I use <code>expr - 1</code> because there is the entry in the matter which I don't want to include and readers don't see.</p>
<pre><code>sand-and-blood$ for i in Chimípu Ganósho Gemènyo Hyonèku Rutejìmo Tejíko Yutsupazéso;do echo -n "$i: "; echo $(expr $(grep $i chapter-01.markdown | wc -l) - 1); done
Chimípu: 2
Ganósho: 0
Gemènyo: 1
Hyonèku: 22
Rutejìmo: 23
Tejíko: 0
Yutsupazéso: 3
sand-and-blood$
</code></pre>
<p>Above you can see that Ganósho and Tejíko were not listed by name but the others did. Regardless, what I thought was a gradual introduction was closer to me throwing a bowlful of names at the reader and saying “here you go!”</p>
<p>Let's see if I did better with <em>Raging Alone</em>:</p>
<table>
<thead>
<tr>
<th style="text-align: right;">Num</th>
<th>Title</th>
<th style="text-align: right;">Count</th>
<th>Characters</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right;">1</td>
<td>Sulking</td>
<td style="text-align: right;">7</td>
<td>Chyojímo, Desòchu, Gemènyo, Hikòru, Hyonèku, Kiramíro, Kiríshi</td>
</tr>
<tr>
<td style="text-align: right;">2</td>
<td>Comfort</td>
<td style="text-align: right;">5</td>
<td>Chyojímo, Desòchu, Hikòru, Hyonèku, Kiramíro</td>
</tr>
<tr>
<td style="text-align: right;">3</td>
<td>The Call</td>
<td style="text-align: right;">8</td>
<td>Chyojímo, Desòchu, Hikòru, Kiramíro, Pikonàga, Somiryòki, Tejíko, Yutsupazéso</td>
</tr>
<tr>
<td style="text-align: right;">4</td>
<td>Baby Brother</td>
<td style="text-align: right;">4</td>
<td>Chyojímo, Desòchu, Rutejìmo, Tachìra</td>
</tr>
<tr>
<td style="text-align: right;">5</td>
<td>Waking Up</td>
<td style="text-align: right;">5</td>
<td>Desòchu, Kiríshi, Rutejìmo, Somiryòki, Tejíko</td>
</tr>
</tbody>
</table>
<p>Well, I've done a lot better job of reducing how many names are thrown at the reader but it also points out I'm still throwing a lot of them in the first chapter. Getting that number down to four or five would make it an easier introduction and points out a possible place to trim down the story. In the writing group, we found a way of cutting out two or three names out of “The Call” which would make it much more manageable.</p>
<p>Now, chapter five is the “inciting event”, so where I would normally split a chapter, I would probably want to add it after chapter five so we get to the “big event” faster. But that's for another blog post.</p>
<h1>Conclusion</h1>
<p>Sometimes, its hard as an author to see problems in the story while writing it. One of the reasons I created Author Intrusion was to help me focus them the problem I struggled to see. In this case, I thought I was being gradual with names but once it was pointed out, it was obvious I was not.</p>
<p>I also realized there are some additional checkers I could include to help identify these problems while writing:</p>
<ul>
<li><a href="https://gitlab.com/author-intrusion/author-intrusion-rust/issues/17">#17</a>: Introduce a checker that counts a number of entries (a JMESPath such as one against characters) against a threshold</li>
<li><a href="https://gitlab.com/author-intrusion/author-intrusion-rust/issues/18">#18</a>: Ensure that every chapter has an epigraph (my style guide)</li>
<li><a href="https://gitlab.com/author-intrusion/author-intrusion-rust/issues/18">#19</a>: Check that a character in narrative is identified in the metadata.</li>
</ul>
Integrating Semantic Versioning into MfGames Writing2018-08-27T05:00:00Zhttps://d.moonfire.us/blog/2018/08/27/mfgames-writing-releases/How to release a novel or story using MfGames Writing.
<p>The final component of this <a href="/tags/mfgames-writing/">MfGames Writing</a> series is how to integrate <a href="/blog/2018/08/13/publishing-processes/">semantic releases</a> into the publication process.</p>
<p>If you want to use <a href="http://sentimentalversioning.org/">sentimental versioning</a>, then you can probably skip this. I like having automatic versioning because it helps me identify the version that beta readers or an editor has when I go to integrate the changes.</p>
<h1>Series</h1>
<p>I appear to be writing a short series of post about the tools I use for publication and writing.</p>
<ol>
<li><a href="/blog/2018/08/13/publishing-processes/">Semantic Versions and Releases</a>: Why semantic versioning helps with the writing process.</li>
<li><a href="/blog/2018/08/21/mfgames-writing-reasons/">Evolution of MfGames Writing</a>: A brief history and reasoning behind the tools.</li>
<li><a href="/blog/2018/08/22/mfgames-writing-init/">First Steps Using MfGames Writing</a>: Starting a new project with MfGames Writing.</li>
<li><a href="/blog/2018/08/23/mfgames-writing-content/">Adding Content to MfGames Writing</a>: Adding front and back matter to novels.</li>
<li><a href="/blog/2018/08/24/mfgames-writing-docker-and-ci/">Working with MfGames Writing, CI, and Docker</a>: Adding automatic building with commits.</li>
<li><a href="/blog/2018/08/25/mfgames-writing-formats/">Additional Formats for MfGames Writing</a>: How to create PDF, MOBI, DOCX, and HTML versions.</li>
<li><a href="/blog/2018/08/26/mfgames-writing-themes/">Theming for MfGames Writing</a>: A light introduction on how to customize the output.</li>
<li>Integrating Semantic Versioning into MfGames Writing: Tying semantic releases into the process.</li>
</ol>
<h1>NPM Packages</h1>
<p>Like everything else, we pull in a number of packages from NPM to handle the release process.</p>
<pre><code class="language-shell">$ npm install
npm install \
@commitlint/cli \
@commitlint/config-conventional \
@semantic-release/changelog \
@semantic-release/git \
commitizen \
cz-conventional-changelog \
husky \
semantic-release
</code></pre>
<p>This is a pretty big and scary list, but there isn't a lot of clean ways to do this. I'll give a brief summary of them.</p>
<p>The <code>@commitlint</code> is to make sure we have a consistent commit messages (the <code>feat:</code> and <code>fix:</code> stuff). This makes sure everything else works smoothly.</p>
<p>The <code>@semantic-release</code> packages are to do the release process.</p>
<p>The two packages, <code>commitizen</code> and <code>cz-conventional-changelog</code>, are used to help guide you through creating the commits if you are unfamiliar with it. When these are installed, you can use <code>git cz</code> instead of <code>git commit</code>.</p>
<p>Finally, <code>husky</code> is used to make sure you follow the commits correctly because it will reject the commit if you don't follow conventions.</p>
<h1>Configuration</h1>
<p>The bulk of the configuration happens inside <code>package.json</code>. This file can get pretty big, so I'm only going to list the differences.</p>
<pre><code class="language-json">{
"scripts": {
"commitmsg": "commitlint -E GIT_PARAMS"
},
"release": {
"branch": "master",
"message": "chore(release): v${nextRelease.version}\n\n${nextRelease.notes}",
"verifyConditions": [
"@semantic-release/changelog",
"@semantic-release/git"
],
"analyzeCommits": ["@semantic-release/commit-analyzer"],
"prepare": [
"@semantic-release/changelog",
"@semantic-release/npm",
"@semantic-release/git"
],
"publish": [],
"success": [],
"fail": []
},
"commitlint": {
"extends": ["@commitlint/config-conventional"]
}
}
</code></pre>
<h1>Updating .gitlab-ci.yml</h1>
<p>To actually use this, we have to modify the GitLab setup slightly.</p>
<pre><code class="language-yaml">image: dmoonfire/mfgames-writing-js:1.1.1
stages:
- publish
publish:
stage: publish
tags:
- docker
script:
- npm ci
- npx commitlint --from=master to=CI_BUILD_REF_NAME
- npx semantic-release
- npm run build
artifacts:
expire_in: 1 week
paths:
- "*.pdf"
- "*.epub"
- "*.mobi"
- "*.docx"
- "*.html"
</code></pre>
<p>The two new lines are what does the release process.</p>
<pre><code class="language-yaml">- npx commitlint --from=master to=CI_BUILD_REF_NAME
- npx semantic-release
</code></pre>
<h1>What Does This Give You</h1>
<p>So, given the amount of setup, what does this give you? Every time you push up a <code>feat:</code> or <code>fix:</code>, it will do the following:</p>
<ul>
<li>Bump the version to the next appropriate one (1.0.0 to 1.1.0 for example).</li>
<li>Generate the EPUB, MOBI, etc file.</li>
<li>Create or update the <code>CHANGELOG.md</code> file with the summary of your changes.</li>
<li>Tag the version in Git with <code>v1.1.0</code> (for example).</li>
</ul>
<p>That means, every change you do will have a distinct version. With everything else tied together, you could put it in a header and use that to figure out where to make the changes (my writing group gives me 4-12 sets of corrections, frequently overlapping).</p>
<p>This isn't for everyone but I have found it very helpful when working with others.</p>
Theming for MfGames Writing2018-08-26T05:00:00Zhttps://d.moonfire.us/blog/2018/08/26/mfgames-writing-themes/Branding requires a distinct style for writing. Here is how you customize a theme for MfGames Writing to match.
<p>One of the big parts about <a href="/tags/mfgames-writing/">MfGames Writing</a> is the ability to customize the appearance of the generated files. If we didn't have that, then every book would look the same and I don't care for that. Books should allow for different themes, chapter numbers, and even spacing. To that regard, everything is funneled through a theme for formatting.</p>
<h1>Series</h1>
<p>I appear to be writing a short series of post about the tools I use for publication and writing.</p>
<ol>
<li><a href="/blog/2018/08/13/publishing-processes/">Semantic Versions and Releases</a>: Why semantic versioning helps with the writing process.</li>
<li><a href="/blog/2018/08/21/mfgames-writing-reasons/">Evolution of MfGames Writing</a>: A brief history and reasoning behind the tools.</li>
<li><a href="/blog/2018/08/22/mfgames-writing-init/">First Steps Using MfGames Writing</a>: Starting a new project with MfGames Writing.</li>
<li><a href="/blog/2018/08/23/mfgames-writing-content/">Adding Content to MfGames Writing</a>: Adding front and back matter to novels.</li>
<li><a href="/blog/2018/08/24/mfgames-writing-docker-and-ci/">Working with MfGames Writing, CI, and Docker</a>: Adding automatic building with commits.</li>
<li><a href="/blog/2018/08/25/mfgames-writing-formats/">Additional Formats for MfGames Writing</a>: How to create PDF, MOBI, DOCX, and HTML versions.</li>
<li>Theming for MfGames Writing: A light introduction on how to customize the output.</li>
<li><a href="/blog/2018/08/27/mfgames-writing-releases/">Integrating Semantic Versioning into MfGames Writing</a>: Tying semantic releases into the process.</li>
</ol>
<h1>HTML</h1>
<p>The entire theme system is built on HTML and CSS. In previous versions of this, I generated XeLaTeX files for PDF and used a custom conversion utility for Word documents. It also increased the complexity significantly. Switching to a HTML-based system simplified the theme creation and made debugging easier.</p>
<p>This is one case where looking at a <a href="https://gitlab.com/mfgames-writing/mfgames-writing-clean-js">reference implementation</a> will be helpful.</p>
<h1>Components</h1>
<p>In the <a href="/blog/2018/08/23/mfgames-writing-content/">content</a> post, there are quite a few references to <code>element</code> on each of the content items. These elements are Liquid-based stylesheets that are provided by the theme.</p>
<p>For example, the <code>title.xhtml</code> in the clean theme looks like this:</p>
<pre><code><div class="element-page-break element-page-break-title">&#160;</div>
{{html}}
</code></pre>
<p>The <code>{{html}}</code> is where the contents in the <code>source</code> given on the content will be placed. In this case, the <code>element-page-break</code> is how I handle page rendering with WeasyPrint.</p>
<p>A more complicated example would be <a href="https://gitlab.com/mfgames-writing/mfgames-writing-clean-js/blob/master/templates/chapter.xhtml">chapter.xhtml</a>:</p>
<pre><code><div class="element-wrapper {{content.element}}">
<div class="element-page-break element-page-break-right">&#160;</div>
<div class="element-page-pad-right">
{% if content.number %}
<div class="element-number">Chapter {{content.number}}</div>
{% endif %}
<h1 class="element-title">{{content.title}}</h1>
{{html}}
</div>
</div>
</code></pre>
<p>You can see more uses of the Liquid tags. In this case, if we have a <code>number</code> property in the content, it will insert the <code>Chapter #</code> above the title of the chapter (from the YAML header) and the contents of the body into <code>{{html}}</code>.</p>
<h2>Recursive Templates</h2>
<p>To reduce copy/paste, there are also recursive templates. For example, the <a href="https://gitlab.com/mfgames-writing/mfgames-writing-clean-js/blob/master/templates/colophon.xhtml">colophon</a> template looks like this:</p>
<pre><code>---
extends: simple-title
---
{{html}}
</code></pre>
<p>Once this page is rendered, the results are formatted by the <a href="https://gitlab.com/mfgames-writing/mfgames-writing-clean-js/blob/master/templates/simple-title.xhtml">simple-title</a></p>
<pre><code><div class="element-wrapper {{content.element}}">
<div class="element-page-break element-page-break-right">&#160;</div>
<h1 class="initial">{{content.title}}</h1>
{{html}}
</div>
</code></pre>
<p>It is recursive, so you can have one template extending another extending another.</p>
<h1>Styling</h1>
<p>You'll notice that there is very little styling in the HTML. Instead, most of it is done with a SASS template in the theme. The base style is called <a href="https://gitlab.com/mfgames-writing/mfgames-writing-clean-js/blob/master/styles/stylesheet.scss">stylesheet.scss</a>:</p>
<pre><code class="language-scss">p {
margin-bottom: 0;
margin-top: 0;
text-align: left;
text-indent: 2em;
}
</code></pre>
<p>Stylesheets can be overridden by specific formats. The most common is having a <a href="https://gitlab.com/mfgames-writing/mfgames-writing-clean-js/blob/master/styles/weasyprint-base.scss">WeasyPrint-specific stylesheet</a>:</p>
<pre><code class="language-scss">@import "stylesheet";
@page {
@bottom-center {
content: counter(page);
vertical-align: top;
padding-top: 1em;
}
size: letter portrait;
margin: 3cm 2cm;
}
</code></pre>
<p>To figure out overrides, the system looks for the format name first (<code>html.scss</code> or <code>weasyprint.scss</code>) and if it doesn't find one, goes to the base (<code>stylesheet.scss</code>).</p>
<h2>Stylesheet Variables</h2>
<p>Because I'm fond of single points of truth, SASS also lets us have variables so we can pull the version, or information from the edition data.</p>
<pre><code class="language-scss">@page :right {
@top-right {
content: themeString("edition.title");
vertical-align: bottom;
padding-bottom: 1em;
}
padding-left: 1cm;
}
</code></pre>
<p>The <code>themeString</code> works just like the Liquid templates.</p>
<h1>Creating New Themes</h1>
<p>To create a new theme, I basically just copy all of the clean theme and start customizing.</p>
Additional Formats for MfGames Writing2018-08-25T05:00:00Zhttps://d.moonfire.us/blog/2018/08/25/mfgames-writing-formats/There are a number of different formats that can be used to generate MfGames Writing output, these are the basics.
<p>Not everyone in the world uses EPUB. Yes, I consider it one of the best formats for me because I can change font size or layout, there is still a big need for print books and Amazon is the source of most ebook sales (though I don't care for that either).</p>
<p>To handle that, <a href="/tags/mfgames-writing/">MfGames Writing</a> provides a pluggable collection of different output formats. This allows you to use only the formats that you way (EPUB2, PDF, HTML) and they can be updated separately.</p>
<h1>Series</h1>
<p>I appear to be writing a short series of post about the tools I use for publication and writing.</p>
<ol>
<li><a href="/blog/2018/08/13/publishing-processes/">Semantic Versions and Releases</a>: Why semantic versioning helps with the writing process.</li>
<li><a href="/blog/2018/08/21/mfgames-writing-reasons/">Evolution of MfGames Writing</a>: A brief history and reasoning behind the tools.</li>
<li><a href="/blog/2018/08/22/mfgames-writing-init/">First Steps Using MfGames Writing</a>: Starting a new project with MfGames Writing.</li>
<li><a href="/blog/2018/08/23/mfgames-writing-content/">Adding Content to MfGames Writing</a>: Adding front and back matter to novels.</li>
<li><a href="/blog/2018/08/24/mfgames-writing-docker-and-ci/">Working with MfGames Writing, CI, and Docker</a>: Adding automatic building with commits.</li>
<li>Additional Formats for MfGames Writing: How to create PDF, MOBI, DOCX, and HTML versions.</li>
<li><a href="/blog/2018/08/26/mfgames-writing-themes/">Theming for MfGames Writing</a>: A light introduction on how to customize the output.</li>
<li><a href="/blog/2018/08/27/mfgames-writing-releases/">Integrating Semantic Versioning into MfGames Writing</a>: Tying semantic releases into the process.</li>
</ol>
<h1>Using Formats</h1>
<p>Formats are consumed in the edition section of the <code>publication.yaml</code> file.</p>
<pre><code class="language-yaml">metadata:
title: Test Project
author: D. Moonfire
language: en
theme: "@mfgames-writing/clean"
outputDirectory: .
outputFilename: test-project-{{edition.version}}.{{edition.editionName}}
editions:
epub:
format: "@mfgames-writing/epub2"
pdf:
format: "@mfgames-writing/weasyprint"
isbn: 978-1-999999
html:
format: "@mfgames-writing/html"
images:
scale: 0.25
</code></pre>
<p>The format is the name of the package installed with <code>npm install</code>.</p>
<h1>EPUB</h1>
<p>The easiest format to use is EPUB2.</p>
<pre><code class="language-shell">$ npm install @mfgames-writing/epub2
$ npm run build:html
</code></pre>
<p>The setup for the NPM commands can be found <a href="/blog/2018/08/24/mfgames-writing-docker-and-ci/">yesterday</a>.</p>
<p>When this runs, it will generate an EPUB2 file that will be verifiable with <code>epubcheck</code>.</p>
<h1>HTML</h1>
<p>HTML is also pretty easy.</p>
<pre><code class="language-shell">$ npm install @mfgames-writing/html
</code></pre>
<h1>PDF</h1>
<p>PDF is the complicated one and why I use a Docker image. I decided to use <a href="https://weasyprint.org/">WeasyPrint</a> for PDF generation because it creates good-quality PDF files and works with HTML, the format <a href="/blog/2018/08/26/mfgames-writing-themes/">themes</a> use.</p>
<pre><code class="language-shell">$ npm install @mfgames-writing/weasyprint
</code></pre>
<p>This assumes that <code>weasyprint</code> and <code>pdftk</code> are both in the PATH.</p>
<h1>MOBI</h1>
<p>There isn't a good package for generating MOBI files directly, so the best approach to creating a Amazon MOBI file for uploading is to create a EPUB file and then use <code>kindlegen</code> to convert it.</p>
<pre><code class="language-shell">$ npm run build:epub
$ kindlegen *.epub
</code></pre>
<h1>DOCX</h1>
<p>Like MOBI, it is hard to write a DOCX in a clean manner. To do that, we create a HTML and then convert it.</p>
<pre><code class="language-shell">$ npm run build:html
$ sed 's@&#173;@@g' < *.html \
| pandoc -f html -t docx -o test-package-0.0.0.docx
</code></pre>
<p>We have the extra <code>sed</code> command to remove the hypenation from <code>@mfgames-writing/hyphen</code> which makes a cleaner file.</p>
Working with MfGames Writing, CI, and Docker2018-08-24T05:00:00Zhttps://d.moonfire.us/blog/2018/08/24/mfgames-writing-docker-and-ci/How to use a continual integration (CI) server to publish projects.
<p>One of the main points of having <a href="/tags/mfgames-writing/">MfGames Writing</a> is to automate the publication process.</p>
<h1>Series</h1>
<p>I appear to be writing a short series of post about the tools I use for publication and writing.</p>
<ol>
<li><a href="/blog/2018/08/13/publishing-processes/">Semantic Versions and Releases</a>: Why semantic versioning helps with the writing process.</li>
<li><a href="/blog/2018/08/21/mfgames-writing-reasons/">Evolution of MfGames Writing</a>: A brief history and reasoning behind the tools.</li>
<li><a href="/blog/2018/08/22/mfgames-writing-init/">First Steps Using MfGames Writing</a>: Starting a new project with MfGames Writing.</li>
<li><a href="/blog/2018/08/23/mfgames-writing-content/">Adding Content to MfGames Writing</a>: Adding front and back matter to novels.</li>
<li>Working with MfGames Writing, CI, and Docker: Adding automatic building with commits.</li>
<li><a href="/blog/2018/08/25/mfgames-writing-formats/">Additional Formats for MfGames Writing</a>: How to create PDF, MOBI, DOCX, and HTML versions.</li>
<li><a href="/blog/2018/08/26/mfgames-writing-themes/">Theming for MfGames Writing</a>: A light introduction on how to customize the output.</li>
<li><a href="/blog/2018/08/27/mfgames-writing-releases/">Integrating Semantic Versioning into MfGames Writing</a>: Tying semantic releases into the process.</li>
</ol>
<h1>Adding Scripts</h1>
<p>We've installed the commands in a previous step, but hooking them up in the <code>package.json</code> makes life a lot easier.</p>
<pre><code class="language-json">{
"name": "test-project",
"private": true,
"version": "0.0.0"
"scripts": {
"build:epub": "mfgames-writing-format build epub",
"build:pdf": "mfgames-writing-format build pdf",
"build:html": "mfgames-writing-format build html",
"build:docx": "sed 's@&#173;@@g' < $npm_package_name-$npm_package_version.html | pandoc -f html -t docx -o $npm_package_name-$npm_package_version.docx",
"build:mobi": "kindlegen $npm_package_name-$npm_package_version.epub",
"build": "npm run build:epub && npm run build:mobi && npm run build:pdf && npm run build:html && npm run build:docx"
}
}
</code></pre>
<p>Basically this sets up my basic suite of scripts that let me generate everything or one specific file.</p>
<pre><code class="language-shell">$ npm run build:epub
$ npm run build
</code></pre>
<p>You can see we use a couple other programs like <code>pandoc</code> to make DOCX files from HTML or MOBI files using the EPUB file and <code>kindlegen</code>.</p>
<p>I use <code>$npm_package_name</code> to pull in the package name (as I said, I use them for consistency) which lets me produce files with a consistent pattern including the version number.</p>
<p>The other formats (HTML, PDF, etc) will be talked about in <a href="/blog/2018/08/25/mfgames-writing-formats/">tomorrow's post</a>.</p>
<h1>CI Configuration</h1>
<p>I use <a href="https://gitlab.com/">GitLab</a> as my primary development platform. I like the company, the fact they open-source much of their code, and their features. Most important is that they provide private repos, which means I can have one repo for every novel I work on.</p>
<p>GitLab has a continual integration (CI) service which will automatically run whenever I push up code. This is controlled by the <code>.gitlab-ci.yml</code> file.</p>
<pre><code class="language-yaml"># This Docker image contains all the libraries and tools to generate files.
image: dmoonfire/mfgames-writing-js:1.1.1
stages:
- publish
publish:
stage: publish
tags:
- docker
# Generate the various publication files.
script:
- npm ci
- npm run build
# Keeping artifacts means we can download them after the fact.
artifacts:
expire_in: 1 week
paths:
- "*.pdf"
- "*.epub"
- "*.mobi"
- "*.docx"
- "*.html"
</code></pre>
<p>If CI is turned on in GitLab and this file is present, then it will automatically run the commands in the <code>script</code> section. It then takes the resulting PDF, EPUB, MOBI, DOCX, and HTML files and zips them up to be found on the website. Basically this runs the same command that you can run manually which makes it easier to test.</p>
<p><a href="/blog/2018/08/mfgames-writing-download.png"><img class="img-responsive post-img-link" src="/blog/2018/08/mfgames-writing-download.png" alt="Download Button" /></a></p>
<h1>Docker</h1>
<p>In the <code>.gitlab-ci.yml</code> file, you may notice we use a Docker image <code>dmoonfire/mfgames-writing-js:1.1.1</code>. This is up on <a href="https://hub.docker.com/r/dmoonfire/mfgames-writing-js/">DockerHub</a> and includes the various programs and utilities needed to generate the files. Some of the formats, in specific <a href="https://weasyprint.org/">WeasyPrint</a>, have specific installations and I found that having a Docker image makes life a lot easier.</p>
<p>The image contains:</p>
<ul>
<li>NPM and required packages</li>
<li>WeasyPrint</li>
<li>KindleGen</li>
<li>Pandoc</li>
<li>PdfTk (pdfcat)</li>
</ul>
Adding Content to MfGames Writing2018-08-23T05:00:00Zhttps://d.moonfire.us/blog/2018/08/23/mfgames-writing-content/Introduction on how to add different types of content to a MfGames Writing project.
<p>In my continuing introduction to <a href="/tags/mfgames-writing/">MfGames Writing</a>, I've only given a very brief introduction to adding content to the publication. Like everything else, there is a lot more complexity in the process to support various components of a novel: bastard and full titles, table of contents, appendixes, and even colophons.</p>
<h1>Series</h1>
<p>I appear to be writing a short series of post about the tools I use for publication and writing.</p>
<ol>
<li><a href="/blog/2018/08/13/publishing-processes/">Semantic Versions and Releases</a>: Why semantic versioning helps with the writing process.</li>
<li><a href="/blog/2018/08/21/mfgames-writing-reasons/">Evolution of MfGames Writing</a>: A brief history and reasoning behind the tools.</li>
<li><a href="/blog/2018/08/22/mfgames-writing-init/">First Steps Using MfGames Writing</a>: Starting a new project with MfGames Writing.</li>
<li>Adding Content to MfGames Writing: Adding front and back matter to novels.</li>
<li><a href="/blog/2018/08/24/mfgames-writing-docker-and-ci/">Working with MfGames Writing, CI, and Docker</a>: Adding automatic building with commits.</li>
<li><a href="/blog/2018/08/25/mfgames-writing-formats/">Additional Formats for MfGames Writing</a>: How to create PDF, MOBI, DOCX, and HTML versions.</li>
<li><a href="/blog/2018/08/26/mfgames-writing-themes/">Theming for MfGames Writing</a>: A light introduction on how to customize the output.</li>
<li><a href="/blog/2018/08/27/mfgames-writing-releases/">Integrating Semantic Versioning into MfGames Writing</a>: Tying semantic releases into the process.</li>
</ol>
<h1>More Than Chapters</h1>
<p>In the <a href="/blog/2018/08/22/mfgames-writing-init/">previous</a> blog post, I gave a short example of <code>publication.yaml</code> that included all the chapters in a folder.</p>
<pre><code class="language-yaml">contents:
- element: chapter
number: 1
directory: chapters
source: /^chapter-\d+.md$/
start: true
page: 1
</code></pre>
<p>This would include any number of chapters, from <code>01</code> to <code>99</code>.</p>
<h1>Covers</h1>
<p>Most EPUB files need a good graphical image for readers. Creating MOBI (via <code>kindlegen</code>) from the EPUB also requires a graphical image. Adding one is pretty easy.</p>
<pre><code class="language-yaml">- element: cover
source: build/repos/fedran-covers/dist/full/0100-00-sand-and-blood-front.jpg
linear: false
exclude:
editions: [pdf]
toc: true
</code></pre>
<p>The <code>element</code> is the name of the component in the <a href="/blog/2018/08/26/mfgames-writing-themes/">theme</a>. Basically that determines how it is rendered when we generate the files.</p>
<p>The <code>source</code> is the relative path to the image. It should be a JPEG image. In my example, I have it pulling from another Git repository but it can be a file checked into the Git. In general, it needs to be a file that will be accessed while building.</p>
<p>We use <code>linear: false</code> because covers are one of those things that reader software will occasionally rearrange. In other words, I don't know, it seems to be required.</p>
<p>We also use <code>exclude:</code> to exclude it from the table of contents (<code>toc</code>) and from the <a href="/blog/2018/08/25/mfgames-writing-formats/">PDF version</a>.</p>
<h1>Titles</h1>
<pre><code class="language-yaml">- element: bastard
source: frontmatter/bastard.html
linear: false
exclude:
editions: [epub]
toc: true
- element: title
source: frontmatter/title.html
linear: false
exclude:
toc: true
</code></pre>
<p>In this case, we are using the <code>bastard</code> and <code>title</code> templates (nearly identical) from the theme. I put my front matter files in a directory creatively called <code>frontmatter</code>. You can probably guess where the back matter goes. I could use a number in the front (like chapters) and a pattern but I like to be explicit with my matters.</p>
<p>The <code>exclude.toc</code> is because title pages don't show up in the table of contents.</p>
<h1>Legal Page</h1>
<pre><code class="language-yaml">- element: legal
source: frontmatter/legal.markdown
liquid: true
linear: false
</code></pre>
<p>The legal page introduces a new concept. The <code>liquid</code> flag lets me use Liquid templates on the page. The variables available for the tags are combined from three places: the <code>package.json</code> file, the <code>metadata</code> section in <code>publication.yaml</code>, and then the edition being generated from the same file. This means you can store variables like the version from <code>package.json</code> but an ISBN from the edition.</p>
<p>Since we are using Liquid, we can also have some simple logic such as the optional ISBN which only shows up in the <code>pdf</code> edition. This would allow us to have two different versions of the book (say hardcover and paperback) with different formats and edition names.</p>
<pre><code class="language-markdown">---
title: Legal
---
Broken Typewriter Press\
[https://broken.typewriter.press/](https://broken.typewriter.press/)
{% if edition.isbn %}
ISBN {{ edition.isbn }}
{% endif %}
Version {{ edition.version }}
</code></pre>
<p>Using versions like this could also be used to create site-specific buy links for the different editions (such as a Amazon or Smashwords-specific edition of the same file).</p>
<h1>Table of Contents</h1>
<p>The table of contents (<code>element: toc</code>) has a special code to generate one if a source isn't given. However, the generated code doesn't do page numbers which is why I currently have it turned off in PDF. For EPUB and MOBI, it will generated a properly linked table for each chapter.</p>
<pre><code class="language-yaml">- element: toc
linear: false
title: Contents
exclude:
editions: [pdf]
</code></pre>
<h1>Pipelines</h1>
<p>The last interesting component to point out is the pipelines. A pipeline is something that modifies the original Markdown before it is converted into HTML (I've needed a post conversion one also but I haven't coded it). These are used to handle word hyphenation (we don't do it by default), either for generic languages or for specific languages (such as the Fedran one to handle <a href="https://fedran.com/miwafu/">Miwāfu</a> names). There is also <a href="https://www.npmjs.com/package/@mfgames-writing/guillemet">one</a> to convert <code><<</code> and <code>>></code> into <code>«</code> and <code>»</code> (I use this for telepathy in my books).</p>
<pre><code class="language-yaml">- element: chapter
number: 1
directory: chapters
source: /^chapter-\d+.markdown$/
start: true
page: 1
pipeline: &pipelines
- module: "@fedran/writing-hyphen"
- module: mfgames-writing-hyphen
</code></pre>
<h1>Sand and Blood example</h1>
<p>Using <a href="https://gitlab.com/fedran/sand-and-blood/blob/master/publication.yaml">Sand and Blood</a> as an example:</p>
<pre><code class="language-yaml">contents:
- element: cover
source: build/repos/fedran-covers/dist/full/0100-00-sand-and-blood-front.jpg
linear: false
exclude:
editions: [pdf]
toc: true
- element: bastard
source: frontmatter/bastard.html
linear: false
exclude:
editions: [epub]
toc: true
- element: title
source: frontmatter/title.html
linear: false
exclude:
toc: true
- element: legal
source: frontmatter/legal.markdown
liquid: true
linear: false
- element: dedication
source: frontmatter/dedication.markdown
linear: false
- element: toc
linear: false
title: Contents
exclude:
editions: [pdf]
- element: preface
source: frontmatter/miwafu.html
linear: false
- element: chapter
number: 1
directory: chapters
source: /^chapter-\d+.markdown$/
start: true
page: 1
pipeline: &pipelines
- module: "@fedran/writing-hyphen"
- module: "@mfgames-writing/hyphen"
- element: appendix
source: backmatter/about.markdown
- element: appendix
source: backmatter/fedran.markdown
- element: appendix
id: license
source: backmatter/license.markdown
- element: appendix
source: backmatter/patrons.markdown
- element: appendix
source: backmatter/credits.markdown
- element: colophon
source: backmatter/colophon.markdown
</code></pre>
<h1>EPUBCHECK</h1>
<p>Once you get the basics in (cover, table of contents, and chapters), the resulting EPUB file will easily get through <code>epubcheck</code> which makes it easier to then convert that into a MOBI file using <code>kindlegen</code>.</p>
First Steps Using MfGames Writing (GitLab, NPM)2018-08-22T05:00:00Zhttps://d.moonfire.us/blog/2018/08/22/mfgames-writing-init/The initial steps for setting up a new project for publication.
<p>Getting started with <a href="/tags/mfgames-writing/">MfGames Writing</a> can be a bit overwhelming. This is the beginning of how to get started. These directions are mainly focused toward technical users who already have a basic skill in Git.</p>
<h1>Series</h1>
<p>I appear to be writing a short series of post about the tools I use for publication and writing.</p>
<ol>
<li><a href="/blog/2018/08/13/publishing-processes/">Semantic Versions and Releases</a>: Why semantic versioning helps with the writing process.</li>
<li><a href="/blog/2018/08/21/mfgames-writing-reasons/">Evolution of MfGames Writing</a>: A brief history and reasoning behind the tools.</li>
<li>First Steps Using MfGames Writing: How to get started with a project.</li>
<li><a href="/blog/2018/08/23/mfgames-writing-content/">Adding Content to MfGames Writing</a>: Adding front and back matter to novels.</li>
<li><a href="/blog/2018/08/24/mfgames-writing-docker-and-ci/">Working with MfGames Writing, CI, and Docker</a>: Adding automatic building with commits.</li>
<li><a href="/blog/2018/08/25/mfgames-writing-formats/">Additional Formats for MfGames Writing</a>: How to create PDF, MOBI, DOCX, and HTML versions.</li>
<li><a href="/blog/2018/08/26/mfgames-writing-themes/">Theming for MfGames Writing</a>: A light introduction on how to customize the output.</li>
<li><a href="/blog/2018/08/27/mfgames-writing-releases/">Integrating Semantic Versioning into MfGames Writing</a>: Tying semantic releases into the process.</li>
</ol>
<h1>Setting up Git</h1>
<p>I always start by creating a new Git project on <a href="https://gitlab.com/">GitLab</a>. You can do the same with GitHub, BitBucket, or just locally. Since I'm focusing on the tools, I'm going to assume you know how to create those projects.</p>
<pre><code class="language-shell">$ git checkout https://oauth2:SecretPassword@gitlab.com/dmoonfire/test-project.git
... stuff happens
$ cd test-project
</code></pre>
<p>You could create a new folder, it doens't really matter.</p>
<pre><code class="language-shell">$ mkdir test-project
$ cd test-project
$ git init
</code></pre>
<h1>Setting up .gitignore</h1>
<p>The way builds work, there are quite a few files that will be generated that wouldn't be checked into the repository. This is one thing that makes NPM so powerful. To avoid that, we create a <code>.gitignore</code> file that ignores those files.</p>
<pre><code># Emacs and VI creates these files and I'm too lazy to turn it off.
*~
\#*
.#*
# This is the output of the various build processes.
*.pdf
*.mobi
*.epub
*.docx
# Sometimes I use `build` for my output but most of the time I just put it in the root.
build/
# This is where Node/NPM puts its file.
node_modules/
</code></pre>
<h1>Setting up .editorconfig</h1>
<p><a href="https://editorconfig.org/">EditorConfig</a> is a tool for providing some consistency in formatting. For novels, it doesn't make sense but I like to have it there to make sure the supporting files are consistent. This is an optional file, I just like having it.</p>
<pre><code># EditorConfig is awesome: http://EditorConfig.org
# top-most EditorConfig file
root = true
[*]
charset = utf-8
end_of_line = lf
indent_brace_style = K&R
indent_size = 4
indent_style = space
insert_final_newline = true
max_line_length = 80
tab_width = 4
trim_trailing_whitespace = true
curly_bracket_next_line = false
[*.{js,ts}]
quote_type = double
[*.yaml]
indent_size = 2
tab_width = 2
[package.json]
indent_size = 2
tab_width = 2
</code></pre>
<h1>Setting up package.json</h1>
<p>The versioning and management comes from <code>package.json</code>. This is the file that says <em>which</em> version of which utilities to use. Now, you can use <code>npm init</code> to figure it out but this is the basic file.</p>
<pre><code class="language-json">{
"name": "test-project",
"private": true,
"version": "0.0.0"
}
</code></pre>
<p>There is other stuff, but these are the key parts. The name is the <em>slug</em> of the file. For me, it is always the name of the folder in the website and the Git repository. I also use this to build my website, so it is a pretty consistent format. You can call it whatever you want and change it as you need.</p>
<p>The <code>private</code> is to prevent you from accidently publishing your novel on <a href="https://www.npmjs.com/">NPM</a>.</p>
<p>Finally the <code>version</code> is used for the <a href="/blog/2018/08/13/publishing-processes/">semantic release</a> and populates data on the legal page of the resulting book. In general, I start with “0.0.0” and then bump it to “1.0.0” when I finalize the book and publish it for the first time.</p>
<h1>Write a Chapter</h1>
<p>Now, we need somethig to put into the story or novel. Since I've started this, I organize even my short story into chapters. To do that, you can put it anywhere, but I always put my chapters in (unoriginally) the <code>chapters/</code> folder. I also use two-digit numbers with zero padding so it sorts correctly (this is useful). You can see this at <a href="https://gitlab.com/fedran/sand-and-blood/">Sand and Blood's Git Repository</a>.</p>
<p>For example, <code>chapters/chapter-01.md</code>:</p>
<pre><code class="language-markdown">---
title: Mistakes Were Made
---
It was a bright and terribly sunny day.
I didn't like it.
</code></pre>
<h1>Setting up publication.yaml</h1>
<p>Because of the flexibility (if you don't want to use <code>chapters/</code>, want to spread it out across deeper folders), nothing is automatic when it comes to formatting the books. To control it, we have a simple file called <code>publication.yaml</code> which tells the system <em>how</em> to create the output. This would go at the top-level file.</p>
<pre><code class="language-yaml">metadata:
title: Test Project
author: D. Moonfire
language: en
theme: "@mfgames-writing/clean"
outputDirectory: .
outputFilename: test-project-{{edition.version}}.{{edition.editionName}}
editions:
epub: # This is the edition.editionName.
format: "@mfgames-writing/epub2"
contents:
- element: chapter
number: 1
directory: chapters
source: /^chapter-\d+.md$/
start: true
page: 1
</code></pre>
<p>Now, since this file is the core of MfGames Writing, it needs a bunch of details.</p>
<h2>Liquid</h2>
<p>Many of the fields use <a href="https://shopify.github.io/liquid/">Liquid</a> templates. That is the stuff between <code>{{</code> and <code>}}</code>. The main reason we use it is so we can simplify some of our logic. For example, instead of putting <code>outputFilename</code> in every edition, we can use variables to fill it in. Also, we can substitute <code>{{ edition.version }}`` to have the version from the </code>package.json<code>file. The</code>{{edition.editionName}}<code>will have</code>epub<code>for the</code>epub:<code>line. This means it wil produce a file</code>test-project-0.0.0.epub` when it runs.</p>
<h2>Theme</h2>
<p>The formatting is driven around the idea of a “theme”. Right now, there are only three themes. Two are private or require custom fonts, the other is <a href="https://www.npmjs.com/package/@mfgames-writing/clean">@mfgames-writing/clean</a> which is a utilitarian theme based on SASS and some templates. These themes are used to create a consistent style. I also have more specialized themes:</p>
<ul>
<li><a href="https://www.npmjs.com/package/@fedran/writing-theme/">Fedran</a>: For the books and stories at <a href="https://fedran.com/">https://fedran.com/</a>. This also is used for my print books.</li>
<li><a href="https://www.npmjs.com/package/@typewriter-press/efferding-writing-theme">Efferding</a>: For Randy Roeder's <a href="https://typewriter.press/randy-roeder/sins-of-intent/">Sins of Intent</a> and the sequel coming out next month.</li>
</ul>
<p>Both of these have custom fonts (Corda and Mr. Eaves) with a bit of logic. I could easily see this being expanded by anyone to create a gallery of formatting for books. Right now, I'm focusing on these three because they are functional enough.</p>
<p>See the later post on theme specifics.</p>
<h2>Editions</h2>
<p>Editions are basically versions of the output. For <em>Sand and Blood</em>, I have one for EPUB, HTML, and PDF. MOBI and DOCX are generated from those three sources. You can have any number of editions, different file names, different output directories, even different variables like ISBN numbers.</p>
<h2>Content</h2>
<p>Content is what goes into the file. Right now, we only have a single element, all the files in the chapter folder.</p>
<pre><code class="language-yaml">- element: chapter
number: 1
directory: chapters
source: /^chapter-\d+.md$/
start: true
page: 1
</code></pre>
<p>The <code>element</code> tells the theme how to format it. Later examples will include more but you can have things like <code>preface</code> or <code>appendix</code> or titles and the like.</p>
<p>The <code>number</code> property basically is used if you want to have “Chapter 1” in the theme but with a custom title. If your chapter is just “One”, “Two”, “Three”, you wouldn't define this. My books do since I use chapter titles.</p>
<p>The <code>directory</code> and <code>source</code> work together to figure out which files are to be included.</p>
<p><code>start</code> is used by EPUB and MOBI to figure out where to automatically navigate when the reader first goes into the page.</p>
<p>Finally <code>page</code> is used by the PDF system to restart page numbering. Usually this would be left blank.</p>
<h1>Installing Tools</h1>
<p>At this point, I have a minimum novel project. Time to make it actually produce something. Like in the <a href="/blog/2018/08/21/mfgames-writing-reasons/">previous</a>, everything is self-contained in the directory. We use <code>npm</code> to install the packages needed to build the project.</p>
<pre><code class="language-shell">$ npm install @mfgames-writing/format @mfgames-writing/clean @mfgames-writing/epub2
</code></pre>
<p>The <code>@mfgames-writing/format</code> provides the command-line tools and does the work. There is also the “clean” theme and a third for the output (EPUB2).</p>
<p>Once these install, running the output should work with the following command.</p>
<pre><code class="language-shell">$ npx mfgames-writing-format build
... lots of colorful output
$ ls *.epub
test-project-0.0.0.epub
</code></pre>
<p>If all goes well, you'll see <code>test-project-0.0.0.epub</code> in the root directory. It won't pass <code>epubcheck</code> check (we are missing a few things), but this is the basics of the publishing system.</p>
The Evolution of MfGames Writing2018-08-21T05:00:00Zhttps://d.moonfire.us/blog/2018/08/21/mfgames-writing-reasons/A brief history and reasoning behind MfGames Writing.
<p>If I wrote the reasons and methods I got to this point, I haven't either <a href="/tags/mfgames-writing/">tagged</a> properly or never got around to writing it. Regardless, here is a very brief history of these tools and how they evolved up to this point. Hopefully it will explain <em>why</em> I did certain things and which problems I was trying to solve.</p>
<h1>Series</h1>
<p>I appear to be writing a short series of post about the tools I use for publication and writing.</p>
<ol>
<li><a href="/blog/2018/08/13/publishing-processes/">Semantic Versions and Releases</a>: Why semantic versioning helps with the writing process.</li>
<li>Evolution of MfGames Writing: A brief history and reasoning behind the tools.</li>
<li><a href="/blog/2018/08/22/mfgames-writing-init/">First Steps Using MfGames Writing</a>: Starting a new project with MfGames Writing.</li>
<li><a href="/blog/2018/08/23/mfgames-writing-content/">Adding Content to MfGames Writing</a>: Adding front and back matter to novels.</li>
<li><a href="/blog/2018/08/24/mfgames-writing-docker-and-ci/">Working with MfGames Writing, CI, and Docker</a>: Adding automatic building with commits.</li>
<li><a href="/blog/2018/08/25/mfgames-writing-formats/">Additional Formats for MfGames Writing</a>: How to create PDF, MOBI, DOCX, and HTML versions.</li>
<li><a href="/blog/2018/08/26/mfgames-writing-themes/">Theming for MfGames Writing</a>: A light introduction on how to customize the output.</li>
<li><a href="/blog/2018/08/27/mfgames-writing-releases/">Integrating Semantic Versioning into MfGames Writing</a>: Tying semantic releases into the process.</li>
</ol>
<h1>Markdown and YAML</h1>
<p>Let's start with the start of this: I write in Markdown. I've written a <a href="/tags/markdown/">few posts</a> about the process and reasons, so I won't go into those. If you want a short version: text files don't corrupt and source control means I can work across multiple machines without accidently deleting an entire chapter.</p>
<p>The problem with Markdown is that most people don't want to read a text file. It has no formatting. It doesn't look pretty. It also doesn't go into books very well. Normally, to get around this, many authors use tools like Microsoft Word or <a href="https://www.literatureandlatte.com/scrivener/overview">Scrivner</a> (which I recommend to non-technical writers) which allow for writing a story/novel and then formatting it into something usable for readers. I don't use Scrivner myself because it uses binary files (doesn't play well with Git) and doesn't run nicely on Linux (my primary platform).</p>
<p>Most people aren't burdened by these self-inflicted limitations.</p>
<p>I also use YAML to keep track of useful information about the page.</p>
<h1>Model-View-Controller as Applied to Writing</h1>
<p>In development, there is the concept of the <a href="https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller">Model-View-Controller</a>. This consists of three parts:</p>
<ul>
<li>The model is the logic and data.</li>
<li>The view renders/formats the model into something useful.</li>
<li>The controller changes the model.</li>
</ul>
<p>The nice thing is that when writing in Markdown, I don't worry about formatting. I have four basic things: bold, italic, blockquotes, and section breaks. I don't really need anything else because of my writing style. I also think this contributes to my productivity, if I can't play with it, I can't distract myself. This is basically the <em>model</em> of my novel; the chunky bits in a text file that constitute the contents of the novel.</p>
<p>The <em>view</em> is what takes that Markdown + YAML file and generates something useful or pretty out of it. That can include:</p>
<ul>
<li><a href="https://fedran.com/sand-and-blood/chapter-01/">Online Reading</a>: HTML pages for reading online</li>
<li><a href="https://fedran.com/sand-and-blood/dmoonfire-100-00-sand-and-blood-3.1.0.pdf">PDF</a>: PDF for print-ready output.</li>
<li><a href="https://fedran.com/sand-and-blood/dmoonfire-100-00-sand-and-blood-3.1.0.epub">EPUB</a>: Ebook formats for those who like readers.</li>
</ul>
<p>The <em>controller</em> is what I write with the novel. I basically use <a href="https://atom.io/">Atom</a> or Notepad++. This is also where <a href="/tags/author-intrusion/">Author Intrusion</a> will fit into the process.</p>
<p>The nice part about having the contents (the <em>model</em>) and the output (the <em>view</em>) is that they are completely separate in this step. If I decided to change the font of my books, I just update the view and reformat the books. All the words and everything will come over and I will just have a new appearance. The same if I want to change how epigraphs are formatted or want to support a new ebook format.</p>
<h1>Automation</h1>
<p>Having a separate view from the model also means I can <em>automate</em> the process of formatting and typesetting the book. This is useful because it is a tedious job formatting the book. Yes, I don't do it that often but I also like consistency. So as my “style” evolves, I also find that I go back and tweak the previous versions for consistency. For example, I moved from a 2 cm lead on the chapter pages to a 4 cm because I thought it looked better. I also made the epigraph a little larger because I can read 8 pt font easily but not everyone else.</p>
<p>Automation also takes out the little details that throw me. I know some books that start with “Chapter 1” and “Chapter 2”, but somewhere in the middle they switch to “Chapter Twenty-Two”. Or sometimes using typographical quotes and other time using ticks. These are things that will throw out a detail-oriented person (like me) but I also feel make an unprofessional looking book.</p>
<p>These are things that Microsoft Word and Scrivner will do, some better than others. I want the same thing because my goal is to create as technically correct and beautiful book as I can possibly create.</p>
<p>MfGames Writing sits in this place, to automate the formatting and handle all of the little details while giving me the ability to focus on one thing—the words—and let computers handle the rest.</p>
<h1>Early Versions</h1>
<p>When I started this epic quest of trying to automate formatting and typesetting, I started with a simple program: Make. It basically used a few tools in Linux to transform the Markdown into something usable. Shell scripts ended up not being enough, so I supplemented those with a combination of Perl, Python, and C# tools. Each language had its strengths and weaknesses (the biggest is how they handle Unicode since I use accents heavily).</p>
<p><em>Where most people write a little game or card program with new languages, I write a publication system.</em></p>
<p>At the time, those languages didn't have good support for installation. Both Perl and Python have a tendency to install packages for the entire machine or a user. C# didn't have NuGet at the time, so it was copying files around (plus it was always the weakest of the three implementations). This meant I could only have one version at a time, so I basically directed everything to my <code>/home/mfgames-writing/</code> folder and used that.</p>
<p>Which… was good enough until I started changing my build process. I've been using this for about a hundred books and short stories now. The novels were the hardest because I would add features for fancy headers (like the PlayStation controller on top of <a href="https://typewriter.press/shannon-ryan/fangs-for-nothing/">Fangs for Nothing</a>) or epigraphs in my page. As I went along, the process got better.</p>
<p>It also broke older versions. A template would change, or the flow. I started having to have <code>v1</code>, <code>v2</code>, and <code>v6</code> hanging around because of the drastic changes as the tools got better at formatting and typesetting. Every time I would rewrite it, I would have slightly different input because of the language changes, and that would break older ones.</p>
<p>That's when I realized I needed to be able to “pin” a version of the tools for a specific book.</p>
<p>The latest attempt at C# was to take advantage of NuGet but then I realized the default NuGet implementation also created a machine-wide version.</p>
<p>I'm aware there are virtual environments for Python. I didn't know about them at the time. After I've played with them, I still find them a bit cumbersome compared to using the approach I went with.</p>
<h1>TypeScript and NPM</h1>
<p>I finally figured it out when I started playing with TypeScript. I like learning new languages and I was looking for a replacement for my scripting language (I'm <em>very</em> good at Perl and pretty good at Python). One of the tools with JavaScript based languages is <a href="https://npmjs.com/">NPM</a> which is the package management (much like NuGet).</p>
<p>The beauty of NPM is that it is <em>self-contained</em> inside the development projects. All of the required files go into <code>node_modules/</code> which means that each project is a folder that contains everything it needs to run and build. This also means that different projects can have different versions of the same library.</p>
<p>This ended up being the solution I went with. A self-contained, project-level packaging system that would let me evolve the tools for one version but still keep another novel at an earlier vesion so it doesn't break.</p>
<p>Fast-forward about a year of coding and tooling and we get to where we are, MfGames Writing.</p>
Semantic Versions and Releases for Publishing2018-08-13T05:00:00Zhttps://d.moonfire.us/blog/2018/08/13/publishing-processes/Over the last few weeks, I've been tweaking my publishing process to automate more of the release and generation process.
<p>Over the last few weeks, I've been tweaking my publishing process. I think I have something pretty stable and started using it for my <a href="/tags/flight-of-the-scions/">next novel</a>. After talking about it on social networks, a number of people suggested I write up what I've learned.</p>
<h1>Series</h1>
<p>I appear to be writing a short series of post about the tools I use for publication and writing.</p>
<ol>
<li>Semantic Versions and Releases: Why semantic versioning helps with the writing process.</li>
<li><a href="/blog/2018/08/21/mfgames-writing-reasons/">Evolution of MfGames Writing</a>: A brief history and reasoning behind the tools.</li>
<li><a href="/blog/2018/08/22/mfgames-writing-init/">First Steps Using MfGames Writing</a>: Starting a new project with MfGames Writing.</li>
<li><a href="/blog/2018/08/23/mfgames-writing-content/">Adding Content to MfGames Writing</a>: Adding front and back matter to novels.</li>
<li><a href="/blog/2018/08/24/mfgames-writing-docker-and-ci/">Working with MfGames Writing, CI, and Docker</a>: Adding automatic building with commits.</li>
<li><a href="/blog/2018/08/25/mfgames-writing-formats/">Additional Formats for MfGames Writing</a>: How to create PDF, MOBI, DOCX, and HTML versions.</li>
<li><a href="/blog/2018/08/26/mfgames-writing-themes/">Theming for MfGames Writing</a>: A light introduction on how to customize the output.</li>
<li><a href="/blog/2018/08/27/mfgames-writing-releases/">Integrating Semantic Versioning into MfGames Writing</a>: Tying semantic releases into the process.</li>
</ol>
<h1>Software Processes</h1>
<p>Much of how I manage writing projects is to treat them like software. It works well for framing what I have done, need to get done, and what the end goal is. Because of this, I use various features of <a href="https://gitlab.com/">GitLab</a> pretty heavily: issue tracking for arcs, issues for the publication process, and even more to remind me to order books; milestones to keep me encouraged; continuous integration (CI) to simplify the production of EPUB, MOBI, and PDF versions of the file.</p>
<p>I've also written a number of framework tools such as <a href="https://gitlab.com/mfgames-writing/">MfGames Writing</a> to work with CI services for the publication or help me manage stuff (<code>markdowny</code>).</p>
<p>Recently, I got into a discussion about treating novels as software. Some of the things I do are… strict and not everyone will care for them, but I like to think I'm making my reader's lives easier by the little details that go into my processes.</p>
<h1>Semantic Versioning</h1>
<p>Somewhere in 2014 or so, I fell in love with <a href="https://semver.org/">semantic versioning</a>. Semantic versioning is a movement to create a common understanding (a grammar) of version numbers that can be understood by anyone to understand the significance of any given release.</p>
<p>A semantic version comes in three parts: major.minor.patch.</p>
<ul>
<li>If a major number increases, it means there was a “breaking change”. In writing terms, it would be rewriting sections, changing motivations, removing scenes, and generally changing the meaning of the novel. I would say adding a new epilogue would be reasonably called a major change because it changes the meaning and plot of the novel.</li>
<li>A minor version is adding features or improvements. As I see it, a minor version is one that I would be adding details to clarify a scene, make it more obvious or more understandable. Maybe adding a paragraph or so would be a minor change.</li>
<li>Finally, there is a patch. This is fixing typos, character names, or the missing quotes. Sadly, I seem to have a lot of patches as I write, because I only notice those annoying little things days or weeks later.</li>
</ul>
<p>I put the version on the legal page so it is present but not obvious unless one is looking for it. There are a number of reasons I do this. The biggest is <em>The Deeds of Paksenarrion</em> by Elizabeth Moon. I have two copies of this omnibus book, mainly because I love it so much. They are two different printings. One of them had a few typos in it that they corrected with a later printing. Having an indicator that there were changes (in this case, a “patch”) would tell me there is something different about these two books.</p>
<p>Lucas, with his reworking of the first three <em>Star Wars</em> movies, would have either new features (the new crowd scenes at the end of <em>The Return of the Jedi</em>) or breaking changes (Greedo shooting first). An example of a patch is redoing the unintended transparency of the X-Wing dash during the fight scenes or reworking the undercarriage of the hover car in the desert (Vaseline verses CGI).</p>
<p>I want to know those changes so when I look at a copy and notice that only the last number changed, it just means there were some typo fixes. If the first, then I start to ask questions, why?</p>
<p>This is also important because some authors will rewrite their books after publication. I also use it when sending books out to alpha and beta readers. Since I get the responses in a semi-random order, it is easier to know that I'm getting feedback from version 0.4.0 instead of 0.5.0. With Git, I can create a branch, manually handle the edits, and then pull them back into the master branch which will then help me consolidate changes between two versions.</p>
<p>Figuring out what goes where (major, minor, or patch) can be a rough.</p>
<p>In <em>Sand and Blood</em>, I'm on version 3.1.0. I switched to version 2.0.0 when I made the book <a href="https://creativecommons.org/">Creative Commons</a>. A license change (from “All Rights Reserved”) felt like a breaking change. I bumped to 3.0.0 when I realized there was a major plot hole about how the clans dealt with the dead (they don't talk about it), so I <em>removed</em> some conversations because I had a better understanding of the world after <em>Sand and Ash</em>.</p>
<h1>Changes</h1>
<p>There is an important part about version numbers: why. Knowing that someone bought version 1.1.2 of a book and now I'm on 3.1.0 would beg the question: what major changes have happened.</p>
<p>That is where the <a href="https://fedran.com/sand-and-blood/changelog/">change log</a> comes in. I'm still working on the format, but I try to document every version of the book, from major to minor to patch. That way, someone can look at the versions and decide for themselves if they want a copy (or to ask for a replacement, which I've done also).</p>
<h1>Semantic Releases</h1>
<p>Of course, the problem with change logs is that they require some effort to maintain. Over the last year, I stumbled on the concept of a <a href="https://semantic-release.gitbook.io/semantic-release/">semantic-release</a>. A semantic release is an extension of the version but it automates much of the processes that are involved with figuring out what type of number has to be bumped in the version and creating the appropriate log entries.</p>
<p>One of the goals is to be <a href="http://sentimentalversioning.org/">unromantic</a> about version numbers. I'll admit, I've been romantic about the versions up to this point, but there is something about the systems approach to versions that takes the human factor out and reduces the amount of work.</p>
<p>Basically, <a href="https://github.com/semantic-release/semantic-release">semantic-release</a> is a workflow for NPM packages (and novels in my case) that automates the calculations of verison numbers and also updates the change. It has plugins for publishing that I could use to update my website with the latest versions whenever I make a minor change.</p>
<p>It also means that the versions are much smaller. Instead of having 5-10 items in a release, there is probably just one change between versions.</p>
<h1>Commit Conventions</h1>
<p>Since I work with Git, there is one component that needs to be done to make semantic-release: conventions. In specific, I use <a href="https://conventionalcommits.org/">conventional commits</a> which is a standards to identify if something is a minor (a “feature”) or a patch (“fix”). These are done by a <em>convention</em> of the commit messages:</p>
<ul>
<li>feat: integrated edits from Marta B</li>
<li>fix: fixed a typo with Rutejimo's missing accent</li>
<li>ci: trying to get it to build on GitLab</li>
</ul>
<p>The semantic-release package parses the commit messages (in specific, the <a href="https://github.com/angular/angular/blob/22b96b9/CONTRIBUTING.md#-commit-message-guidelines">Angular</a> conventions) and figures out if it should be releases (any “BREAKING CHANGES”, "feat:", or “fix:” that it sees). Other commits don't bump up the version but still give me the ability to make changes.</p>
<p>I like the structure of using conventional commits and having it automatically feed into new version numbers and creating the change log. Later, it can be used to automate my <a href="https://fedran.com/">website</a> and maybe even publish books on various vendors.</p>
<p>This automation is also important for authors published by the various <a href="https://typewriter.press/">Typewriter Press</a> imprints. All of them have access to their source files in GitLab. They can make changes without me approving of them. If they follow the same conventions, then it will bump up to the versions for <em>their</em> corrections without me tracking them or asking what changed. In that way, it allows non-technical writers to also participate in this process; all without getting overwhelmed by the technical details while still not requiring them to keep track of changes.</p>
<h1>Tracking</h1>
<p>Since the version number is pulled into the legal page, it also means that I can easily manage alpha and beta readers, different rounds of editors, and even coordinate changes after giving out multiple copies at the same time (my alpha and beta readers are concurrent).</p>
<h1>Software</h1>
<p>Much of these concepts also apply to my tools, so I also switched all of MfGames Writing tools to use semantic-release and automate the deployment. Like writing, it lets me focus on what I want to do (write) and automate the tedious stuff (release management).</p>
MfGames Tasks v1.1.22018-07-13T05:00:00Zhttps://d.moonfire.us/blog/2018/07/13/mfgames-tasks-v1.1.2/I decided to spend two days extending the features of MfGames Tasks to include calendar dates and a simplified budgeting.
<p>Earlier this year, I <a href="/blog/2018/02/23/mfgames-tasks/">wrote</a> a tool to help me gather various assigned issues from <a href="https://gitlab.com/">GitLab</a> and <a href="https://github.com/">GitHub</a>. Recently, we missed a dentist appointement and I decided to include the ability to pull in iCalendar records (specifically Google Calendar). I got it done (along with some patches) and published it.</p>
<h1>Calendars</h1>
<p>It was actually fairly easy to do, I spent most of the time refactoring the code to make it easier. Once I got that done, I found a couple of libraries on NPM that would parse and manage iCalendar. The next thing I knew it, I had it pulling in calendar entries.</p>
<p>The configuration is pretty simple:</p>
<pre><code class="language-yaml">sources:
- id: household
type: icalendar
title: events / household
sort: _000
for: 1 month
url: https://calendar.google.com/path/basic.ics
</code></pre>
<p>The useful fields are:</p>
<ul>
<li><code>type</code>: Tells the system to use the iCalendar mode.</li>
<li><code>sort</code>: If provided, lets you move the events to the top or bottom. I'm using <code>_000</code> to jam it up at the top of the file.</li>
<li><code>for</code>: This is in the format of “X type” such as “3 days”, “1 year”, “2 weeks”.</li>
</ul>
<p>When run, it generates a Markdown like:</p>
<pre><code class="language-markdown"># events / household
- Payday {2018-07-16}
</code></pre>
<h1>Budgeting</h1>
<p>I'll preface this with “I'm not good at money management skills”. Tracking money? I love doing that, but there are too many variables and inconsistencies that make paying bills consistently hard for me. Having a kid in day care doesn't help.</p>
<p>To help that, I decided to extend the iCalendar feature to include a simple budget that I can put at the bottom of my notes file. The main thing is to let me know if I'm going to expect an overdraft.</p>
<pre><code class="language-yaml">sources:
- id: budget
type: icalendar-budget
title: budget / household
sort: zzz
categories:
- title: Until Payday
dayOfMonth: [1, 16]
- title: Next Month
for: 1 month
- title: Next Quarter
for: 3 months
- title: Per Paycheck
for: 1 year
avg: 24 # semi-monthly paychecks
url: https://whichUrl
</code></pre>
<p>In the calendar, I have entries with titles like:</p>
<ul>
<li>$-170.00 Day Care</li>
<li>$300.00 Payday</li>
</ul>
<p>The system parses all of those (including repeating rules) and then combines them together into a notes section.</p>
<pre><code class="language-markdown"># budget / household
- Until Payday (in 2 days): $-222.22 = 111.11 - 333.33
- Next Month: $-2,222.22 = 1,111.11 - 3,333.33
- Next Quarter: $-22,221.22 = 11,111.11 - 3,3333.33
- Per Paycheck: $111.00 = 444.44 - 333.33
</code></pre>
<p>Like before, the <code>sort</code> is used to jam it to the bottom.</p>
<p>Ideally, I'd get better at money management… but that will take a bit longer. I have a lot of bad habits to unlearn.</p>
<h1>Development</h1>
<p>With this release, I decided to use some automated tools for development. One of them is <a href="https://conventionalcommits.org/">conventional commits</a>. This is commit-line specification that tools can use to automatically build up a <a href="https://gitlab.com/dmoonfire/mfgames-tasks-js-cli/blob/master/CHANGELOG.md">change log</a> to give something useful for users to read.</p>
<p>I know not a lot of people use my libraries, but learning to be more structured about development on my open-source software projects. It will also help with working on <a href="/tags/author-intrusion/">Author Intrusion</a> with the assumption that the project will get far bigger in the following years.</p>
<p>Of course, it probably won't stop the “push a version… found a bug in ten seconds, push another… then another” that I need to work on.</p>
<p>Overall, having a commit message format such as:</p>
<ul>
<li>fix(calendar): calendar dates should be exclusive of end date</li>
</ul>
<p>Isn't too hard, but the structure means that the change log will be automatically updated as part of the release. It also means I don't have to figure out <a href="https://semver.org/">version number</a> bumping.</p>
<p>I suspect I'll start doing this with novels also.</p>
<h1>Installation</h1>
<p>I wrote this in Typescript because I really like it as a scripting language and it has a nice ecosystem. Installation is pretty simple:</p>
<pre><code class="language-shell">npm install --global mfgames-tasks-cli
</code></pre>
<h1>Documentation</h1>
<p>Yeah, I still don't have a lot of documentation, just a single file.</p>
<ul>
<li><a href="https://gitlab.com/dmoonfire/mfgames-tasks-js-cli">GitLab</a></li>
<li><a href="https://www.npmjs.com/package/mfgames-tasks-cli">NPM</a></li>
</ul>
Author Intrusion v0.10.02018-07-11T05:00:00Zhttps://d.moonfire.us/blog/2018/07/11/author-intrusion-0.10.0/I finished all my assigned issues for Author Intrusion v0.10.0. Here is a brief summary of the changes made to this unstable project.
<p>Even though it isn't quite Saturday, I finished all the issues for Author Intrusion <a href="https://gitlab.com/author-intrusion/author-intrusion-cil/milestones/1">v0.10.0</a> so I finalized the milestone and decided to give a celebratory post to announce it.</p>
<p>This still isn't even remotely stable enough to use but I think I need a cadence to keep working on it instead of letting it atrophy for a few months before going back and losing track of things. So, to keep it fresh in my mind, I'm going to try keeping with a two week cadence where I do at least <em>something</em> on the project to keep it going, updated, and working. Over time (not unlike writing a novel), this should produce useful results for other people.</p>
<h1>v0.10.0</h1>
<p>The <code>v0.10.0</code> release is mainly to improve developing on the project. Most of the changes aren't very sexy: allowing packages to be force installed for debugging, adding scripts, reducing noise.</p>
<h2>Paragraph Splitting</h2>
<p>One of the reasons this is complicated is that I have to break apart English (a non-structured way of communicating) into discrete components. Looking at the first three words of a paragraph requires the system to know what a paragraph is for.</p>
<p>In the v0.9.0 version, I did a quick and dirty paragraph splitter. It failed on some of my bigger projects so I rewrote it to be faster and use less memory (<code>foreach</code> loop instead of <code>RegEx</code>).</p>
<p>I still have to do one for the tokens and I realized that I also need to find a better way of handling large documents. C# doesn't like objects over 85 kB. My largest story (single file document) is 43 kw (kilowords) and 238 kB. Also, C# uses UTF-16 which means loading that entire thing into memory requires a bit over 480 kB of RAM. That will be a bigger mess but it is low enough it needs to be dealt with sooner than later.</p>
<h2>XSLT Functions</h2>
<p>I added a <code>length()</code> function for the XSLT calls. That way, plugins like echo detection can ignore short words.</p>
<pre><code>plugins:
analysis:
- compare: text()
error: 5
plugin: EchoDetection
key: echo-1
warning: 2
within: 200
select: //token[length() > 4]
</code></pre>
<h2>Logging</h2>
<p>One of the biggest things was reducing information overload by breaking apart the logging into different categories. Like <em>MPlayer</em>, there are a lot of things going on, so I added a switch.</p>
<pre><code>./pcli analyze --log NuGet:verbose --verbose
</code></pre>
<p>The <code>--verbose</code> turns on what is logged to the console, the <code>--log NuGet:verbose</code> turns the NuGet management section from it's default warning to verbose to get the tedious details.</p>
<pre><code>./pcli log-list
</code></pre>
<p>The <code>log-list</code> version will let you see the categories. Plugins can add additional logging targets which is why it's a project-based command but even without a project, it should work (we'll find out).</p>
<h1>v0.11.0</h1>
<p>The next sprint, starting next Sunday, will be focused on those memory management problems. I think it will take me a while to puzzle through them.</p>
<h1>v0.12.0</h1>
<p>The sprint after that is currently slated to be working on server mode. This is going to be used by the <a href="https://langserver.org/">Language Server Protocol</a> which will let me hook up to <a href="https://atom.io">Atom</a> and get real-time analysis, highlighting, and other fancy features.</p>
<h1>Development</h1>
<p>Author Intrusion is currently being managed via its <a href="https://gitlab.com/author-intrusion/author-intrusion-cil">Gitlab project</a>. I'm not sure if it would be worthwhile for anyone to consider joining, but if you want to watch it, this would be the place.</p>
<p>If you have questions, please don't hesitate to poke me on any <a href="/contact/">social network I'm on</a>. I always love to bounce ideas or talk about future place. The more I do, the more I can make it useful for everyone, not just myself.</p>
Author Intrusion v0.9.02018-06-26T05:00:00Zhttps://d.moonfire.us/blog/2018/06/26/author-intrusion-0.9.0/After a few weeks of work, the current rewrite of Author Intrusion got to a stopping point. This has the minimum functionality to detect echo works but it has a long way to go.
<p>After a few weeks of work, the current rewrite of Author Intrusion got to a stopping point. This has the <strong>barest</strong> minimum functionality to detect echo words but it has a <strong>long</strong> way to go. Depressingly long way, but I need to let it settle a little before I jump back into it.</p>
<p>I also want a little encouragement so I'm going to toot my own horn and show some progress.</p>
<h1>Starting Over Again</h1>
<p>I think this is the ninth attempt I've had to write <span class="missing-link" data-path="/tags/author-intrusion">Author Intrusion</span>. Each time, I've encountered various walls where my ideas didn't have the performance or couldn't conceptually move beyond the proof of concept. Over the last eight years, I've learned a lot about writing this and each time I hope “this is it”.</p>
<p>The current goal is to write a command-line interface (CLI) inspired by compilers (like GCC and TypeScript <code>tsc</code>) and Git. Basically, have AI have a core set of function but don't worry about re-implementing a text editor (which was, in all honesty, one of my more common mistakes for previous versions).</p>
<p>I spent a <em>lot</em> of time trying to get a solid, cross-platform GUI for this. That includes various Gtk# and Electron implementations before I decided to switch to CLIs.</p>
<p>For reference, this is some of my efforts five years ago:</p>
<img class="img-responsive post-img-link" src="/assets/2011/03/editor-screenshot.png" alt="Author Intrusion v0.1.0" />
<img class="img-responsive post-img-link" src="/assets/2013/10/author-intrusion-0.4.0.png" alt="Author Intrusion v0.4.0" />
<img class="img-responsive post-img-link" src="/assets/2013/11/author-intrusion-0.5.0-3.png" alt="Author Intrusion v0.5.0" />
<p>The “mmm” was a search and replace on one of my larger commissions to test performance. In this case, it was a 100k commissioned novel.</p>
<p>There were a lot of attempts in there and I spent a lot of time trying to create an editor. While I was “fairly” successful, I think it made the project too big for one person to do. Each major iteration, I've been removing features trying to get to a core functionality while still honoring my core goals of helping me write.</p>
<h1>Current Implementation</h1>
<p>After the attempt to write it in Typescript, I decided this version is going to use .NET Core. The folks at Microsoft have done an amazing job of writing something that is fast and capable while running on both Windows and Linux (one of my requirements). It also is my core language, so I'm not struggling with learning and the tools like I was with Typescript. My primary environment is Visual Studio 2017 with <a href="http://www.jetbrains.com/resharper/">ReSharper</a>, <a href="http://www.jetbrains.com/rider/">Rider</a>, or <a href="https://code.visualstudio.com/">Visual Studio Code</a>. While I use <a href="https://atom.io/">Atom</a>, global configuration options pretty much means my Atom setup is specifically for writing, not coding.</p>
<h1>NuGet</h1>
<p>However, after the Typescript implementation and later code, I realize that the approach <code>npm</code> uses to manage packages is perfect for what I'm looking for. My struggles with various incarnations of <a href="https://gitlab.com/mfgames-writing">MfGames Writing</a> showed me that bitrot is a major problem while writing over years. The earlier incarnations of the build framework would evolve to handle new novels but then it would break older generation in the process. With <code>npm</code>, I can have a specific version in one project, then the library could continue to evolve while still providing the ability to stay at an older version for those older works.</p>
<p><a href="https://nuget.org/">NuGet</a> has a number of C# libraries for writing a client that would give me the same thing. Various utilities, analyzers, and libraries can be packaged up as NuGet packages and then installed. If they evolve, the older version can remain behind and still work.</p>
<p>This version has the basics of this in, I can install packages and have various functionality available for processing.</p>
<img class="img-responsive post-img-link" src="ai-plugins.gif" alt="4 screenshots of adding plugins to author intrusion" />
<p>This part is actually one of the neatest parts, I think. I'm using <a href="https://autofac.org/">Autofac</a>, a bit of reflection, and the NuGet libraries to install packages and then load assemblies from those packages without needing to pull them into a central location.</p>
<p>As soon as the plugins load, the system rebuilds the plugins and injects functions. This can be various plugins, new XSLT functions, or anything else.</p>
<h1>Layout Plugins</h1>
<p>The above screenshot is an example of the layout plugin. It tags files in the project as to their purpose. I made a mistake earlier on this when I started doing grammar checking on notes. In this case, a plugin can indicate something as “content” (e.g., the novel or story) or “lookup” (notes) or something else.</p>
<p>Right now, I'm only implementing my “standard” project layout that I've used for the last few projects. Eventually I'll write more but I'm trying to get end-to-end before fleshing out ideas as needed.</p>
<p>A layout plugin is responsible for gathering metadata about a file so decisions can be made. Eventually this will go into the YAML header for the file. This means I'll be able to identify files that have a specific point of view with something like:</p>
<pre><code>---
title: Chapter 1
pov: Dylan
swain: scene
---
It was a bright and depressingly sunny day....
</code></pre>
<p>The idea for this is to be able to list all chapters of a given POV and then arrange them chronologically while listing the location. Or tag a file as being a <a href="https://en.wikipedia.org/wiki/Scene_and_sequel">scene or sequel</a> so they are interspersed correctly.</p>
<h1>Structure Plugins</h1>
<p>A structure plugin basically figures out the structure of a file, such as figuring out if it is broken into paragraphs, sentences, and words. I didn't want to hard-code this because sentence splitting is hard and expensive plus my <a href="https://fedran.com/sand-and-blood/chapter-01/">fantasy novels</a> all have epigraphs. Those who like Scrivner may want to arrange it into scenes.</p>
<p>Structure plugins are boring but critical.</p>
<h1>Xpath</h1>
<p>An important part of the structure is <em>not</em> applying a structure to certain files. Lookup files don't need to know the individual words or paragraphs. To limit it, I'm using a <code>scope</code> variable which is an Xpath into the project.</p>
<pre><code><project>
<file path="/chapters/chapter-01.md" class="content" />
<file path="/characters/dylan.md" class="lookup" />
</project>
</code></pre>
<p>This means, using a path of <code>/content[is-content()]</code> will select only the content files but not the lookup. Originally, I implemented this as a CSS-like library which I eventually realized I was going down a rabbit hole (I still broke apart the library for later if I need it).</p>
<p>Again, C# has the ability to have defined XSLT functions so I wrote <code>is-content()</code> (injected via Autofac) that does custom logic.</p>
<p>As the various structure plugins operate (defined by the <code>author-intrusion.aipy</code> file), it will extend the XML structure used for selections.</p>
<pre><code><project>
<file path="/chapters/chapter-01.md" class="content">
<para start="0" length="10" />
<para start="11" length="21" />
</file>
<file path="/characters/dylan.md" class="lookup" />
</project>
</code></pre>
<pre><code><project>
<file path="/chapters/chapter-01.md" class="content">
<para start="0" length="10">
<token start="0" length="3" />
<token start="5" length="4" />
<token start="9" length="1" />
</para>
<para start="11" length="21">
<token start="11" length="3" />
<token start="16" length="4" />
<token start="20" length="1" />
</para>
</file>
<file path="/characters/dylan.md" class="lookup" />
</project>
</code></pre>
<p>This is because of another previous mistake (yeah, I made a lot). Various implementations tried to normalize contents while writing. This meant it would correct double spaces after periods or adjust the text.</p>
<p>That didn't work.</p>
<p>I also couldn't break it down into a simple tree structure because English didn't fit well. So, this XML just goes into the original file to get the text.</p>
<h1>Analysis Plugins</h1>
<p>The entire reason to break apart a document into a structure is for the analysis. An analysis plugin, such as echo detection, uses the XML structure to gather information.</p>
<p>All plugins are configured in the <code>author-intrusion.aipy</code> (project file in YAML format).</p>
<pre><code>plugins:
analysis:
- plugin: EchoDetection
key: echoes
scope: content
select: //token[length() > 3]
compare: text()
within: 20
warning: 2
error: 5
</code></pre>
<p>The <code>plugin</code> attribute is the class to use the plugin. The key is just a label for Atom's linter in case someone uses multiple echo detections in a file.</p>
<p>The second section is to figure out what is being detected. The <code>scope</code> uses <code>content</code> which is a shorthand for <code>/file[is-content()]</code> or <code>/file[has-class("content")]</code>. This breaks apart the search process. Using <code>/</code> for the path would do echo detection across chapters (and require more memory and would be slower) while the file-level ones makes it more efficient by only comparing a single file against itself.</p>
<p>For every scope, the <code>select</code> figures out what is going to be compared. In this example, for every file, we select every word over three characters long (<code>//token[length() > 3]</code>). If we had three chapters of a thousand words each, this is a different of a single list of three thousand items (<code>scope: /</code>) verses three lists of a thousand each (<code>scope: //file[is-content()]</code>), or three hundred paragraphs of ten words each (<code>scope: //file[is-content()]/para</code>).</p>
<p>The third section is the <code>compare</code>. This basically figures out what to compare. The <code>text()</code> means the raw text of the file. However, functions to handle case-insensitivity, stemming (base words so <code>I jumped over the jumper</code> would have two echoes), Soundex (to find similar-sounding words near each other), or whatever else I need.</p>
<p>Finally, the echo detection has the rules for what is a detection. It basically counts how many identically entries (as determined by <code>compare</code>) in each <code>select</code>ed tag inside the <code>scope</code> within <code>within</code> entries. If that number is equal to or greater than <code>error</code>, then it marks that <code>select</code>ed element as an error.</p>
<p>In the above example, the echo detection says “for every word in a file, look at the twenty surrounding words. If there are five or more, it's an error, otherwise if there is two more then it's a warning”.</p>
<img class="img-responsive post-img-link" src="ai-analyze.png" alt="Author Intrusion v0.9.0 analyze example" />
<h1>Functionality</h1>
<p>This isn't even remotely polished at this point. The project is self contained in that building it will generate the correct data, it just isn't… pretty.</p>
<pre><code>dotnet run -v:q --no-build --no-restore --project src/AuthorIntrusion.Cli -- file-list -p "Examples/Sand and Blood"
</code></pre>
<p>Eventually, it should be something like:</p>
<pre><code>aicli file lists
</code></pre>
<p>There is also a lot of missing functionality, using it probably requires me to understand, though I'd like to think it is pretty simple. Then again, I wrote it, of course it's simple.</p>
<h1>Complexity</h1>
<p>You may have noticed that the <code>author-intrusion.aipy</code> file is somewhat complicated. This is actually intentional. There are a lot of tools for writers. I'm looking for something very specifically to help with flaws I'm aware of in my fiction, not a generic “one size fits all” application. Because of that, I need it to work for me instead of trying to inflict</p>
<p>This desire to customize to the author is purely inspired by James White's <a href="http://www.sectorgeneral.com/shortstories/fasttrip.html">Fast Trip</a>. If you have a chance, consider reading it. It talks about modifying the environment the way you need to work, not the other way around.</p>
<p>However, flexibility comes at a price: simplicity. I considered trying to make it easy, but I'm looking for something that looks for overuse of adverbs (technically correct) or gerunds. I want to be able to make sure a character only speaks in past tense or doesn't use a pronoun to identify themselves. One of the earlier ones I'm going to get done is looking for present tense outside of a quote. These are pie in the sky items, but I think possible.</p>
<p>Later, if this gains traction (e.g., users besides me), someone may come up with a fancy GUI or configuration wizard to add common settings.</p>
<h1>Development</h1>
<p>Author Intrusion is currently being managed via its <a href="https://gitlab.com/author-intrusion/author-intrusion-cil">Gitlab project</a>. I'm not sure if it would be worthwhile for anyone to consider joining, but if you want to watch it, this would be the place.</p>
<p>If you have questions, please don't hesitate to poke me on any <a href="/contact/">social network I'm on</a>. I always love to bounce ideas or talk about future place. The more I do, the more I can make it useful for everyone, not just myself.</p>
MfGames Tasks v0.02018-02-23T06:00:00Zhttps://d.moonfire.us/blog/2018/02/23/mfgames-tasks/To combat being overwhelmed by my many projects and obligations, I wrote a little tool to aggregate my todo lists across GitLab and GitHub.
<p>As some of you know, I have a lot of irons in the fire. Some of them are the natural problem of programmers where many interesting things never take root but sometimes there are a few that do and they quickly blossom in needing more time. In other cases, it is commissions due, weekly chapters, and the normal household to do list.</p>
<p>I keep most of my issues in <a href="https://gitlab.com/">GitLab</a> including things I have to do for sales tax, filing dates, and even per-project items. I also keep my novels and commissions there, along with associated issues to keep each moving.</p>
<p>So, when I get stressed out, I write things to help me. In this case, I decided to write <code>mfgames-tasks</code> which pulls all of those assigned issues and throws it into a single Markdown file which I keep on Dropbox across all my machines. This way, it picks up the items from each of those repositories and jams it into a file that looks like this:</p>
<pre><code class="language-markdown"># dmoonfire / mfgames-culture-js
- Errors representing some seconds and milliseconds
- https://api.github.com/repos/dmoonfire/mfgames-culture-js/issues/3
# dmoonfire / mfgames-locking-cil
- [simple] Add/update unit tests
- https://gitlab.com/dmoonfire/mfgames-locking-cil/issues/2
- [simple] Get this packaged and uploaded to NuGet
- https://gitlab.com/dmoonfire/mfgames-locking-cil/issues/1
# typewriter-press / events
- {2018-08-30} Register as an attendee
- https://gitlab.com/typewriter-press/events/issues/2
- {2018-03-30} Register as a vendor and get a table
- https://gitlab.com/typewriter-press/events/issues/1
</code></pre>
<p>It handles due dates (in <code>{2018-03-30}</code>) and labels (such as <code>[simple]</code>). If there are none, it skips them. On GitLab, if the issue is assigned to a milestone and doesn't have a due date of its own, it uses the milestone's due date.</p>
<p>Another big feature is that GitLab issues that are assigned to a milestone that hasn't started, it doesn't include them. This lets me set up quarterly or monthly milestones (e.g., taxes) and have them pop up when they are needed.</p>
<h1>Installation</h1>
<p>I wrote this in Typescript because I really like it as a scripting language and it has a nice ecosystem. Installation is pretty simple:</p>
<pre><code class="language-shell">npm install --global mfgames-tasks-cli
</code></pre>
<h1>Configuration</h1>
<p>As my preferred configuration file format, I have a <code>mfgames-tasks.yaml</code> file on my laptop. It looks somewhat like this:</p>
<pre><code class="language-yaml"># This is what defines the sources to scan for outstanding tasks.
sources:
- id: github-dmoonfire
type: github
token: TOKEN # Get from GitHub settings
- id: gitlab-dmoonfire
type: gitlab
token: TOKEN # Get from GitLab settings
# You can have multiple GitLab and GitHub accounts.
# This defines how the files are written out. There is only one choice and it doesn't have formatting.
outputs:
- type: markdown
id: md
path: /path/to/output
</code></pre>
<p>With the above file, I just have this added to my hourly cron:</p>
<pre><code class="language-shell">mfgames-tasks write path/to/mfgames-tasks.yaml
</code></pre>
<h1>Documentation</h1>
<p>Yeah, there isn't a lot of documentation here.</p>
<ul>
<li><a href="https://gitlab.com/dmoonfire/mfgames-tasks-js-cli">GitLab</a></li>
<li><a href="https://www.npmjs.com/package/mfgames-tasks-cli">NPM</a></li>
</ul>
<p>Now, to work on the 332 lines of items on my list.</p>
NaNoGenMo Retrospective2017-12-04T06:00:00Zhttps://d.moonfire.us/blog/2017/12/04/nanogenmo-retrospective/A retrospective of my month working on NaNoGenMo.
<p>Well, a month has gotten and I've finished this year's entry on time for <a href="https://github.com/NaNoGenMo/2017/issues/12">NaNoGenMo</a>. I managed to follow my original plan pretty closely, though I ended up using the contingency plan in case I ran out of time. This entry focused mainly on a “complicated” plot with various twists and interplay between characters (actors).</p>
<p>My <a href="https://gitlab.com/dmoonfire/girl-kills-dragon">repository</a> public for the entire thing and plan on keeping it open. Actually, this will probably end up the foundation for <em>next</em> year's entry when I focus less on plotting out a novel and more on writing the individual scenes and paragraphs.</p>
<h1>Overall Design</h1>
<p>The structure of the plot remained from the above link. The plot starts with a simple plot (something bad happens, hero goes to the villain, there is a battle, hero comes home). To make it a more interesting plot, I went through a series of iterations where I injected a plot element (a <em>node</em>) into the acyclic directed graph (DAG) where it fits. So, a simple trip to the villain has a few detours, there is a training montage, or a trial. The system also allows for new characters (<em>actors</em> because it is shorter) to be introduced on both sides, have adventures, and may be die.</p>
<p>The final result had 1,516 nodes with an additional 530 nodes of “back story” which I later intend to be fodder for the relationship scenes. It also created 187 actors at 392 locations across 14 countries. The final word count was 52,587 words.</p>
<p>The final result, as a single Markdown file, can be found in the <a href="https://gitlab.com/dmoonfire/girl-kills-dragon/blob/master/NaNoGenMo.md">repository</a>.</p>
<h1>Retrospective</h1>
<p>I thought it would be useful to mention things that worked and didn't work for me. There is no specific order to these, just interesting points.</p>
<h2>Worked: Using C#</h2>
<p>Even though I'm working on many Typescript projects, I decided to write this in C# which is currently my “best” language where I know all the tools, libraries, and how it works together. Not having to fight the tools made life a lot easier to code for an hour at a time (my lunch break) or steal a few hours while the kids take a bath.</p>
<p>It wasn't all “well known” though. This was my first real .NET Core language which had excellent support with Linux (my preferred platform at home) and Windows. With a set seeded random, it produced identical results on both platforms which made it a <em>lot</em> easier to debug some of the nastier bugs.</p>
<h2>Worked: Dependency Injection (DI) and Inversion of Control (IoC)</h2>
<p>I knew before I started that I was going to build as much as possible using <a href="https://en.wikipedia.org/wiki/Inversion_of_control">Inversion of Control</a>. I've toyed with it for a few projects but this was the first one that used DI/IoC from the beginning. I went with <a href="https://autofac.org/">Autofac</a> because it had .NET Core support. It worked out really well. It took a bit to wrap my mind around the disconnected nature of items but overall, it ended up being very smooth (though some of my constructors had a dozen parameters).</p>
<h2>Didn't Work: Not using IoC with Tests</h2>
<p>I typically switch to TDD when I'm solving problems. When I started, I was manually creating the various objects. As the primary code started getting more and more parameters in those constructors, I found myself writing default ones but it was hard to keep up. Things got easier to work after I spent two days setting up a test-specific IoC container.</p>
<p>This is something I should have done on the first week, not the third.</p>
<h2>Didn't Work: Public Properties</h2>
<p>When I first started using the IoC, I made most of the parameters public auto-properties. This worked fine until I had to make a change and realized that I had other classes using those properties instead of trusting the IoC to provide them directly. This coupling made life difficult which is why I ended up making most of the DI-injected properties private to that class. It also created a far more encapsulated logic, which was good.</p>
<p>There was one case where I did use property injection for a base class, but it was specific because <em>every</em> plot injection needed a dozen classes to function and I didn't want to mess with constructors that just passed it into the base class.</p>
<p>I could have created a <code>PlotInjectorContext</code> class, which is probably what I would have done in the future to avoid property injection (which made me feel dirty).</p>
<h2>Worked: Validation and Resolvers</h2>
<p>With the iterative approach, I encountered a few places where someone would leave in two different directions. These were some of the most complicated problems to solve (probably about a third of the month). I ended up creating various validations to test when the graph failed. Every time I encountered one, I would write a validation to help detect it.</p>
<p>The resolvers were ones that went through after all the plots were injected and figured out when everything happened, the names of people involved, etc. Having these as distinct classes that were injected by interface (<code>IPlotResolver</code>).</p>
<p>I handled resolution order by having a <code>ResolveOrder</code> property on each one. When one resolver (<code>SecondPlotResolver</code>) needed a first one (<code>FirstPlotResolver</code>), I would have the second one take the first as a constructor parameter and then use that to determine order.</p>
<pre><code>public int ResolveOrder => _firstPlotResolver.ResolveOrder + 1;
</code></pre>
<p>This ended up being a really nice pattern. If there were multiples, I would just use <code>Math.Max()</code> to figure out the order.</p>
<h2>Didn't Work: Post-Processing Validation and Resolution</h2>
<p>Validations were put into week one to run after all the plot injections. This worked pretty well, but I still struggled to find when something would go horribly wrong. This didn't get better until the fourth week when I split the validation and resolvers into ones that run at the end of the process and those which ran at the end of every injected plot.</p>
<p>Having the validation run at the end of each injection meant the system stopped <em>right</em> at the point that things became unstable instead of having me adjust the length to track it down. I also introduced a <code>plotId</code> which was an unused variable simply to let me put a breakpoint on the iteration to trace through the code.</p>
<p>The last major change was to have the timing resolution be able to sweep out all calculated times for the nodes and rebuild it. Once I had that, I was able to have a clean count of narrative nodes verses back story so 1,500 nodes wouldn't end up with 287 narrative (what I wrote) and the rest as back story.</p>
<p>The important lesson is to “fail fast” and move the validation as close as possible to where it can fail.</p>
<h2>Worked: ReSharper</h2>
<p>I can't describe how much I love <a href="http://www.jetbrains.com/resharper/">ReSharper</a>. It isn't available on Linux, so I used <a href="https://code.visualstudio.com/">Visual Studio Code</a> on my home machine. The refactoring and clean-up tools are sadly lacking outside of ReSharper, so I would throw crappy code wherever it fit while working at home, then the first thing I would do is clean up using ReSharper to ensure things were neat.</p>
<p>I would have used <a href="https://www.jetbrains.com/rider/">Rider</a> but I can't afford it and don't have any OSS project large enough to beg for a license.</p>
<h2>Didn't Work: Naming from Scratch</h2>
<p>I tried to come up with names by starting with a consonant and vowel inventory. That didn't work and I didn't have time to figure it out. In the end, I banged up a Markov chain generator and fed it English and Japanese names to come up with something.</p>
<h2>Didn't Work: Details</h2>
<p>I wanted to build up the specific details (like red hair or gender expression) that I could use in a madlib-style story generation. I think the idea has sound but it is <em>complicated</em>. I think I could spend an entire month (next year) working that out.</p>
<p>In the end, I just stopped with my third attempt at the system and just jammed the names into it. I'm not happy with <code>Detail</code> and <code>Noun</code> but it will take a lot more to get those workable. It just don't “feel” right to me.</p>
<h2>Worked: Refactoring</h2>
<p>My coworkers aren't fond of this, but I refactor <em>constantly</em> when I'm trying to solve problems. It like flipping one of those metal puzzles over trying to find a new way of getting it to work.</p>
<p>Having tools like ReSharper to reorganize the code was a godsend as I tried one thing and then another. Using different names, splitting code or combining helped with getting over the nastier bugs.</p>
<p><em>Though, Jetbrains, please, please make a command-line tool for apply code formatting. I would use it with every compile if I had it.</em> It would also be nice if there were other formatting tools I could have used in Linux but… no.</p>
<h2>Worked: Dumps</h2>
<p>Along the process, I ended up creating DGML, JSON, and Dot file outputs that I could look at the results at any point. This let me visually see how the scenes and chapters connected together.</p>
<p>Sadly, there is no good DGML visualization tools outside of Visual Studio. I'm surprised at this, but it was worth the difficulties of using the DGML viewer when I tried to arrange the chapters.</p>
<h1>Conclusion</h1>
<p>For my first NaNoGenMo, this was a lot of fun. The output is decades away from what I can write by hand, but for what I set out, I think I produced something I could be proud to show off.</p>
Changing Gears2017-08-07T05:00:00Zhttps://d.moonfire.us/blog/2017/08/07/changing-gears/Well, my vacation is all over and I had a blast. I was up in northern Wisconsin for two weeks with coming home in time to do an author event at the local mall. I got a lot done, but time to change gears and focus on commissions for a week or three.<p>Well, my vacation is all over and I had a blast. I was up in northern Wisconsin for two weeks with coming home in time to do an author event at the local mall. I got a lot done, but time to change gears and focus on commissions for a week or three.</p>
<p>I haven't done a “long” vacation in a long time and it was strange. However, with EDM going into school properly (first grade starts in a few weeks and they are seven next week), we can't do the 4-day weekends throughout the year as much. Not to mention, with the drive time, we really need to stay for a number of days to settle in and explore instead of drive up/stay for a few meals/drive home which has been our pattern previously.</p>
<p>It was also the longest either of the boys had been away from home. It was going pretty good up until day ten or so when we started getting a <em>lot</em> of meltdowns. That was one of the signs it was time to head home.</p>
<p>While on vacation, I got a number of things done. As usual, most of them were writing related but not all writing.</p>
<p>I caught up on edits from the writing group on <a href="https://fedran.com/flight-of-the-scions/">Flight of the Scions</a>. Overall, I think it's a pretty solid piece even though it is older than <a href="https://fedran.com/sand-and-blood/">Sand and Blood</a>, so I can see some places where I've improved over time. Since I've been working on this novel for over a decade, I don't think I'm capable of seeing which flaws are out there. I'm still looking for beta readers for this (it's gone through the group twice now). <a href="https://fedran.com/patrons/">Patrons</a> can read the entire thing as part of their subscription. Beta readers will get a free year as a patron in exchange.</p>
<p>I managed to write and rewrite five chapters of <a href="https://fedran.com/second-hand-dresses/">Second-Hand Dresses</a>, my Regency-inspired romance set in the same world as the rest of my work. I'm pretty happy with the results, though I have a bit of backfilling once I finish the novel. Everyone will see those over time, it is being shown as a weekly chapter with updates on my website (but probably not Wattpad).</p>
<p>I also wrote an <a href="https://atom.io/packages/autocorrect">autocorrect</a> plugin for Atom, my current writing platform. This handles things like “teh” being turned into “the” as I type along with double-capital words. Something minor, but it slows me down. Eventually, this will tie into <a href="/tags/author-intrusion/">Author Intrusion</a> so it can handle the accented names (such as “Rute” being expanded into “Rutejìmo” and “Jimo” being expanded into “Jìmo”). I'll also eventually use Atom's snippets to handle a list of characters in the novel, which would also be fun once I get there.</p>
<p>At the author signing on Saturday, I had a little snafu with my <em>Sand and Blood</em> books being in a different city but it was resolved pretty quickly. I sold a few things, got to talk with 15-20 other authors, and generally had fun networking and chatting for six hours.</p>
<p>I was reminded that I had to follow up with a few things related to <a href="http://iowa-icon.com/icon42/index.html">ICON</a> this year, mainly trying to get into readings and join a panel or two. We'll see if I did. At least I remembered to get the email in for the Barnes and Noble signing on the Thursday before. Like last year, I have a table so the rest of the <a href="https://broken.typewriter.press/">Broken Typewriter Press</a> authors and I will be there in addition to any panels we may be on.</p>
<p>In a fit of being overly complicated, I also redid the mailing list for Broken Typewriter Press with the intent of also doing one for Dusty Typewriter Press. I also shut down my forum website because no one was visiting and I'm going to switch that to an announcements-only mailing list. If you want, you can subscribe the BTP one on the bottom of the pages for that website.</p>
<p>Now that all that excitement is passed, I need to focus on commissions and obligations for the next few weeks. I originally said I was going to stop, but two factors came up: three people asked me to write four commissions in the same few weeks; I want to save up to get <em>Flight of the Scions</em> and <em>Second-Hand Dresses</em> properly edited and out the door.</p>
<p>If you like my writing, consider becoming a <a href="https://fedran.com/patrons/">patron</a>, reviewing my books, or encouraging me on social networks. Thank you.</p>
Changes to MfGames Culture2017-07-17T05:00:00Zhttps://d.moonfire.us/blog/2017/07/17/mfgames-culture/In the process of [updating my website](/blog/2017/07/16/website-improvements/), I ended up doing significant amount of work on my arbitrary culture library, [mfgames-culture](https://gitlab.com/mfgames-culture/).<p>In the process of <a href="/blog/2017/07/16/website-improvements/">updating my website</a>, I ended up doing significant amount of work on my arbitrary culture library, <a href="https://gitlab.com/mfgames-culture/">mfgames-culture</a>. This was used to let me enter the events and dates in a world-specific formats (such as ‘1454/7/41 MTR 17::61’) and convert them into other formats (including the ISO standard used by the timeline library, <a href="http://visjs.org/docs/timeline/">vis.js</a>).</p>
<h1>Why Arbitrary?</h1>
<p>I can't find the page where I talk about the reason <em>why</em> I'm creating this library, so I'll give a brief run down. In the process of creating world-specific calendars (such as <a href="/blog/2013/09/21/mansupi-tachira-ripochya-solar-calendar/">here</a> and <a href="/blog/2015/01/01/tarsan-standard-calendar/">here</a>), I don't base all the work off the Gregorian calendar. Instead, I built it up from the ground-up based on the culture, keeping in mind their politics, holy numbers, and other elements.</p>
<p>This creates a good flavor for the world but makes it difficult to create a good timeline for a novel or the world. Most organization programs don't work outside of Gregorian calendars or formats. That means I have a choice of ignoring my world-specific elements for tracking purposes or find some way of making those in-world calendars to work.</p>
<p>That is the crux of <em>mfgames-culture</em>, to create an API and tools for working with arbitrary cultures that are described purely with data files. Eventually, I'll have other versions of this but I'm using the <a href="https://gitlab.com/mfgames-culture/mfgames-culture-js/">Typescript</a> library as the reference implementation.</p>
<h1>Examples</h1>
<p>Below are some examples using a command-line program that uses the library. I also have an alias that uses my own culture information.</p>
<pre><code>$ sudo npm install --global mfgames-culture-cli
$ alias fedran="mfgames-culture --culture=kyo --data=$SRC/fedran-culture-data/dist"
</code></pre>
<h1>Parsing</h1>
<p>Right now, I'm only focusing on dates and times. There are two parts: the structure and the appearance. The structure is the components of the calendar, such as there are twelve months in a year, 28-31 days in a month, one hundred years is called a “century”. The appearance is how those components are formatted, such as the US using <code>MM/DD/YYYY</code> but the UK uses <code>DD/MM/YYYY</code>. It is the same calendar (structure) but different names for months.</p>
<p>The basic idea of the library is to take a <a href="https://gitlab.com/mfgames-culture/mfgames-culture-data/blob/master/data/gregorian.json">JSON</a> or YAML file that describes a calendar and provide an API for using it like many other calendar libraries, such as <a href="https://momentjs.com/">moment.js</a>. It doesn't many any assumptions of about how many months, days, or even seconds in a day. Instead, everything is calculated based on the file and it produces a rich object that contains that information.</p>
<pre><code class="language-shell">$ mfgames-culture parse 7/7/2017 --indent 2
{
"julian": 2457941.5,
"type": "instant",
"year": 2017,
"century": 20,
"millenniumCentury": 0,
"decade": 201,
"centuryDecade": 1,
"centuryYear": 17,
"decadeYear": 7,
"millennium": 2,
"yearDay": 187,
"yearMonth": 6,
"monthDay": 6,
"hour24": 0,
"meridiem": 0,
"hour12": 0,
"hourMinute": 0,
"minuteSecond": 0
}
$
</code></pre>
<p>Of course, I need to be able to do the same with my in-world calendar.</p>
<pre><code class="language-shell">$ fedran parse "1454/7/41 MTR 17::61" --no-json --yaml
julian: 2383794.0875
type: instant
mochyu: 1454
mochyuRote: 317
mochyuRichi: 6
richiRote: 40
koga: 16
mugi: 60
bidashu: 0
$
</code></pre>
<h1>Data Files</h1>
<p>In both examples, the data files are located somewhere the CLI can find it. In the first case, the files from <a href="https://gitlab.com/mfgames-culture/mfgames-culture-data">mfgames-culture-data</a> are packaged with the utility. For the Fedran data, you can find it <a href="https://gitlab.com/fedran/fedran-culture-data/">here</a>.</p>
<p>If you want to download them somewhere, they are just NPM packages.</p>
<pre><code class="language-shell">$ npm install mfgames-culture-data fedran-culture-data
$ ls node_modules/mfgames-culture-data/dist/
combined.json duodecimal-period.json gregorian.json
combined.min.json duodecimal-period.min.json gregorian.min.json
duodecimal.json en-US.json index.json
duodecimal.min.json en-US.min.json index.min.json
$ ls node_modules/fedran-culture-data/dist/
combined.json mifuno-kopachi-period.json
combined.min.json mifuno-kopachi-period.min.json
index.json natural.json
index.min.json natural.min.json
kyo.json seasons.json
kyo.min.json seasons.min.json
mansupi-tachira-ripochya.json tar.json
mansupi-tachira-ripochya.min.json tar.min.json
mifuno-kopachi.json tarsan-standard-calendar.json
mifuno-kopachi.min.json tarsan-standard-calendar.min.json
$
</code></pre>
<p>If you are curious why I have the data files as npm packages, I decided to make it easier to version them so the website won't break until I'm ready to upload it. Not to mention, <a href="http://semver.org/">semantic versions</a> is useful for more than libraries.</p>
<h1>Calendars</h1>
<p>I'm not going to go into too much directions of how to create the format. I have decent <a href="https://gitlab.com/mfgames-culture/mfgames-culture-data/blob/master/docs/index.md">documentation</a> on the purpose and setup.</p>
<p>The main part is a calendar consists of cycles. For example, the Gregorian is based on a year. A month is calculated from days within a year and decades are a formula based on decades. The system recursively goes through these cycles to figure out the individual amounts that you saw above.</p>
<h1>Cultures</h1>
<p>Cultures are where dates (instants) and periods (durations) are formatted. A culture has many ways of formatting any given instant, once you have one, you can pull it out in a variety of ways.</p>
<pre><code class="language-shell">$ mfgames-culture format "2017-07-14 23:19:04" --output-style markdown-table
| format | results |
| --------------------- | ---------------------------- |
| JD | 2457949.47157407407407407407 |
| M/D/YYYY | 7/14/2017 |
| M/D/YYYY h:mm tt | 7/14/2017 11:19 PM |
| MM/DD/YYYY | 07/14/2017 |
| MM/DD/YYYY h:mm:ss tt | 07/14/2017 11:19:04 PM |
| MMM DD, YY | Jul 14, 17 |
| YYYY-MM-DD | 2017-07-14 |
| YYYY-MM-DD HH:mm:ss | 2017-07-14 23:19:04 |
| default | 7/14/2017 11:19 PM |
| h:mm tt | 11:19 PM |
| shortDate | 7/14/2017 |
| shortDateTime | 7/14/2017 11:19 PM |
| shortTime | 11:19 PM |
$ mfgames-culture format "2017-07-14 23:19:04" --output-format shortDateTime
7/14/2017 11:19 PM
$
</code></pre>
<p>Again, this can be done with my fantasy calendar also.</p>
<pre><code class="language-shell">$ fedran format "1454/7/41 MTR 17::61" --output-style markdown-table
| format | results |
| ------------------- | ---------------------- |
| JD | 2383794.0875 |
| YYYY | 1454 |
| YYYY MTR | 1454 MTR |
| YYYY/OOO | 1454/318 |
| YYYY/OOO K:: | 1454/318 17:: |
| YYYY/OOO K::M | 1454/318 17::61 |
| YYYY/OOO K::M:B | 1454/318 17::61:1 |
| YYYY/OOO MTR | 1454/318 MTR |
| YYYY/OOO MTR K:: | 1454/318 MTR 17:: |
| YYYY/OOO MTR K::M | 1454/318 MTR 17::61 |
| YYYY/OOO MTR K::M:B | 1454/318 MTR 17::61:1 |
| YYYY/R | 1454/7 |
| YYYY/R MTR | 1454/7 MTR |
| YYYY/R/O | 1454/7/41 |
| YYYY/R/O K:: | 1454/7/41 17:: |
| YYYY/R/O K::M | 1454/7/41 17::61 |
| YYYY/R/O K::M:B | 1454/7/41 17::61:1 |
| YYYY/R/O MTR | 1454/7/41 MTR |
| YYYY/R/O MTR K:: | 1454/7/41 MTR 17:: |
| YYYY/R/O MTR K::M | 1454/7/41 MTR 17::61 |
| YYYY/R/O MTR K::M:B | 1454/7/41 MTR 17::61:1 |
| default | 1454/7/41 17::61 |
| shortDate | 1454/7/41 |
| shortDateTime | 1454/7/41 17::61 |
| shortTime | 17::61 |
$ fedran format "1454/7/41 MTR 17::61" --output-format shortDateTime
1454/7/41 17::61
$
</code></pre>
<h1>Typescript/Javascript</h1>
<p>These libraries can also be used directly. This is how I implemented the <a href="https://fedran.com/timeline/">timeline</a> on my website. It is also used by the build process to create the date/time tooltips like you can see on <a href="https://fedran.com/rutejimo/">Rutejìmo's</a> page.</p>
<p>The first part is to load a culture.</p>
<pre><code class="language-typescript">import * as mfgamesCulture from "mfgames-culture";
import * as mfgamesCultureData from "mfgames-culture-data";
// Create a data provider for the basic culture data. The entire JSON files is
// exported as `.combined` which is passed into the property provider.
var provider = new mfgamesCulture.PropertyCultureDataProvider(
mfgamesCultureData.combined
);
var cultureData = provider.getCultureSync("en-US");
var culture = new mfgamesCulture.Culture(cultureData);
</code></pre>
<p>(There is a promise version in <code>getCulturePromise</code>).</p>
<p>Once a culture is loaded, it can be used to create an in-memory representation of a format described in the culture file. A single point in time is called an <code>instant</code> and a range of time is called a <code>period</code>.</p>
<pre><code class="language-typescript">var instant = culture.parseInstant("2015-01-02 13:14:15");
console.log(instant);
</code></pre>
<p>This produces an output of:</p>
<pre><code class="language-json">{ julian:
{ [String: '2457025.0515625']
s: 1,
e: 6,
c: [ 2, 4, 5, 7, 0, 2, 5, 0, 5, 1, 5, 6, 2, 5 ],
constructor: { [Function: Big] DP: 20, RM: 1, E_NEG: -7, E_POS: 21 } },
type: 'instant',
year: 2015,
century: 20,
millenniumCentury: 0,
decade: 201,
centuryDecade: 1,
centuryYear: 15,
decadeYear: 5,
millennium: 2,
yearDay: 1,
yearMonth: 0,
monthDay: 1,
hour24: 13,
meridiem: 1,
hour12: 1,
hourMinute: 14,
minuteSecond: 15 }
</code></pre>
<p>It can also be used to format results.</p>
<pre><code class="language-typescript">var format = culture.formatInstant(instant, "YYYY-MM-DD HH:mm:ss");
console.log(format);
</code></pre>
<p>... which produces:</p>
<pre><code>2015-01-02 13:14:15
</code></pre>
<p>To get the timeline page, I basically parse the in-world dates, convert them to Julian Dates (using the <code>JD</code> format), then use that number to figure out the ISO date and time. Then I override the labels to take the Gregorian date and convert it back to the in-world dates so the reader never sees how I use Gregorian as the intermediary format.</p>
<p>This is why I also have every instant also figure out an effective Julian Date. That way, I can convert between any instant, even between multiple cultures.</p>
Fedran Website Improvements2017-07-16T05:00:00Zhttps://d.moonfire.us/blog/2017/07/16/website-improvements/Over the last week, I've focused on improving the [Fedran](https://fedran.com/) website. This took a fair amount of work but there is a lot more content and easier navigation as a result.<p>I wasn't really sure what to do for the week (I like week-based projects), so I put out a request on my social networks. By the end, I had an overwhelming suggestion to work on <a href="https://fedran.com/second-hand-dresses/">Second-Hand Dresses</a> but initial responses were to focus on improving my fantasy website, <a href="https://fedran.com/">Fedran</a>. Since I only asked an hour before my lunch break (the first time I work on personal projects), I decided to spend the week on the website and this week on <em>Second-Hand Dresses</em>.</p>
<p>A lot of the items on my to do list for the site were annoyances: getting it to look right on my phone, making it easier to read, and missing connections between elements. I also wanted to adding some advertising for my big projects (the <a href="https://fedran.com/rutejimo/">Rutejìmo</a> series right now).</p>
<h1>Appearances</h1>
<p>One of the first things I did was fix some of the little things that were bothering me about the website, links and appearances. Somehow, I managed to redo the colors and SASS stylesheets of the page giving a more varied appearance.</p>
<p><a href="./e5e41ea437eeb088.png" class="img-link"><img src="./e5e41ea437eeb088.png" alt="Screenshot of website" class="img img-responsive" width="256"/></a></p>
<p>The new colors were chosen from a fantastic website called <a href="http://paletton.com/#uid=72A0u0kllllaFw0g0qFqFg0w0aF">Paletton</a>. This lets you pick a color from a wheel and it gives you the triads or tetrads based on it. The color I picked ended up having a nice mixture and it quickly grew on me.</p>
<h1>POV and Volumes</h1>
<p>I also started to really embrace the POV and volume organization that I've talked about earlier. Most of the links that can be associated with a specific POV, volume, or both have a new appearance. The number is also the alias, so <a href="https://fedran.com/0100-02">https://fedran.com/0100-02</a> will send readers to <a href="https://fedran.com/sand-and-bone/">Sand and Bone</a>.</p>
<p>I also got the list of volumes on the proper character pages, so <a href="https://fedran.com/rutejimo/">Rutejìmo</a> will list every book or story that he's the main character.</p>
<h1>Journals of Fedran</h1>
<p>The first “big rock” of getting stuff done was integrating my <a href="https://fedran.com/journals-of-fedran/issue-00/">Journals of Fedran</a> with the website. <em>Journals</em> started off as an in-world newspaper but eventually morphed into a single-author collection of short stories, in-world essays, and poems.</p>
<p>I decided to make all of the stories and essays “first-class” entries on the website. This was because I already had already created top-level pages for every play, essay, and even person referenced by the epigraphs that I put on top of the chapters. If I was going to do that, why not the others.</p>
<p>Naturally, this led to creating a standard <a href="https://fedran.com/sand-and-bone/">landing</a> page for all those things which also reference the POV character, related pieces (some of them will have multiples later). This made a lot of broken links, but I think it is looking good as a framework. Everything is usable, I just haven't written up every person or point of view character that showed up in those short stories.</p>
<p>Each of those stories also show up on the front page, down in the “Sources” section. This is also a good way of encouraging myself since I can see a list of “what I've done”. Later, the front page will be only significant projects while the sources page will have the full list.</p>
<h1>Timelines</h1>
<p>The final major task was putting in a <a href="https://fedran.com/timeline/">timeline</a>. I've wanted this for some time now, mainly because I'm tagging the chapters of when they happened but I also want to show the overlapping stories that may or may not interact with each other.</p>
<p>This took a <em>lot</em> of work, another post-worth actually. In the end, I can annotate almost any character, story, or chapter with information about when they happened and it will show up on the timeline along with the ability to link back to the original.</p>
<p><a href="./timeline.png" class="img-link"><img src="./timeline.png" alt="Screenshot of the timeline" class="img img-responsive" width="256" /></a></p>
<p>I also have an info-box like entry on the individual pages, which then are scanned by the build process, inserted into the YAML header, then gathered by another process which renders the JSON file that runs the timeline. (Like I said, a lot of work.)</p>
<p><a href="./timeline-page.png" class="img-link"><img src="./timeline-page.png" alt="Screenshot of the timeline on page" class="img img-responsive" width="256" /></a></p>
<p>As time goes on, the timeline should get a lot more populated. I also found some typos with dates, which I'll fix as I go. Mostly, I think it's already cool to see how things relate to each other. The control will let me zoom and pan, which means the more complicated events will spread out as needed.</p>
<h1>What's Next</h1>
<p>Because of the overwhelming response, I'll be working on <em>Second-Hand Dresses</em> in the next week and see if I can get further along with that story.</p>
State of writing with Markdown, YAML, and Git 20172017-05-24T05:00:00Zhttps://d.moonfire.us/blog/2017/05/24/git-and-writing/A year ago, at one of my more successful panels at [WisCon](http://wiscon.info/), I was on a panel with [K. Tempest Bradford](https://tempest.fluidartist.com/find-a-tempest-wiscon-40/) and [Kristine Smith](http://www.kristine-smith.com/index.php/2016/05/home-5/) talking about writing processes. I got to see a lot of cool gadgets but I also got a chance to talk about my processes of writing with Markdown, YAML, and Git.<p>A year ago, at one of my more successful panels at <a href="http://wiscon.info/">WisCon</a>, I was on a panel with <a href="https://tempest.fluidartist.com/find-a-tempest-wiscon-40/">K. Tempest Bradford</a> and <a href="http://www.kristine-smith.com/index.php/2016/05/home-5/">Kristine Smith</a> talking about writing processes. I got to see a lot of cool gadgets but I also got a chance to talk about my processes of writing with Markdown, YAML, and Git.</p>
<p>I'm not going to WisCon this year, but I thought this would be a good opportunity to write up where I am using all those technologies to write, both for my personal projects and as a <a href="https://broken.typewriter.press/">publisher</a>.</p>
<p>This is a long post in a single part. If you've read my blog, there is some rehashed information.</p>
<h1>Markdown and YAML</h1>
<p>Probably the one part that hasn't changed is my use of <a href="https://daringfireball.net/projects/markdown/">Markdown</a> and <a href="http://yaml.org/">YAML</a>. I originally used Creole with a makeshift header but after playing with <a href="https://jekyllrb.com/">Jekyll</a> for a while, I jumped whole-heartedly on Markdown with a YAML header.</p>
<p>Below is an example of one of the files. I'll be referencing it a bit in this post so please forgive the length.</p>
<pre><code>---
availability: public
when: 1471/3/28 MTR
duration: 25 gm
date: 2012-02-18
title: Rutejìmo
locations:
primary:
- Shimusogo Valley
characters:
primary:
- Shimusogo Rutejìmo
secondary:
- Shimusogo Hyonèku
referenced:
- Funikogo Ganósho
- Shimusogo Gemènyo
- Shimusogo Chimípu
- Shimusogo Yutsupazéso
concepts:
referenced:
- The Wait in the Valleys
purpose:
- Introduce Rutejìmo
- Introduce Hyonèku
- Introduce naming conventions
- Introduce formality rules
- Introduce the basic rules of politeness
summary: >
Rutejìmo was on top of the clan's shrine roof trying to sneak in and steal his grandfather's ashes. It was a teenage game, but also one to prove that he was capable of becoming an adult. He ended up falling off the roof.
The shrine guard, Hyonèku, caught him before he hurt himself. After a few humiliating comments, he gave Rutejìmo a choice: tell the clan elder or tell his grandmother. Neither choice was good, but Rutejìmo decided to tell his grandmother.
---
> When a child is waiting to become an adult, they are subtly encouraged to prove themselves ready for the rites of passage. In public, however, they are to remain patient and respectful. --- Funikogo Ganóshyo, *The Wait in the Valleys*
Rutejìmo's heart slammed against his ribs as he held himself still. The cool desert wind blew across his face, teasing his short, dark hair. In the night, his brown skin was lost to the shadows, but he would be exposed if anyone shone a lantern toward the top of the small building. Fortunately, the shrine house was at the southern end of the Shimusogo Valley, the clan's ancestral home, and very few of the clan went there except for meetings and prayers.
</code></pre>
<p>I seem to be moving from the <code>.markdown</code> to <code>.md</code> extension. It's minor but I haven't quite jumped on it; I've only done it for the last two new projects.</p>
<h1>Atom</h1>
<p>My current editor of choice for novels is <a href="https://atom.io/">Atom</a> with a backup of Emacs. Unlike Emacs, Atom has standard keyboards and looks good with the fonts I use. It is a little slow, but for the most part, works pretty well for belting out words.</p>
<p>I'm writing some extensions for it including a modified spell check (<code>project-spell</code>) that can handle project-specific dictionaries (<code>language.json</code>). This lets me tell the spell-checker not to mark character names as misspelled.</p>
<p>Later, I'll integrate my work into Author Intrusion with this but that isn't nearly ready for prime time even for my purposes.</p>
<h1>Metadata</h1>
<p>Over the last year I noticed I am putting lot more information in the header than before. Some of it is done long after the project but even while writing, I'll put in notes about the characters, the reason I'm writing the chapter, time of day, even the outline which I remove as I write.</p>
<p>I just redid the <a href="https://fedran.com/">Fedran</a> website, otherwise I'd show you how the <code>characters</code> and related fields show up in the sidebar. I'll get that worked in the next few months, but it let me cross-link things into my wiki. Having it in the header also means I can query it to get a summary of the page using a TypeScript tool I wrote called <a href="https://www.npmjs.com/package/markdowny">markdowny</a>.</p>
<pre><code class="language-shell">$ markdowny table *.markdown -f _basename title when
| _basename | title | when |
| :------------------ | :---------------------- | :------------ |
| chapter-01.markdown | Rutejìmo | 1471/3/28 MTR |
| chapter-02.markdown | Confession | 1471/3/28 MTR |
| chapter-03.markdown | Morning | 1471/3/29 MTR |
| chapter-04.markdown | Rivals | 1471/3/29 MTR |
| chapter-05.markdown | Decisions | 1471/3/29 MTR |
$
</code></pre>
<p>Another nice thing about <code>markdowny</code> is that it also lets me show those YAML lists in a useful manner.</p>
<pre><code class="language-shell">$ markdowny table -f _basename characters.secondary
| _basename | characters.secondary
| :------------------ | :-------------------------------------------------
| chapter-01.markdown | Hyonèku
| chapter-02.markdown | Somiryòki, Tejíko, Gemènyo
| chapter-03.markdown | Desòchu, Gemènyo, Hyonèku, Mapábyo, Opōgyo, Panédo
| chapter-04.markdown | Desòchu, Karawàbi, Tsubàyo
$
</code></pre>
<p>I also has writing synopsis at the end, so what I can do is put each individual chapter in its header (in my case <code>summary</code>) and then use <code>markdowny</code> to pull them out. That way, I can write the synopsis as I go and not be overwhelemed at the end.</p>
<pre><code class="language-shell">$ markdowny sections *.markdown | head -n 5
# Rutejìmo
Rutejìmo was on top of the clan's shrine roof trying to sneak in and steal his grandfather's ashes. It was a teenage game, but also one to prove that he was capable of becoming an adult. He ended up falling off the roof.
The shrine guard, Hyonèku, caught him before he hurt himself. After a few humiliating comments, he gave Rutejìmo a choice: tell the clan elder or tell his grandmother. Neither choice was good, but Rutejìmo decided to tell his grandmother.
$
</code></pre>
<p>The final bit is word counting. Because of the YAML metadata, I can't get good word counts using most tools becaues the header skews the number. So I use the tool to get me counts of the content minus the YAML header.</p>
<pre><code class="language-shell">$ markdowny count *.markdown
chapter-01.markdown: 1520
chapter-02.markdown: 2144
chapter-03.markdown: 2905
chapter-04.markdown: 1173
chapter-05.markdown: 2570
</code></pre>
<p>I also have an alias for <code>mdwc</code> to the count since I use it pretty heavily.</p>
<h1>Directory Structure</h1>
<p>I'm leaning toward a semi-standard layout for my projects. Some of this was built up over the last few years but even my short stories have been migrating over to it.</p>
<ul>
<li><root>
* `README.md` contains a summary of the project and my tasks list.
* `chapters/` contains all the chapters in `chapter-01.md`. I have never had a project with more than 89 chapters, so I stick with two digit zero pad so it remains alphabetical.
* `frontmatter/` contains the frontmatter (dedication, legal) files.
* `backmatter/` contains the backmatter (colophon, about, also by).
* `characters/` has one Markdown file per character, for future use with [Author Intrusion](/tags/author-intrusion/).
* `covers/v1`: The first variant of the cover. I have a couple and each one goes into a different `vX` folder so I can keep them apart.
</li>
</ul>
<p>Even with short stories, I have a <code>chapters/</code> folder. I'm just picking up a “muscle memory” of going into chapters. I consider switching to putting things into a <code>src/</code> like most of my programming projects I just haven't because I couldn't see the advantage other than having a neat folder.</p>
<h1>Renumbering</h1>
<p>One drawback of having <code>chapter-01.md</code>, <code>chapter-02.md</code>, etc is when I have to add a new chapter. I wrote a <code>renumber</code> script that lets me inject a chapter.</p>
<pre><code class="language-shell">$ ls -l
total 8
-rw-r--r-- 1 dmoonfire dmoonfire 22 May 24 08:06 chapter-01.md
-rw-r--r-- 1 dmoonfire dmoonfire 127 May 24 08:06 chapter-02.md
$ touch chapter-01a.md
$ renumber *.md
$ ls -l
total 8
-rw-r--r-- 1 dmoonfire dmoonfire 0 May 24 08:07 chapter-01.md
-rw-r--r-- 1 dmoonfire dmoonfire 22 May 24 08:06 chapter-02.md
-rw-r--r-- 1 dmoonfire dmoonfire 127 May 24 08:06 chapter-03.md
$
</code></pre>
<p>As you can see, using <code>01a</code> puts it before the first chapter. It works the same with deleting chapters. Sadly, this script isn't very clean but eventually I'll merge it into <code>markdowny</code>.</p>
<h1>Git</h1>
<p>One of the biggest advantages of using Markdown (text files in general) is how well is plays with source control, Git in specific. I can't stress how much Git has helped me over the years. I still remember accidentally losing a chapter because I copied the wrong file in the wrong direction. Or the painful way of tracking versions (<code>chapter-01a.doc</code>, <code>chapter01b.doc</code>, <code>chapter-01b-final.doc</code>, etc.).</p>
<p>At one point, I had a single master repository which all completed pieces in the <code>master</code> branch and branches off that for the works-in-progress. That actually worked out pretty well… until I started publishing. While working with text is great, the binary files of working through the covers ended up creating huge repositories which took forever to clone.</p>
<p>Last year, I hadn't quite split apart all of my repositories but that's pretty much done now. This also meant I can give access to editors and others to a single novel without exposing the other ones. I don't have to worry about Git modules or jumping through hoops to have the binaries.</p>
<p>I tried LFS (large file system) for a while but then dropped it. With it not baked in, it was a little difficult to coordinate with continuous integration servers. I think in another year or so, it could work out fairly well. I also suspect it has to do with my comfort with LFS more than technical issues.</p>
<h1>Gitlab</h1>
<p>Related to LFS is where I'm hosting Git. A year ago, I used Git over SSH with my ISP. However, <a href="https://gitlab.com/">Gitlab</a> has been fantastic, both as their hosted environment and also on an instance running on Dreamhost (my provider). There are a lot of reasons to use Gitlab as a writer (and a publisher).</p>
<p>Probably the biggest is private repositories. Even if I didn't how it on my own site, being able to make each of the novels private is <em>fantastic</em>. Github, which is only slightly slicker, doesn't allow private repositories. Even if they did, I have over a hundred projects when you count works-in-progress, completed novels, and websites. Hosting that on Github would be expensive.</p>
<p>I also have a Gitlab instance. The <a href="https://broken.typewriter.press/">Broken Typewriter Press</a> business and my private novels on there. This gives me full control over the site (though I trust Gitlab) and I like it. The only people on the site are me, authors, and editors.</p>
<p>Like Github, Gitlab has some pretty nice features. When publishing the last few books, we've used the issue tab for authors to ask for me to order books or make corrections to their book. I can give them access to make their own typo corrections, which reduces the amount of work. At the same time, because of Git, I can be doing other changes and I don't have to worry about losing or screwing up their work. For the latest book, <em>Sins of Intent</em>, the author did a fantastic job of using the features.</p>
<p>I use the milestones for the various tasks of publishing a book. That way I can set up milestones for when the book has to be done, when the release it done, and various conventions. The issues are assigned to the milestones for both the author and myself and we can track what is missing or remaining to do. In effect, we can use a public project management to handle a book's release.</p>
<p>I normally turn off wiki and snippets though, they usually don't help.</p>
<p>One of the things with using Gitlab this way is that the author has full access to the raw files. Typically, I get a Word document and break it into individual Markdown files. Corrections are done against the Markdown as my baseline format for everything else. If they want to leave BTP or something catastropic happens, they can have exactly what I published. This is because I've been through cases where local edits were made but I couldn't get them back myself. In this case, they get and can see everything I do.</p>
<p>Which leads to the last feature of Gitlab which I use heavily, their Continious Integration service. This was difficult to set up at first, mainly because I had to educate myself. For most novels, the <code>.gitlab-ci.yml</code> looks like this:</p>
<pre><code>image: dmoonfire/mfgames-writing-js:0.4.0
stages:
- review
- publish
review:
stage: review
only:
- master
tags:
- docker
script:
- git lfs pull # Yeah, I'm using LFS with this project.
- npm install
- npm run build
artifacts:
expire_in: 1 week
paths:
- "*.pdf"
- "*.epub"
- "*.mobi"
publish:
stage: publish
script: "echo published"
when: manual
artifacts:
paths:
- "*.pdf"
- "*.epub"
- "*.mobi"
dependencies:
- review
</code></pre>
<p>Every time I check in, this rebuilds the project. For the little changes, the “staging” lets me and authors test the results of the file which show up underneath the build. They expire after a week but that's okay. “Releases” are done by moving a task to “publish”.</p>
<p>The nice part is I can grab the resulting PDF, MOBI, and PDF about 5-10 minutes after I check in. This is good to ensure that not only I have a reproducible build and also that I won't lost the ability to recover the output if my laptop catches on fire (it is almost seven years old, four majors cracks, and has a hernia).</p>
<p>This ability to see the final version is great because me or another author can make changes and see it, test it, and make sure it is exactly what they want. If I change formatting, I can see the results without overloading my computer.</p>
<h1>mfgames-writing-js</h1>
<p>Of course, using CI requires some way of actually formatting the books. Markdown is fantastic for some things but there are relatively few tools to format it into good-looking EPUB, MOBI, and PDF files.</p>
<p>Over the years, I have <em>many</em> variants of this. The original few were based on Makefiles and various Python or Perl tools. I've also integrated <a href="http://pandoc.org/">pandoc</a> into the mix. Nothing ever worked <em>quite</em> the way I wanted for what I considered to be a “proper” book. Files were put in the wrong place, dedications don't need titles, making sure chapters start on the right. There were little things that pushed me closer to making something more specific.</p>
<p>I also had a problem that extending the features for a later book broke the generation of older books. I had to start over or rebuild (usually copy/paste/edit). Now this is a problem that has been addressed by NuGet and NPM, having the build process inside the project instead of using shared programs and libraries.</p>
<p>Between this and using Gitlab CI runner, I decided to create a new framework that specifically was geared toward having specific versions that could be used to reproduce the book even if the underlying libraries were updated. I ended up using NPM, mainly because I'm learning TypeScript for the last few months. It also had a better story for installing (<code>npm install</code>), wasn't whitespace-based, and specifically designed to be isolated to the project.</p>
<p>The result is <a href="https://gitlab.com/mfgames-writing-js">mfgames-writing-js</a>, a framework for creating EPUB and PDF files from Markdown files. This entire thing is controlled by a single file checked into the Git repository.</p>
<pre><code>editions:
epub:
format: mfgames-writing-epub
theme: ./lib/efferding
outputDirectory: .
outputFilename: sins-of-intent-{{edition.version}}.epub
pdf:
format: mfgames-writing-weasyprint
theme: ./lib/efferding
outputDirectory: .
outputFilename: sins-of-intent-{{edition.version}}.pdf
isbn: 978-1-940509-24-2
metadata:
title: Sins of Intent
series: Cletus Efferding
author: Randy Roeder
language: en
contents:
- element: cover
source: covers/v2/front.jpg
linear: false
exclude:
editions: [pdf]
toc: true
- element: bastard
source: frontmatter/bastard.html
linear: false
exclude:
toc: true
- element: title
source: frontmatter/title.html
linear: false
exclude:
toc: true
- element: legal
source: frontmatter/legal.md
liquid: true
linear: false
exclude:
toc: true
- element: dedication
source: frontmatter/dedication.md
linear: false
exclude:
toc: true
- element: toc
linear: false
exclude:
editions: [pdf]
- element: chapter
number: 1
directory: chapters
source: /^chapter-\d+.md$/
start: true
page: 1
pipeline: &pipelines
- module: mfgames-writing-hyphen
exceptions:
# https://www.hyphenation24.com/word/driving/ says "driv-ing"
- driving
- drive
- element: acknowledgement
source: backmatter/acknowledgments.md
pipeline: *pipelines
</code></pre>
<p>This actually has turned out better than I expected. Yeah, I had trouble with using WeasyPrint for PDF generation, but it has almost all the features I needed to do right-side chapter openings, first page headers differently, and stitching everything into a single PDF with properly embedded fonts.</p>
<p>The one thing I <em>can't</em> do is create Smashword's Microsoft Word. I'm still trying to decide if it is worth it.</p>
<h1>Website</h1>
<p>I also release chapters every week. I used to use WordPress but it got cumbersome to add new chapters so I switched to a static site generator over the years. I started with DocPad, then Jekyll, and then extended Jekyll with Python and Perl programs to handle my requirements.</p>
<p>Well… I decided to write my own. This uses the same input files as the rest of my system (Markdown + YAML). It also lets me change a header to release the chapter and include it into the build.</p>
<p>One of the complexities I had to figure out is the different repository for each novel. In the building process, I actually clone every project I'm publishing or have published, then use scripts to pull them into a single website before generating it. Jekyll couldn't handle it easily which is one reason I created my own.</p>
<p>The tool for making the website is <a href="https://gitlab.com/cobblestone-js/">CobblestoneJS</a> but it has no documentation and I'm still fumbling through it. On the other hand, it handles the ten or so repositories needed to build <a href="https://fedran.com/">fedran.com</a> without my conlang, world data, and individual novels.</p>
<p>As part of the weekly release, I've gotten the tasks down to:</p>
<ul>
<li>Update a header in the YAML (or use the scheduler to do it once)</li>
<li>Write a blog post about the chapter</li>
<li>Copy the chapter and the post over to <a href="https://ello.co/dmoonfire/">Ello</a></li>
<li>Copy the chapter from fedran.com to Wattpad</li>
<li>Copy the post over to Patreon</li>
<li>Run the diaspora sync post</li>
</ul>
<p>It takes about 1-2 hours which is much shorter than the 4 is used to be.</p>
<h1>Summary</h1>
<p>Well, there it is, my current state of writing with Git. You have everything from writing the chapters, supplying metadata, how to store it, various tools for getting through the publication process, and even formatting it for the various vendors.</p>
<p>I've gotten lost trying to automate and simplify a lot of this, but I think the current state is mostly usable by others and has been pretty solid for my own needs. I'm sure I'll improve and expand on it.</p>
<p>If you have question or comments, please ask. I love talking about processes, looking for improvements, or explaining in more detail why I do these things.</p>
<p>Thank you.</p>
Author Intrusion status for May 1, 20172017-05-01T05:00:00Zhttps://d.moonfire.us/blog/2017/05/01/author-intrusion/After a rough start of the previous week, I got a chance to really focus on [Author Intrusion](/tags/author-intrusion/) (AI). Despite the rather significant amount of check-ins and coding, I wasn't quite able to get to a really good "show off" point. Instead, this week ended up being a [black triangle](http://rampantgames.com/blog/?p=7745) (significant progress but nothing visible).<p>After a rough start of the previous week, I got a chance to really focus on <a href="/tags/author-intrusion/">Author Intrusion</a> (AI). Despite the rather significant amount of check-ins and coding, I wasn't quite able to get to a really good “show off” point. Instead, this week ended up being a <a href="http://rampantgames.com/blog/?p=7745">black triangle</a> (significant progress but nothing visible).</p>
<h1>Introduction</h1>
<p>I haven't really talked about AI for a while. The entire idea started in 2009 or so when I realized I had some major blindspots in my writing. At the time, it was an overuse of gerunds that was hanging over me but as my skill improves, the problems areas also shift around. My <a href="http://noblepencr.org/">writing group</a> has been a great help in tracking these down but I felt a lot of them were things that could be detected ahead of time; basically something that would let me file off the rough edges before someone spent the effort to correct them.</p>
<p>To my surprise, the current offering of grammar checkers doesn't actually look for the same thing. They seem to review things sentence-by-sentence but I wanted something that looked at the entire document and looked for trouble areas. For example, using the same word repeatedly in a few paragraphs (echo words).</p>
<p>At the same time, I'm a programmer. I use a wonderful program called <a href="https://www.jetbrains.com/resharper/">ReSharper</a> which has a lot of refactoring and analysis. There are times when I wanted to see every time a character shows up (name outside of dialog) or is talked about. I also rename characters (frequently) and suffer from the occasional search-and-replace bugs.</p>
<p>That all lead to me want to create an IDE for writers, basically a Visual Studio and ReSharper that was geared toward authors.</p>
<p>One of my major influences is a short story by James White called <a href="http://www.sectorgeneral.com/shortstories/fasttrip.html">Fast Trip</a>. I love that story because it comes down to “change the environment to succeed.” Well, that meant a couple of things for AI, it had to be flexible to work the way the <em>author</em> wants and it needed to use the paradigm that I was comfortable with.</p>
<h1>Iterations</h1>
<p>I've worked on off-and-on for about eight years. I've tried a lot of iterations, gotten to a point, and then hit some conceptal problem that threw everything into disarray. Some of them were understandings on my part (you can't reformat text while writing), getting caught working on the wrong thing (I should reinvent the word processor), and a ton of other dead ends.</p>
<p>This means that it isn't done until it is stable. So if you are not interested in alpha software, this might not be what you are looking for at this point.</p>
<p>The current design goals is to create a separate, independent program that can be called by text editors. Inspired by <a href="http://www.omnisharp.net/">OmniSharp</a>, I figured it was better to create a program that did one thing well (analysis and refactor novels) and then create hooks that can talk to other editors like Emacs and <a href="https://atom.io/">Atom</a>.</p>
<p>I'm also using <a href="https://gruntjs.com/">Grunt</a> and <a href="https://www.npmjs.com/">NPM</a> from my own experiences. Some of the earlier implementations of AI were self-contained with all the plugins installed there. However, as I've been working on my <a href="https://gitlab.com/mfgames-writing-js/">Javascript publishing framework</a>, versioning is very important. I can't break the analysis of novels I wrote ten years ago just because the code has improved to handle the novels of today. NPM has a great way of handling that with the <code>package.json</code> file that lists specific packages and versions, which can be installed and used. So, AI is done the same way, the system uses <em>specific</em> versions of packages to ensure that it always produces the same data. If you upgrade the underlying packages, then you can deal with the changes and check in the results knowing it won't change tomorrow.</p>
<p>I've written AI in a number of languages (Javascript, Typescript, C#, Pythong) along with different UIs. I've gotten lost in a lot of them. At the moment, my skills in C# are considerably more advanced than the others, but more importantly, the tools (ReSharper) make me a lot more effective. I don't want this to be a learning experience at this point because I have a lot of novels I need to write and I have to focus on either making fun tools (AI) or finish books.</p>
<p>I don't want this to be an argument of language. I like C# and I'm good at it, so I'm using it.</p>
<h1>Overall Status</h1>
<p>It isn't there yet. I'm working for an end-to-end which means setting up the files and being able to run a program (<code>aicli</code>) and have it produce an output that shows echo words (the first analysis I want to write and my current blindspot). What does work requires a relatively specific setup (not that bad, everything is contained inside the <a href="https://gitlab.com/author-intrusion/author-intrusion-cil/">repository</a>) and a bit of hand-holding.</p>
<p>This is still in the “black triangle” point since the technical framework is set in place, I'm still combining them to produce something that looks cool.</p>
<p>All of the examples are based on a subset of <em>Sand and Blood</em> used for testing. It can be found <a href="https://gitlab.com/author-intrusion/author-intrusion-cil/tree/drem-0.0.0/Examples/Sand%20and%20Blood">here</a>.</p>
<h1>Configuration</h1>
<p>Like Grunt, there is a single file that controls how AI works with a project. I took inspiration from <a href="https://code.visualstudio.com/">Visual Studio Code</a> and Atom in that every file isn't explictly added to the project. Instead, AI assumes that all files underneath the directory containing the <code>author-intrusion.yaml</code> file is part of the project. If you add a chapter, the YAML file won't change but the system will pick it up.</p>
<p>I used YAML because it is easy to use, doesn't require a lot of noise, and pretty much handles the relatively simplistic data. Also, unlike JSON, YAML can let you copy a section from one place to another which I consider to be pretty useful.</p>
<p>I am trying to avoid changes a lot of files whenever you add a chapter, that's just noise in most cases. Related to that, AI will create a <code>.ai</code> directory for its internal caches, that shouldn't be checked into source control at all.</p>
<p>Eventually, <code>aicli</code> will search for <code>author-intrusion.yaml</code>. That way, like Grunt, it can be called anywhere inside the directory tree and it will “do the right thing” to produce consistent results. This is the same thing <code>git</code> does among other CLI implementations.</p>
<p>The basic <code>author-intrusion.yaml</code> file is pretty simple:</p>
<pre><code class="language-yaml">file:
- plugin: AddClassFromPath
match: chapters/*.markdown
class: chapter
data:
- plugin: AddClassFromData
select: file.chapter
class: pov-{pointOfView}
layout:
- plugin: SplitLines
- plugin: SplitParagraphs
- plugin: OpenNlpSplitTokens
select: para
- plugin: WordTokenClassifier
analysis:
- plugin: Echoes
select: file.chapter token.word
scope: token.word:within(10)
threshold: { error: 5, warning: 2 }
</code></pre>
<p>In the example file from the link above, there is a lot more in the file either for documentation purposes or just to work out some concept or idea.</p>
<p>There are four sections: <code>file</code>, <code>data</code>, <code>layout</code>, and <code>analysis</code>. These are various operations that are performed on each file to classify (<code>file</code> and <code>data</code>), organize (<code>layout</code>), and analyze (<code>analysis</code>) the files.</p>
<h1>Plugins</h1>
<p>Most of the processes are based on the idea of a “plugin”. A plugin is a discrete class (typically in its own <a href="https://www.nuget.org/">NuGet</a> package) that is versioned and specific to the current project. They are identified by the <code>plugin</code> property inside the list. A plugin can be used more than once.</p>
<pre><code class="language-yaml">analysis:
- plugin: Echoes
id: Words Within 10
select: file.chapter token.word
scope: token.word:within(10)
threshold: { error: 5, warning: 2 }
- plugin: Echoes
id: Sounds Alike
select: file.chapter token.word
scope: token.word:within(5)
compare: :root:attr(soundex)
threshold: { error: 5, warning: 2 }
</code></pre>
<p>In each case, the <code>plugin</code> property of each plugin identifies which plugin to use. This is usually the base class but it means I'll have to have a registry to list which plugins do what. The reason for doing this is because if someone else writes a plugin (extends AI), they can push it up to NuGet (about that later) and everyone can use it. It doesn't require a release of the main code (assuming <code>AuthorIntrusion.Contracts</code> remains the same).</p>
<p>The rest of the properties are based on that plugin. There is a bit of interesting complexity in making this work (e.g., hack but I wrote it) but everything is type safe when it comes to coding.</p>
<h1>Selectors</h1>
<p>In the above examples, <code>select</code> and <code>scope</code> are based on CSS selectors. I couldn't find a library that did it generically, so I wrote one that just creates an abstract syntax tree of CSS selectors which is used by this library. Like most of the plugins, I'll break it into a separate project once AI gets stable.</p>
<p>The advantage of using CSS, such as <code>file.chapter token.word</code> or <code>:root</code> is that many developers understand how CSS selectors works. Except for the specifics (pseudo-classes for example), how they chain together, how you combine them, those are known enough that I don't have to force someone to learn something new.</p>
<p>I'm setting up the layout of a file (using the <code>layout</code> plugins) to look like HTML. Those plugins create the tag/element structure that CSS uses.</p>
<p>Right now, the following are implemented:</p>
<ul>
<li><code>:first-child</code></li>
<li><code>:last-child</code></li>
<li><code>:nth-child</code></li>
<li><code>:root</code> is the top-level item being selected (usually the <code>file</code> but can be something else)</li>
</ul>
<p>There is also scoped pseudo-classes. These are used to get elements based on their relationship to another element. In every case so far, the <code>scope</code> property is based on each element found in the <code>select</code> of the plugin.</p>
<ul>
<li><code>:before(x)</code> finds the elements before the scope. It can be used like <code>token.word:before(5)</code> to get the five words before the current one.</li>
<li><code>:after(x)</code></li>
<li><code>:within(x)</code> which is basically <code>:before(x)</code> combined with <code>:after(x)</code>.</li>
<li><code>:self-</code> versions of the three above.</li>
<li><code>:parent</code> matches an item that is the parent of the scope.</li>
</ul>
<p>These scoped variables are used so we can control what we are looking for. For example, the echoes plugin may look to see if the same word has been used somewhere within ten words of the current one:</p>
<pre><code class="language-yaml">analysis:
- plugin: Echoes
select: token.word
scope: token.word:within(10)
</code></pre>
<p>It can also be used to make sure the same word isn't used at the beginning of the surrounding paragraphs:</p>
<pre><code class="language-yaml">analysis:
- plugin: Echoes
select: para token.word:first-child
scope: para:within(1) token.word:first-child
</code></pre>
<p>There is a third category used for comparisons (used by the Echoes). These are where I hack the CSS system but basically let me define operations.</p>
<ul>
<li><code>:lower</code> get a lowercase version of the text.</li>
<li><code>:attr(attribute-name)</code> uses the value of the attribute instead of the text.</li>
</ul>
<p>This lets us analyze for something more than just the text. For example, words that sound the same:</p>
<pre><code class="language-yaml">analysis:
- plugin: Echoes
select: token.word
scope: token.word:within(10)
compare: :root:attr(soundex) # :attr() isn't done yet
</code></pre>
<p>Or ones that have the same base word:</p>
<pre><code class="language-yaml">analysis:
- plugin: Echoes
select: token.word
scope: token.word:within(10)
compare: :root:attr(stem)
</code></pre>
<p>A third example is if we are looking for too many sentences that start with the same pattern of parts of speech. This is to find where “bob did this”, “bob did that”, “mary did something”.</p>
<pre><code class="language-yaml">layout:
- plugin: ClassifyElementRange # Not done yet
select: sent token.word:first-child
scope: sent:parent token.word:self-or-after(2)
tag: sent-leading
analysis:
- plugin: Echoes
select: sent-leading
scope: sent-leading:within(5)
compare: :root:attr(pos)
</code></pre>
<h1>File Plugins</h1>
<p>Now, how all those topics are used. The first set of plugins are simply based on the structure of the project. Every file has one automatic element, the <code>file</code> which is the <code>html</code> of the file. We can add classes and identifiers to that to limit how the rest of the code works.</p>
<p>The biggest one is AddClassFromPath. You can use that to add the <code>chapter</code> class to files in the <code>chapters</code> directory. That way, the later selectors can use <code>file.chapter</code> or <code>.chapter</code> to limit processing (don't need to do grammar on your notes).</p>
<pre><code class="language-yaml">file:
- plugin: AddClassFromPath
match: chapters/*.markdown
class: chapter
</code></pre>
<h1>Data Plugins</h1>
<p>I write using Markdown with a YAML header. My chapters look like this:</p>
<pre><code>---
availability: public
when: 1471/3/28 MTR
duration: 25 gm
date: 2012-02-18
title: Rutejìmo
locations:
primary:
- Shimusogo Valley
characters:
primary:
- Shimusogo Rutejìmo
secondary:
- Shimusogo Hyonèku
referenced:
- Funikogo Ganósho
- Shimusogo Gemènyo
- Shimusogo Chimípu
- Shimusogo Yutsupazéso
concepts:
referenced:
- The Wait in the Valleys
purpose:
- Introduce Rutejìmo
- Introduce Hyonèku
- Introduce naming conventions
- Introduce formality rules
- Introduce the basic rules of politeness
summary: >
Rutejìmo was on top of the clan's shrine roof trying to sneak in and steal his grandfather's ashes. It was a teenage game, but also one to prove that he was capable of becoming an adult. He ended up falling off the roof.
The shrine guard, Hyonèku, caught him before he hurt himself. After a few humiliating comments, he gave Rutejìmo a choice: tell the clan elder or tell his grandmother. Neither choice was good, but Rutejìmo decided to tell his grandmother.
---
> When a child is waiting to become an adult, they are subtly encouraged to prove themselves ready for the rites of passage. In public, however, they are to remain patient and respectful. --- Funikogo Ganóshyo, *The Wait in the Valleys*
Rutejìmo's heart slammed against his ribs as he held himself still. The cool desert wind blew across his face, teasing his short, dark hair. In the night, his brown skin was lost to the shadows, but he would be exposed if anyone shone a lantern toward the top of the small building. Fortunately, the shrine house was at the southern end of the Shimusogo Valley, the clan's ancestral home, and very few of the clan went there except for meetings and prayers.
</code></pre>
<p>Given that, I would use a plugin to let me add identifiers and classes to files based on the YAML header (the bit between the <code>---</code> lines). For example, the following would add a <code>pov-ShimusogoRutejìmo</code> class to the file.</p>
<pre><code class="language-yaml">data:
- plugin: AddClassFromData # Not done
select: file.chapter
class: pov-{characters.primary}
</code></pre>
<p>There will be ways of manipulating it, but basically it lets you tag the chapter with “scene” or “sequel” if you follow the <a href="http://amzn.to/2pBEypw">Techniques of the Selling Writer</a> or want to identify if a chapter is combat or talking. It doesn't matter how you want to tag it, you can use any element in the data to filter or make decisions later. This will also be used for the querying to let you say “what chapters are from Rutejìmo's point of view” or “what scenes happen at night”. Eventually, I'll tie it into my culture library so you can also say “show me the chapters in chronological order”.</p>
<h1>Layout Plugins</h1>
<p>The bulk of my effort in the last two weeks has been in the layout. This is what carves up the contents of the file into something that can be selected via the CSS system. The default is only to have the <code>file</code> element which contains everything.</p>
<pre><code class="language-yaml">layout:
- plugin: SplitLines
tag: line # implied
</code></pre>
<p>The above splits everything into <code><line></code> elements. We need that for reporting line number errrors. Below, we have a plugin that splits things into paragraphs based on Markdown rules (a blank line separates a paragraph). Using the above examples, this is what lets us use the <code>para .word:first-child</code> for selectors.</p>
<pre><code class="language-yaml">layout:
- plugin: SplitParagraphs
tag: para # implied
</code></pre>
<p>It gets complicated when we start adding tokens. A token is a word or puncutation. I'm using <a href="https://opennlp.apache.org/">OpenNLP</a> to break apart the words at the moment. This splits up the contents of the paragraph (because of the <code>select: para</code>) into tokens.</p>
<pre><code class="language-yaml">layout:
- plugin: OpenNlpSplitTokens
select: para
</code></pre>
<p>Once I have tokens, I can add the <code>word</code> class to the actual words.</p>
<pre><code class="language-yaml">layout:
- plugin: WordTokenClassifier
class: word # implied
</code></pre>
<p>Eventually, there will be a <code>OpenNlpSplitSentences</code> plugin where you would split the paragraph into sentences and then split the sentences into tokens. That isn't done but we don't need it for the base. I will also eventually create a <code>ParagraphSentenceWord</code> plugin that does most of this and makes it easier to add it as a single line.</p>
<p>The end result is that we have an abstract tree that represents the file. Eventually there may be a lot more such as dialog identification, blockquotes and epigraphs, and whatever else makes sense.</p>
<p>The main reason for the layout plugins is to make the selectors work for the analysis. It is also a complex bit of code that has to run in order unlike analysis which can be multi-threaded for a given file.</p>
<h1>Analysis Plugins</h1>
<p>Finally, the last bit. I'm not done with anything else, but the analysis plugin is the bit that finds errors and warnings. It is the parts that “does” something. In other words, the entire reason I'm writing this.</p>
<pre><code class="language-yaml">analysis:
- plugin: Echoes
select: file.chapter token.word
scope: token.word:within(10)
threshold: { error: 5, warning: 2 }
</code></pre>
<p>I'm working on this next, but I hope to have <code>aicli check</code> produce an error like <code>gcc</code> or a complier that gives a specific file and line number (via the <code>SplitLines</code> plugin) that lists the errors. It will be the same thing Atom uses to highlight problems.</p>
<p>Since analysis plugins only add/remove errors, they don't change the structure. This means they can use all the CPUs using worker processes to make it as efficient as possible. I'm also going to eventually have some optimizations put in (most elements are hashed so they can be cached).</p>
<h1>Status and Plans</h1>
<p>This last week, I've gotten the folllowing:</p>
<ul>
<li>CSS selectors are working.</li>
<li>CSS pseudo-classes working for scope to get ranges of words.</li>
<li>CSS for doing transformations to make text lowercase before comparison.</li>
<li>The basic layout plugins needed to break a file into lines, paragraphs, tokens, and words.</li>
<li>A search path for plugins can find assemblies in a different directory.</li>
<li>The starting code for using NuGet for packages is there.</li>
<li>Echoes can find the words, just not report them.</li>
</ul>
<p>The goal is to get a simple echo words analysis done, basically looking for duplicates within ten words.</p>
<ul>
<li>Get <code>TextQuery</code> to compare text between the select and the scope.</li>
<li>Add warnings and errors (notices) to elements.</li>
<li>Report the notices to the console for the <code>aicli check</code> command.</li>
</ul>
<p>I really want to get this to the point it can be used. I think it will be benefical even with the basics for finding problem spots with my personal projects and can benefit others. I also think that once I get the end-to-end, additional functionality will be easily added for other needs.</p>
<h1>Development</h1>
<p>Development is currently on the <a href="https://gitlab.com/author-intrusion/author-intrusion-cil">drem-0.0.0</a> branch on Gitlab. I'm adding issue of secondary items while tagging them with expected complexity. While I don't expect anyone to contribute, if someone does get inspired the <code>simple</code> and <code>trivial</code> items are good starting points.</p>
<p>I plan on adding notes for contributing “soon”.</p>
<p>So far, everything is MIT licensed but there will be other licenses already involved. OpenNLP is Apache licensed so I'll need to figure out how that interacts with MIT. Once I split the packages into individual files, it will probably be easier but for the time being, life is faster if I keep it as a single repository.</p>
Federated Opinions2017-04-26T05:00:00Zhttps://d.moonfire.us/blog/2017/04/26/federated-opinions/I'm been thinking about trust and relationships on the Internet, specifically social networking and news. Here is one possible approach to handle it.<p>I'm been thinking about trust and relationships on the Internet, specifically social networking and news. Here is one possible approach to handle it.</p>
<h1>Impetus</h1>
<p>In the last few days, I briefly got into a discussion about Twitter's poor management of death threats verses how they handle swearing to verified accounts or banning for using specific posts. It was frustrating for folks because they felt that a tweet that said “I hope you die in a fire” somehow didn't violate community standards but a relatively mild hash tag did.</p>
<p>At the same time, there was a couple flame wars on my feed (he said, she said) where there was blocking left and right as people picked sides. There were a couple of egg accounts (people starting up anonymous accounts to attack others) involved.</p>
<p>This was all against a backdrop of fake news sites that influenced the recent elections and the general attack against science.</p>
<h1>Problem</h1>
<p>The problem comes in that Twitter, Facebook, and most social networks aren't capable of policing their networks. Twitter has <a href="http://www.internetlivestats.com/twitter-statistics/">six thousand tweets per second</a>) and Facebook is close to a <a href="https://zephoria.com/top-15-valuable-facebook-statistics/">quarter million per second</a>. Assuming that the conflict rate (the number of people who disagree or agree passionately with each other) is relatively constant, that is a <em>lot</em> of reported posts.</p>
<p>This also means that when a human is reviewing it, they are dealing with a deluge of data. It doesn't matter if it is a social network reported post, Amazon reviewing books being published, or even a slush pile reader, going through so many posts of vile hatred, disgust, and terrible things will break anyone. That is one reason why <a href="https://www.wired.com/2014/10/content-moderation/">content moderation</a> is a difficult job that inflict PTSD on even the most stalwart of individual.</p>
<p>Also because humans are so adaptable, it is difficult to automate flagging content in a consistent manner. That means that one person may not think “die in a fire” is offensive while another (after spending an hour looking at car accidents) thinks it nothing compare to death while a third is offended and will mark it as such. Regardless of how the item is marked, there is a high probability of it being escalated to a “manager” which is just another person inundated with too much information.</p>
<h1>Basic Idea</h1>
<p>I've mulled over this idea a few times over the last year, but I was mowing yesterday and got to thinking about it. The general idea is that I think crowd-sourcing “community standards” might be the approach but as I learned “common sense” doesn't exist. You have individuals who are opposite ends of the same idea (regardless of who is wrong) and they don't like when the “other side” is in charge of content (usually the words “fake news” or “censorship” shows up when this happens).</p>
<p>Everyone thinks their beliefs are “common sense” and “community standard”. In reality, there really isn't such a thing. Seeing the hatred on the network points that out, there are simply too many people who think that sending death threats (I've gotten my share) is perfectly “acceptable.”</p>
<h1>Web of Trust</h1>
<p>I think this can be done by establishing a <a href="https://en.wikipedia.org/wiki/Web_of_trust">web of trust</a> between individuals. There are a few components to the web that is needed for this to work.</p>
<p>The first is a <em>relationship</em> between an individual and a target (which is an individual's presence). This is an opinion between two people. So, if Alice blocks Gary, then is represented by a negative relationship (<code>Alice - Gary</code>). If Alice vouches for Mary, then it would be a positive relationship (<code>Alice + Mark</code>). In this case Alice, Mary, and Gary could be social accounts, Twitter handles, or Facebook identifiers.</p>
<p>A relationship would determine someone is blocked by having a negative score.</p>
<p>The second is a <em>connection</em>. A connection is a relationship between two individuals where the first person trusts the second person's opinions. For example, if Steve feels that Alice fits his community standard, he “trusts” her (<code>Steve < Alice</code>). This can also build up chains, such as <code>Mark < Steve < Alice</code>. It is a web because an individual may have multiple connections, such as <code>Mark < Steve, Mark < Jenny</code>.</p>
<p>The direct relationships are pretty easy. Alice doesn't like Gary (<code>Alice - Gary</code>). The difficulty comes into passing that relationship down. This leads to the third concept, the <em>threshold</em>. A threshold is relatively simple but there is both a <em>positive threshold</em> and a <em>negative threshold</em>. A positive threshold is the point where the sum of all connections's opinion about an individual becomes a positive rating. The negative is when the sum is lower and becomes a negative opinion.</p>
<p>This might need a few examples. Say, Steve says “if anyone hates someone, then I hate them”, this is a negative threshold of -1. So when <code>Alice - Gary</code> is established, then the total that Steve sees is -1 and then they <em>also</em> create a negative relationship with Gary (<code>Steve ?- Gary</code>). I'm using the “?” to say it it comes from the trust instead of explicitly being set.</p>
<p>On the other hand, say Mark requires two trusted connections. When Steve established a negative opinion of Gary, then Mark saw a total of -1 for Gary. However, he requires -2 so he doesn't establish a negative (blocking) connection to Gary. Now, if Jenny also had a negative opinion (<code>Jenny - Gary</code>), then Mark would see -2 and would then create their own opinion of Gary (<code>Mark ?- Gary</code>).</p>
<p><em>We could also create a warning threshold, but the idea is pretty much the same. Likewise, we could also establish a reverse connection (whatever Alice likes, Gary hates or <code>Gary > Alice</code>).</em></p>
<p>Say Jenny doesn't have an opinion of Gary (get rid of the <code>Jenny - Gary</code>) but Mark also trusts Alice's opinion (<code>Mark < Alice, Mark < Steve, Mark < Jenny</code>). When Alice established a negative relationship with Gary, it fed into Steve's (as above) and also Mark's. Mark then sees -2, one from Alice and one from Steve. This would then create that negative relationship.</p>
<p>The advantage of this system is that someone can have a low negative threshold when they are having trouble with being overwhelmed or a high negative threshold which requires more people to have an opinion (if everyone at the table disagrees…). It could also be dynamic based on their current situation.</p>
<h1>Federation</h1>
<p>One of the biggest complexities with this is how to communicate these relationships. I don't trust a single site, organization, or company to manage these relationships. Sooner or later, they will have to figure out some way to pay the bills and that means using the data for advertising or income.</p>
<p>Multiple providers, on the other hand, helps reduce that since individuals don't have to deal with as much network, process, or requirements. It is easier to spread the costs among others.</p>
<p>I wasn't sure <em>how</em> to make this happen until recently. In the last month, <a href="https://mastodon.social/">Mastodon</a> has gotten a bit of press. The idea behind it is a good one, a federated social network. It isn't an original idea, though, but they have a pretty solid implementation of the GNU Social network.</p>
<p>We could use the same concept for sharing and communicating. The federation and protocol would work out nicely using existing and tested protocols. It also means we can use relatively simple text messages to communicate the various relationships. At the same time, “following” an account would be the same as establishing a trust relationship.</p>
<p>So, the <code>Alice - Gary</code> would actually be <code>alice@some.network</code>.</p>
<h1>Identifiers</h1>
<p>The target of a relationship (the <code>Gary</code> in <code>Alice - Gary</code>) doesn't have to be an account. Instead, it is just an arbitrary identifier. There is already a pretty <a href="https://en.wikipedia.org/wiki/Uniform_Resource_Identifier">good standard</a> for that. In effect, we can use the URL of an account as the target with some standard rules of how to parse specifics URLs (trailing slashes, capitalization).</p>
<ul>
<li><a href="https://twitter.com/dmoonfire">https://twitter.com/dmoonfire</a></li>
<li><a href="https://d.moonfire.us/">https://d.moonfire.us/</a></li>
<li><a href="https://ello.co/dmoonfire/">https://ello.co/dmoonfire/</a></li>
</ul>
<p>Of course, these can be really long which causes a problem. Fortunately, we can easily hash this result (MD5) to reduce it to a consistent length and then encoding it into hex. We don't have to worry about duplicate hash entries since we are dealing with URIs usually don't have the “room” to create a duplicate hash.</p>
<p>When we do this, we get:</p>
<ul>
<li>1dca72d547e37c33cbdbdeeecd3d6119 <a href="https://twitter.com/dmoonfire">https://twitter.com/dmoonfire</a></li>
<li>c877a19c5bdb62a2e3e946411fb75dd5 <a href="https://d.moonfire.us/">https://d.moonfire.us/</a></li>
<li>1a7e49dc5912887d23a727c3bd933750 <a href="https://ello.co/dmoonfire/">https://ello.co/dmoonfire/</a></li>
</ul>
<h1>Equality</h1>
<p>Now there are cases where two identifiers/targets are the same one. This can be done the same thing, with a trusted network identifying an equality or inequality.</p>
<ul>
<li>My two URLs are the same: <code>1dca72d547e37c33cbdbdeeecd3d6119 = c877a19c5bdb62a2e3e946411fb75dd5</code></li>
<li>My two URLs aren't the same: <code>c877a19c5bdb62a2e3e946411fb75dd5 != 1a7e49dc5912887d23a727c3bd933750</code></li>
</ul>
<p>We could say an <em>equality threshold</em> and <em>inequality threshold</em> to then pass it through the network like relationships above.</p>
<h1>Operations</h1>
<p>All of this can be done with messages posted to the federation through a GNU Social based network. Other instances can listen to the operations and make changes on the individual accounts.</p>
<p>For example:</p>
<ol>
<li><code>steve@home.social</code> follows <code>alice@some.network</code></li>
<li><code>alice@some.network</code> posts <code>ADD 1a7e49dc5912887d23a727c3bd933750</code></li>
<li>The message is passed from <code>some.network</code> to <code>home.social</code></li>
<li><code>steve@home.social</code> gets the message and updates their record.</li>
</ol>
<p>As I see it, there are only a few operations:</p>
<ul>
<li><code>ADD</code> to create a positive relationship</li>
<li><code>SUB</code> to create a negative one</li>
<li><code>EQ</code> for establish an equality</li>
<li><code>NE</code> for establishing an inequality</li>
</ul>
<h1>Commands</h1>
<p>There are also a few commands to retrieve data. The only one I can see is the need to retrieve the current list.</p>
<ol>
<li><code>steve@home.social</code> follows <code>alice@some.network</code></li>
<li><code>steve@home.social</code> sends a message to <code>alice@some.network</code>: <code>GET</code>.</li>
<li><code>alice@some.network</code> responds with a temporary URL with a full list.</li>
</ol>
<h1>Privacy</h1>
<p>Now, I could easily see there a privacy problem. Even with encoding, it would be easy enough to look for <code>SUB 1a7e49dc5912887d23a727c3bd933750</code> as someone who dislikes me. The accounts don't give <em>too</em> much privacy because they have to be posted in a semi-public location.</p>
<p>To make it hard to detect these, accounts can establish an AES encryption key that all messages are encrypted. For simple ofuscation, the key can be posted in the public profile of that account. Anyone subscribing can decrypt it, but it would take significant effort to scan the network to find specific operations since every account would require a different encryption key.</p>
<p>Or, an account to have an encryption key set but share it out-of-band for private federation.</p>
<h1>Consolidation</h1>
<p>The end result is we would have an account on a federated instance. The account would have the number of private variables:</p>
<ul>
<li>positive threshold: 1</li>
<li>negative threshold: 1</li>
<li>equality threshold: 1</li>
<li>inequality threshold: 1</li>
<li>private key: binary</li>
</ul>
<p>There would also be a flag if the private key is exposed to users.</p>
<p>It would have the relationships with their optional encryption keys.</p>
<ul>
<li><code>alice@some.network encryptionKey</code></li>
</ul>
<p>It would have the relationships that the user specifically set on their own (they trump everything).</p>
<ul>
<li><code>ADD 1dca72d547e37c33cbdbdeeecd3d6119</code></li>
<li><code>EQ c877a19c5bdb62a2e3e946411fb75dd5 1a7e49dc5912887d23a727c3bd933750</code></li>
</ul>
<p>We also keep track of the relationships we got through our connections.</p>
<ul>
<li><code>alice@some.network ADD c877a19c5bdb62a2e3e946411fb75dd5</code></li>
<li><code>alice@some.network ADD 1a7e49dc5912887d23a727c3bd933750</code></li>
</ul>
<p>These would be combined into a single unified list (assuming thresholds of 1 for this example).</p>
<ul>
<li><code>ADD 1dca72d547e37c33cbdbdeeecd3d6119</code> (maybe a score of 100)</li>
<li><code>SUB c877a19c5bdb62a2e3e946411fb75dd5</code> (score 1)</li>
<li><code>SUB 1a7e49dc5912887d23a727c3bd933750</code> (score 1)</li>
</ul>
<p>As operations are federated, they would filter through the various inputs and produce the final consolidate list above.</p>
<p>Overall, these are very simple database structures.</p>
<h1>REST</h1>
<p>The final step for this is how to use it. With the above example, it could be a very simple REST request to determine the status of any relationship. It would require an application key (something that Mastodon already has so easily done) but would just request a HTTP code or status.</p>
<p><code>http://social.network/steve/1dca72d547e37c33cbdbdeeecd3d6119</code></p>
<p>The output would be a relatively simple JSON response that gives the final state (<code>ADD</code> or <code>SUB</code>) but maybe allow for the equivalents to also be retrieved. I'm not sure what is needed.</p>
<p>Instead of just returning <code>ADD</code> and <code>SUB</code> from the REST, it could return the actual amount with the user's explicit setting. That would allow for hidden, obscured, warning, or a “verified” check mark of sorts.</p>
<h1>Potential Usages</h1>
<p>Here is how I could see it being used. A user script (TamperMonkey) could easily be developed that uses an application key and URL to request the block status for any identifier. It could then look at the page as it loads, construct a URL, encode it into MD5, and then use REST to get the status and determine if the specific elements of the page needed to be hidden, highlighted, or otherwise indicated of a positive or negative result.</p>
<p>Assuming the results were cached reasonably, it could remain relatively performant. Likewise, we could use a POST operation that retrieves multiple records at once to avoid network overhead.</p>
<p>Websites and applications could use the REST directly to do the same thing. That way, sites like Ello or Mastodon could integrate the calls into their own retrieval so they don't ever show up on the user's website.</p>
<p>It could also be used to identify the reliability of websites (it's just a URL) with a TamperMonkey script putting a banner up (this is fake news!)</p>
<h1>Possible Expansion</h1>
<p>I could also see hashtags used to identify specifics or communicate additional ideas. For example, <code>SUB 1a7e49dc5912887d23a727c3bd933750 #klingon</code> to indicate negative opinions about Klingons. That is a more complicated step and I think the rest needs to settle beyond one night's writeup.</p>
<h1>Conclusion</h1>
<p>Well, that's my idea along with rough implementation ideas of how to create a network of trust and distrust between people that would be relatively organic and doesn't rely on a single company to ensure it works. It is based on existing libraries, frameworks, and infrastructure which means I think it is reasonable to implement.</p>
<p>Opinions are always appreciated.</p>
<h1>Edit 1: Modifying Results</h1>
<p>The same tools could also be used to create the direct relationships. So a POST that adds the “ADD” or “SUB” above based on a TamperMonkey-added button next to an account entry. Likewise, websites to pass the like/dislike through a REST POST call for the same effect. That way, there is almost no friction for contributing to the federation.</p>
The Overhead of Blogging2017-03-02T06:00:00Zhttps://d.moonfire.us/blog/2017/03/02/overhead-of-blogging/I got lost with a tangent where I try to create simple images for blog posts and ended up spending two days buildng a generic system.<p>After nearly burning myself out to finish 10k words before the end of the month, I decided to switch back over to working on my <a href="https://broken.typewriter.press/">publishing business</a> and getting ready to release my <a href="https://fedran.com/sand-and-blood/">own books</a> but also the next book I'm publishing.</p>
<p>With some recent realizations, I found out my current incarnation of generating EPUB, MOBI, and PDF files was fragile. In specific, it required <em>my</em> laptop to be set up properly. If I lost it—not an unreasonable suggestion given that it has a hernia and a cracked shell—then I would be significantly impacted.</p>
<p>To fix that, I decided to rewrite the system (again) using TypeScript and NodeJS. This time, the packages are pushed up to shared servers which means others can contribute (yeah, right) but I also can rebuild the entire environment without having my specific machine.</p>
<p>This also means that I can set up my <a href="https://gitlab.com/">GitLab</a> installation to automatically build the EPUB, MOBI, and PDF files whenever I check files in. This also means the authors who have access to their own books can also make small corrections and see the results. In the end, it <em>should</em> reduce the work it takes for me to publish which helps more than just myself.</p>
<p>To simplify the logic, I'm building everything with HTML (EPUB's native format and MOBI's second language). However, there is not a good <a href="https://www.sharelatex.com/learn/XeLaTeX">XeLaTeX</a> conversion from HTML so I decided to try out <a href="http://wkhtmltopdf.org/">wkhtmltopdf</a> which I use for BTP's monthly statements. That worked out pretty well… up to the point I found out I couldn't do inner and outer margins.</p>
<p>Those are kind of important for books.</p>
<p>When one thing creates a block, try a different one. After a while, I found a nice library called <a href="http://weasyprint.org/">WeasyPrint</a>. This handled inner and outer margins but… it couldn't handle recto chapter pages (chapters opening on the right side) nor could it handle chapter-specific leading pages (no headers).</p>
<p>So I have two ways of getting past this. I figured I could either hack the code by assembling the book and doing a bit of magic with the WeasyPrint code or do the “right” thing and teach that system how to handle recto and page numbers. I don't know which one is “easier” but I'm pretty sure there is a good chance I won't have time, but I'm willing to attempt it.</p>
<p>I can always go back to XeLaTeX, I know how to convert that relatively quickly.</p>
<p>Which… leads me to my <a href="https://xkcd.com/974/">974 Problem</a>. I wanted to do a quick blog post about the stuff above. However, I didn't want to spent twenty minutes looking for an image to tag the post. My alternative was to create a “generic” image for posts that I didn't find an image. My novel posts will still probably have appropriate images.</p>
<p>Instead of spending twenty minutes for an image, I spent two days writing a generic system including properly documented and tested library to integrate into my build system. Because ten hours verses twenty minutes is somehow better when I get lost in the forest.</p>
Gulping down websites2016-11-04T05:00:00Zhttps://d.moonfire.us/blog/2016/11/04/website-changes/I've spent the last few days reworking the processes used to generate this website.<p>For the last few years, I've been using a Jekyll-based website. I started with a mostly stock setup, but it wasn't long before I was supplementing the results with Perl, Python, and a fairly complicated Makefile. This worked (and sucked out many hours out of my life) and I was pretty happy.</p>
<p>As I've been posting repeatedly, the entire process has been getting slower and slower. The Jekyll version was taking ten to fifteen minute, which was frustrating when I'm trying to write a post (no preview).</p>
<p>I also started doing serials. The weekly nature of posting could really use some scheduling. That way if I have a few hours to work ahead of time, I could write up the posts and get them queued up. Yes, I'm lazy but I also know that I've had more than a few postings minutes why of midnight.</p>
<p>This year, I'm also planning on doing <a href="http://www.frathwiki.com/Lexember">Lexember</a>, a conlang-a-day challenge for December. This could <em>really</em> use scheduled posts.</p>
<p>Over the last week, I've created a Gulp-based system that creates the entire website. This will make it easier to work with Gitlab's CI service, which I trigger with a cron job. It also runs a lot faster.</p>
<p>The <em>really</em> hard part is to migrate the <a href="https://fedran.com/">Fedran</a> site over to this system because I have a <em>lot</em> of logic, but that is an adventure for another year.</p>
<p>Eventually, I'll actually write up the twenty Gulp plugins I've created to make this. Mainly, I just wanted to test the CI and scheduled posts.</p>
Writing with Markdown and YAML2016-10-31T05:00:00Zhttps://d.moonfire.us/blog/2016/10/31/writing-with-markdown/I realized that it's been a while since I wrote up how I write with Markdown and YAML. This is the current state of my technique of writing.<p>Four years ago, I wrote a <a href="https://www.reddit.com/r/writing/comments/un2x9/how_plain_text_editors_became_my_writing_tool_of/">Reddit post</a> about why I switched to pure text files instead of using Microsoft Word, Libreoffice, or even Google Docs. The short answer is: text is forever and it is <em>really</em> hard to corrupt a text document.</p>
<p>There was a secondary reason for using text format and that was because I like keeping track of metadata while I write. A lot of authors do, actually, but I didn't care for the more common methods of keeping a spreadsheet in sync with the writing or getting <a href="https://www.literatureandlatte.com/scrivener.php">Scrivner</a> to behave. At the time, it wasn't so great, from what I've seen, a lot of what I do here can also be done in that program. That said, I still don't use it because of their file format doesn't play well with Git (I wrote a <a href="/blog/2015/05/09/gitlab-projects/">post about that</a> last year).</p>
<p>Now, I have been doing this for a while. However, last year, someone at WisCon asked me about it and I <em>thought</em> I made the post but apparently I didn't. Today, someone asked me the same question ("how do you use Markdown and YAML to write?") and I went to point them to my article I didn't write last year and realized that I didn't write it.</p>
<p>So… here it is.</p>
<h1>Format</h1>
<p>When I write, the basic file looks like this:</p>
<pre><code>---
title: Rutejìmo
---
> When a child is waiting to become an adult, they are subtly encouraged
> to prove themselves ready for the rites of passage. In public, however,
> they are to remain patient and respectful. --- Funikogo Ganósho, *The
> Wait in the Valleys*
Rutejìmo's heart slammed against his ribs as he held himself still. The
cool desert wind blew across his face, teasing his short, dark hair. In the
night, his brown skin was lost to the shadows, but he would be exposed if
anyone shone a lantern toward the top of the small building. Fortunately,
the shrine house was at the southern end of the Shimusogo Valley, the
clan's ancestral home, and very few of the clan went there except for
meetings and prayers.
</code></pre>
<p>The <code>></code> is a block quote which I use for my epigraphs. There is a quote at the beginning of every chapter for world building. Everything after that is <a href="https://daringfireball.net/projects/markdown/">Markdown</a>. I use <code>*italics*</code> and <code>**bold**</code> but otherwise not a lot.</p>
<h1>Front Matter</h1>
<p>The really important part is the <a href="http://yaml.org/">YAML</a> front matter. I picked YAML because it was really easy to read and change. It also parsed very well by computers. The only thing I don't like is that it doesn't like tabs… but can't be perfect.</p>
<p>There is a lot more in this header than I showed above. For the first chapter of <a href="https://fedran.com/sand-and-blood/">Sand and Blood</a>, the front matter looks like this:</p>
<pre><code>---
availability: public
when: 1471/3/28 MTR
duration: 25 gm
title: Rutejìmo
locations:
primary:
- Shimusogo Valley
characters:
primary:
- Shimusogo Rutejìmo
secondary:
- Shimusogo Hyonèku
purpose:
- Introduce Rutejìmo
- Introduce Hyonèku
- Introduce naming conventions
- Introduce formality rules
summary: >
Rutejìmo was on top of the clan's shrine roof trying to sneak in and
steal his grandfather's ashes. It was a teenage game, but also one to
prove that he was capable of becoming an adult. He ended up falling off
the roof.
The shrine guard, Hyonèku, caught him before he hurt himself. After a
few humiliating comments, he gave Rutejìmo a choice: tell the clan elder
or tell his grandmother. Neither choice was good, but Rutejìmo decided to
tell his grandmother.
---
</code></pre>
<p>As you might be able to tell, I keep a fair amount of information in the top of my files. This basically contains references to characters, locations, why the chapter exists. I could also put tone (action scene, introspection), scene types (scene or sequel), or anything else I want.</p>
<p>The nice reason for this is when I rename or arrange the file, the header goes with it. If I need to list all chapters that have a character references, I can with a simple “find in files”.</p>
<p>I also have it tied into a future plan of creating a master book that has every novel I have with the chapters interspersed so the entire world is chronological. I think it will be cool once I start overlapping these stories. I also plan on having locations so I can show them on the map, if I ever write it down.</p>
<h1>Tools</h1>
<p>Now, “find in files” can only go so far. Over the years, I've written a number of tools to help me work with these type of files. I've tried it out in <a href="https://github.com/dmoonfire/mfgames-writing-python">Python</a>, <a href="https://github.com/dmoonfire/mfgames-writing-perl">Perl</a>, <a href="https://github.com/dmoonfire/mfgames-writing-cil">C#</a>. Most of them have bits and pieces that work, I actually use a combination of all of them.</p>
<p>Today, I realized I was tired of working with tools from three different languages so I decided to add a fourth language that works with some future plans I have: Typescript. So, over my lunch and while kids were showing up the door for treating, I got the first version of <a href="https://git.mfgames.com/author-intrusion/markdowny">Markdowny</a> written.</p>
<pre><code>npm install --global markdowny
</code></pre>
<p><code>markdowny</code> basically duplicates the most common functionality I use from the other languages and puts it into a fairly generic framework that can show off the usefulness of using YAML.</p>
<h1>Counting</h1>
<p>The major drawback of YAML headers is that it is hard to figure out how many words something is. The built-in word count features of most editors counts the large YAML. So, <code>markdowny</code> has a simple feature for giving me a chapter-by-chapter breakdown of word lengths without the headers.</p>
<pre><code>markdowny count chapters/*.markdown --total
</code></pre>
<p>The output is pretty simple actually. I mostly use this to select the chapters I want to submit to the writing group; we have a four thousand word limit with submissions.</p>
<pre><code>chapter-01.markdown: 1,520
chapter-02.markdown: 2,144
chapter-03.markdown: 2,908
chapter-04.markdown: 1,173
chapter-05.markdown: 2,570
Totals : 10,315
</code></pre>
<p>If you don't include the <code>--total</code> parameter, you don't get the Totals line.</p>
<p>It is also useful for other submissions or finding out how big the novel is.</p>
<pre><code>markdowny count chapters/*.markdown --total | tail -n 1
</code></pre>
<h1>Synopsis</h1>
<p>The <code>summary</code> key is a useful one. I write the chapter synopses as I go. It isn't nearly as overwhelming then trying to write the entire book in a single shot. That means when I have to come up with a chapter-by-chapter synopsis, I can do it fairly easily.</p>
<pre><code>markdowny sections chapters/*.markdown > synopsis.markdown
</code></pre>
<p>The above command line goes through all of the chapter files, pulls out the title and summary block and creates a new Markdown file that contains all the summaries.</p>
<pre><code># Rutejìmo
Rutejìmo was on top of the clan's shrine roof trying to sneak in and
steal his grandfather's ashes. It was a teenage game, but also one to
prove that he was capable of becoming an adult. He ended up falling off
the roof.
The shrine guard, Hyonèku, caught him before he hurt himself. After a
few humiliating comments, he gave Rutejìmo a choice: tell the clan elder
or tell his grandmother. Neither choice was good, but Rutejìmo decided
to tell his grandmother.
# Confession
Rutejìmo returned to his grandmother's cave. He tried to sneak around,
but she caught him entering. When she asked what he had done, he told
her that he had been up at the shrine. His grandmother got upset and
began to berate and beat him, chasing him out of the cave.
Gemènyo interrupted her beating to ask if it was justified. When Tejíko
explained the reason, he agreed but she had lost her anger. As she went
back into the cave, Gemènyo sat down with Rutejìmo and asked
questions. The discussion came to the upcoming rite of passage and how
Rutejìmo would gain magical powers once he experienced it. Gemènyo
wouldn't give details about the rite, but he did tell Rutejìmo that
magic wouldn't change him.
Rutejìmo snapped back, saying that Gemènyo wasn't the greatest warrior
in the clan. Gemènyo didn't seem bothered, but suggested that if
Rutejìmo wanted to be the best, he needed to do something. Rutejìmo
agreed, even knowing he wouldn't.
# Morning
Rutejìmo was sent out to get breakfast for his grandparents. Along the
way to the valley floor, he got distracted by Mapábyo, Hyonèku's adopted
daughter, riding on top of Opōgyo, one of the clan's mechanical dogs
used for dragging heavy loads around. They talk for a short while before
she asked him to come with her to take the load up to her father, who is
at the lookout above the valley entrance.
Even though Hyonèku had caught him the night before, Rutejìmo
agreed. When they get there, Hyonèku started to read a naughty letter
from his wife before Gemènyo interrupted him. Embarrassed, Hyonèku
focused on his daughter while Gemènyo talked with Rutejìmo more.
</code></pre>
<p>A little bit of effort ahead of time and I have a chapter-by-chapter breakdown without getting overwhelmed.</p>
<h1>Table</h1>
<p>The final thing <code>markdowny</code> does is give me that summary table. I use this to compare chapters with each other and try to get an idea of what is going on with the “big picture”.</p>
<pre><code>markdowny table chapters/*.markdown markdowny table -f _basename title _words characters.secondary -t File Title Words:rs "Secondary Characters"
</code></pre>
<p>The <code>-f</code> selects the fields inside the YAML. The <code>-t</code> lets me have a pretty name for them. The results happen to be a Markdown table, so it looks a bit prettier if I just show the end results.</p>
<table>
<thead>
<tr>
<th style="text-align: left;">Filename</th>
<th style="text-align: left;">Title</th>
<th style="text-align: left;">When</th>
<th style="text-align: left;">Duration</th>
<th style="text-align: left;">Location</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">chapter-01.markdown</td>
<td style="text-align: left;">Rutejìmo</td>
<td style="text-align: left;">1471/3/28 MTR</td>
<td style="text-align: left;">25 gm</td>
<td style="text-align: left;">Shimusogo Valley</td>
</tr>
<tr>
<td style="text-align: left;">chapter-02.markdown</td>
<td style="text-align: left;">Confession</td>
<td style="text-align: left;">1471/3/28 MTR</td>
<td style="text-align: left;">35 gm</td>
<td style="text-align: left;">Shimusogo Valley</td>
</tr>
<tr>
<td style="text-align: left;">chapter-03.markdown</td>
<td style="text-align: left;">Morning</td>
<td style="text-align: left;">1471/3/29 MTR</td>
<td style="text-align: left;">2 m</td>
<td style="text-align: left;">Shimusogo Valley</td>
</tr>
<tr>
<td style="text-align: left;">chapter-04.markdown</td>
<td style="text-align: left;">Rivals</td>
<td style="text-align: left;">1471/3/29 MTR</td>
<td style="text-align: left;">1 m</td>
<td style="text-align: left;">Shimusogo Valley</td>
</tr>
<tr>
<td style="text-align: left;">chapter-05.markdown</td>
<td style="text-align: left;">Decisions</td>
<td style="text-align: left;">1471/3/29 MTR</td>
<td style="text-align: left;">3 m</td>
<td style="text-align: left;">Shimusogo Valley</td>
</tr>
</tbody>
</table>
<p>Okay, the format is based on the browser, but overall it is somewhat useful. Of course, the early chapters of that book is in the same location. Likewise, I could show the POV (which is useful for some writers) but it doesn't help much since I write single POV novels.</p>
<pre><code>markdowny table chapters/*.markdown -f title _words:r characters.primary purpose
</code></pre>
<p>This gives me a decent summary of how long each chapter is, who the main character is, and a list of reasons for that chapter. From there, I have a decent idea of where I need to trim or expand the piece.</p>
<table>
<thead>
<tr>
<th style="text-align: left;">title</th>
<th style="text-align: right;">_words</th>
<th style="text-align: left;">characters.primary</th>
<th style="text-align: left;">purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">Rutejìmo</td>
<td style="text-align: right;">1520</td>
<td style="text-align: left;">Shimusogo Rutejìmo</td>
<td style="text-align: left;">Introduce Rutejìmo, Introduce Hyonèku, Introduce naming conventions, Introduce formality rules</td>
</tr>
<tr>
<td style="text-align: left;">Confession</td>
<td style="text-align: right;">2144</td>
<td style="text-align: left;">Shimusogo Rutejìmo</td>
<td style="text-align: left;">Introduce Tejíko, Introduce Gemènyo, Introduce Rutejìmo lack of talent, Introduce style of punishment, Second chapter to get used to names</td>
</tr>
</tbody>
</table>
<h1>Other Tools</h1>
<p>I have some other tools. There are ones that take these files and create EPUB, MOBI, and PDF files. I've been using those programs since <em>Sand and Blood</em> was first published. There are others that I've started wht convert the in-world dates (MTR in the above example) into a master timeline or identify every chapter of every novel that happen in a given location.</p>
<p>I also use these chapters to build up the various pages on my <a href="https://fedran.com/">Fedran</a> website so <a href="https://fedran.com/patrons/">patrons</a> can read ahead (that is the <code>availability</code> in the YAML).</p>
<h1>Using Markdown</h1>
<p>Using Markdown isn't fancy but I find it is easier. Since there is so little control over format, it is easier to focus on the words that need to be written instead of making it pretty.</p>
<p>Also, it is a text file. It plays <em>very</em> well with Git and source control. I don't have to worry about an editor corrupting the file and losing it, it takes up little space, and I can easily switch between Notepad++, Emacs, or Atom without a second thought. I can use the best tool for the job.</p>
Well, that was anti-climatic2015-12-11T06:00:00Zhttps://d.moonfire.us/blog/2015/12/11/dates/After working on a project for a long time, sometimes it is anti-climatic when it finally gets used.<p>In the process of updating my website, I encountered an interesting little problem: I have to coordinate the dates between two novel series. <a href="https://fedran.com/rutej%C3%ACmo/">Rutejìmo</a>'s third book is about the same time as <a href="https://fedran.com/kan%C3%A9ko/">Kanéko</a>'s fourth book. To add complexity to the issue, she is from a much different part of the world. Naturally, I couldn't do something simple like having the entire world use a single unified calendar and time system. No, I <em>had</em> to make each culture evolve its own way of handling things like that which means those two series don't exactly mesh together.</p>
<p>There were a bunch of ways of handling it, but in the end I decided to actually <em>use</em> the library and environment that I spent the last month working on and merged everything together for my <a href="https://fedran.com/">world website</a>.</p>
<p>It also gave me an excuse to create the data files for my two big calendars: <a href="/blog/2015/01/01/tarsan-standard-calendar/">Tarsan Standard Calendar</a> and <a href="/blog/2013/09/21/mansupi-tachira-ripochya-solar-calendar/">Mansupi Tachira Ripochya</a>. This is something that was on my to do list and this was a (fun) excuse to actually work on it. Actually, much of what I do on that site is pretty much me having fun putting everything together and make a good-looking site that showcases the breadth of my writing.</p>
<p>It took about six hours of working, but the end result is:</p>
<img class="img-responsive post-img-link" src="hover.gif" />
<p>When someone hovers over a date field on the site, they will be able to see what that day is across the entire world. And to make things a bit easier, I also have the rough season when it happened. (I hope, still need to heavily test the seasonal thing.)</p>
<p>The frustrating part is that creating a tooltip over the date is somewhat anti-climatic. It doesn't look like much, but there are weeks of work behind it. Weeks of working on data files, puzzle through bugs, and learning new things.</p>
<p>It doesn't even look that exciting at the other end.</p>
<pre><code>{{event: 1454/7/41 MTR, kyo, Born}}
</code></pre>
<p>But, it doesn't matter. That little tooltip is the result of a lot of hard work and I'm pretty proud of it. It takes a lot to make something look simple. You can play with it on <a href="https://fedran.com/rutej%C3%ACmo/">Rutejìmo</a>'s page. It also has the beginning of the new inline events, which will eventually lead into building a timeline automatically down the line.</p>
Finishing up this round of MfGames Culture2015-12-06T06:00:00Zhttps://d.moonfire.us/blog/2015/12/06/finished-up-culture/Retrospective on a month of working on MfGames Culture.<p>For November, I decided to focus on <a href="https://mfgames.com/mfgames-culture/">MfGames Culture</a>. It was a pretty productive month in many ways, but the development cycle has ended and it is a good time to report where I am and what needs to happen next.</p>
<h1>Reasons</h1>
<p>I've been trying to write this library for a number of years. I've tried it in a few forms, the C# version being the previously most successful one, but it hit some conceptual snags and overwhelming details (the bulk of the reasons why my bigger projects fail).</p>
<p>This time, I tried something entirely different, I wrote it in Javascript (well, Typescript). Typescript is <em>not</em> one of my primary languages, but I think it has a good chance of becoming one. I've played with it in a number of other iterations, mostly related to <a href="/tags/author-intrusion/">Author Intrusion</a>.</p>
<p>The library is a small but rather complex part of my master plan for <em>Author Intrusion</em>. I want to use this library to tag the headers of my various chapters and tell me when and where it happens. The library is also to let me build a timeline of the entire world of <a href="https://fedran.com/">Fedran</a>.</p>
<h1>Confusion</h1>
<p>The biggest problem with Typescript is that I couldn't figure out the <em>One True Way</em> of getting it to work. Whenever I thought I figure out how to get something working, it broke with the next step. And then I'd puzzle through that and come up with a new way, it ended up breaking up with the next one. It was a rather frustrating experience, to say the least.</p>
<p>My first thought was to write using ES6 modules (<code>tsc --target es6</code>). I figured that Author Intrusion is going to take five years, so going with the edge development seemed like the right idea.</p>
<p>Needless to say, I <em>love</em> the ES6 import syntax. It compiled great and everything was perfect.</p>
<p>At least until I needed to get it tested. I really like to use TDD (Test-Driven Development) which means I usually write with tests to help prove out small, discrete concepts. I do not do BDD or top-level testing, but working on internal logic to make sure the foundation was correct.</p>
<p>To get testing working, I ended up having to go to <code>jasmine-es6</code> but after trying <code>mocha</code>, <code>jasmine</code>, and a bunch of other things.</p>
<p>Eventually, I got <a href="https://github.com/dmoonfire/mfgames-culture-js/">mfgames-culture-js</a> working (well, it was called something else at the time).</p>
<p>Then I encountered the next stumbling block. I tried to put it up as a <a href="https://mfgames.com/mfgames-culture/fiddle">fiddle</a>. This was a fun experience, mainly because I have not really made a web-based Javascript page before, but also because my nice and stable system crumbled. Absolutely failed because I couldn't figure out how to get the ES6 module to work with the website.</p>
<p>I tried a lot of things, <code>babel</code> and <code>webpack</code>, but I couldn't figure out them out. There were errors messages that apparently <em>no one else in the universe had ever had</em>. I spent over a week trying to get it working. I also tried creating two versions of the <code>mfgames-culture-js</code> (AMD and CommonJS) to use that.</p>
<p>Eventually, I went back to <code>mfgames-culture-js</code> and redid it as UMD (universal module definition). That required me to rework the testing framework, but I could go back to the regular <code>jasmine</code> which I was happy about. I did have to reduce the entire project to one file (as opposed to breaking it apart) because Typescript and RequireJS don't seem to support multiple files being reexported or combined together. <em>I really hate having a 1,000+ line source file.</em></p>
<p>Following the refactoring that came with it, I finally got my website working. I figured I was free and clear.</p>
<p>And then I started working on the command-line client (<code>mfgames-culture-cli</code>). So, the CLI is based on NodeJS and CommonJS modules while the website used AMD modules. I had to split my JS library in two because it was the only way I could figure things out.</p>
<p>Another week later, I managed to get everything working smoothly in a Frakenstein-like system. I don't think it is the <em>right</em> answer, but it worked for me. At least until I try to do something else, that is.</p>
<ul>
<li>UMD modules for the core libraries</li>
<li>Jasmine for testing</li>
<li>NPM for various scripts (as opposed to <code>gulp</code> or <code>grunt</code>)</li>
<li>No <code>babel</code> or <code>webpack</code></li>
</ul>
<p>I also ended up with four packages and one website:</p>
<ul>
<li><code>mfgames-culture-data</code>: JSON data files.</li>
<li><code>mfgames-culture-js</code>: Typescript/Javascript core module.</li>
<li><code>mfgames-culture-node</code>: NodeJS-specific modules to support the main one.</li>
<li><code>mfgames-culture-cli</code>: Command-line interface to use it.</li>
<li>My fiddle website</li>
</ul>
<p>I also spent another week trying to figure out how to get Typescript “typings” working. That still hasn't been resolved yet, but I ran out of time. I'll puzzle that out the next round, mainly to “finish up”.</p>
<p>I also spent a week documenting it and I feel that I'm only 50% done with that task.</p>
<h1>Complexity</h1>
<p>Calendars are complex. This weekend, I tried to explain to my dad how my system works. Going through the JSON files, I realized that this really isn't a trivial system and I don't think I can make it simple. I can make it easy to isolate and work, but this will never be “easy”.</p>
<p>I did create a <a href="http://discuss.moonfire.us/c/mfgames/mfgames-culture">forum</a> to talk about it. I'll be glad to help if anyone does decide to join in or try to use it.</p>
<h1>Retrospective</h1>
<p>I spent a <em>lot</em> of time trying to figure things out. All my web searches didn't really give a good answer, so I fumbled through it. I may have gotten a decent work flow, but it doesn't feel “right” yet.</p>
<p>I also uploaded everything to npmjs.com, so anyone can use them.</p>
<h1>Next for the library</h1>
<p>There are a few steps still left for the library:</p>
<ul>
<li>Get my Fedran calendars implemented</li>
<li>Get the Aztec/Mayan calendar working</li>
<li>Finish documentation</li>
<li>Get various typings submitted so others can use it</li>
<li>Looking into the <code>typings</code> infrastructure</li>
<li>Get Typescript to reference modules</li>
</ul>
<p>I'm sure as my understanding and the ecosystem both evolve, I'll change how to these systems are arrange. That is why I kept them at the 0.1.0 level. I am nowhere close to being version 1.0.0.</p>
<h1>December plans</h1>
<p>The main reason I worked on this library for November was to take a break from writing. I posted on Facebook and a few other places, but generally I was frustrated with my weekly serials. I made a decision to start releasing <a href="/tags/flight-of-the-scions/">Flight of the Scions</a> for <a href="https://www.patreon.com/dmoonfire?ty=h">Patreon</a> subscribers only. I'll do another blog post on that, but I'm getting excited about writing again.</p>
MfGames Culture work, Reviews, and other plans2015-11-15T06:00:00Zhttps://d.moonfire.us/blog/2015/11/15/culture-reviews-plans/Among many other things, I've been working on a Javascript version of MfGames Culture and got something to show off.<p>So, lately I've been feeling pretty bad about not blogging about anything besides my weekly fantasy serial, <a href="https://sand-and-ash.fedran.com/">Sand and Ash</a>. But, I'm hitting some interesting points on a side project and wanted to take a break to talk about it.</p>
<h1>Priorities</h1>
<p>There are two topics I don't post a lot about here: work and family.</p>
<p>Work is, well, it is easier to keep my personal life separate from my professional one (though having a last name of Moonfire makes that difficult in general), but I usually don't do anything besides mention how much I'm working. I love my job, even with the occasional long hours and challenges, but I don't think this is the right place to talk about it.</p>
<p>Family is in the same boat. I have a family, they are important, but I don't think this is the venue for speaking about things besides high-level items. In this case, we've had plague slowly working its way through the household which makes it hard to find time to do something secondary like writing a post.</p>
<p>Which makes it difficult when I have a conflict of priorities. A long time ago, I had been sitting in a doctors office while paging through a Reader's Digest when I came up to an inspirational quote that stuck with me: You have time in your life for only three things.</p>
<p>Well, I have those three things: Family, Work, and Writing. Almost every single side project I have is writing something or writing programs to support my writing. However, those takes a side burner when work is getting overwhelming or there is plague.</p>
<h1>Reviews</h1>
<p>Like many readers and writers, I have a large “to read” stack of books. I also have a large “to review” since writing a review, be it good or bad, is the best way to support authors (well, and game writers too). A few weeks ago, I thought I'd work on my to review pile by writing one review a day.</p>
<p>Well, that didn't work.</p>
<p>After three weeks of trying, I wrote ten reviews (two, four, and four). Any more than four and I get burned out. So, it looks like I have a cap there. I don't know how well I'm doing, or if they are good reviews, but at least I'm writing and posting them.</p>
<p>I'm probably going to aim for a review a week, though, given how much time it takes. I'm not a great review writer, mainly because I probably focus on things that I want to get out of reviewers. Plus, I'm occasionally critical of stuff, including my friend's writing.</p>
<h1>MfGames Culture</h1>
<p>The biggest project I've worked on the last month or so is <a href="https://mfgames.com/mfgames-culture/">MfGames Culture</a>*. I started this on a lark because I wanted something “easy” (hah!) to write and finish, something that would give me a feeling of success to handle a low-level depression that has been haunting me.</p>
<p>* <em>Most of my libraries are prefixed with “MfGames” (Moonfire Games) because I don't like naming collisions.</em></p>
<p>The problem with <em>Culture</em> is that I'm <em>terrible at estimates</em>. Seriously, I am such an optimist about it that I've gotten written up about it at work. The problem is that I <em>can</em> do it in the time I want, I'm just scattered with family, work, and interruptions that it takes twenty times as long as it would otherwise.</p>
<p>But, that's the nature of my life.</p>
<h2>Programming</h2>
<p>I previously spent a month working on a C# version of <em>MfGames Culture</em> but I stalled out. This time, I decided to write a Typescript (Javascript) version to support <a href="/tags/author-intrusion/">Author Intrusion</a> since I'm working on a Javascript version of that. <em>Culture</em> has also been something that I've been meaning to write for a while.</p>
<p>My primary goals for the library are:</p>
<ul>
<li>To create an API for handling my fantasy calendars (which are not based on Gregorian).</li>
<li>To have the ability to sort every chapter in every novel or story in chronological order.</li>
<li>To create a time line of events in my world. Ideally, this would be one of those fancy HTML5 scrolling one but I'd be happy with a vertical list of every event that happened in the world.</li>
</ul>
<p>The long term goal is being able to automatically create a video that shows the events of my world when and where they are happening. I think this would be pretty cool.</p>
<h2>Fiddle!</h2>
<p>Now, the whole reason I wanted to post this blog post. I have an example that someone can play with, a <a href="https://mfgames.com/mfgames-culture/fiddle/">fiddle</a> that allows you to see the library actually in action.</p>
<p>What the fiddle lets you do is change the calendar and see the results in real time. So, if you decide that January is only 25 days, then change it and see the results.</p>
<p>There isn't a lot of documentation on it (see below), but I'm hoping to work on that.</p>
<p>This actually is the first time I've written a Javascript library from end to end, but it uses my data files and library to populate a culture and a calendar. I think it is pretty cool.</p>
<h2>Remaining Tasks</h2>
<p>In the C# version, I had two calendars for the <code>en-US</code> culture: the Gregorian calendar and the duodecimal (base-12) for hours and minutes. I still need to convert and merge that into the new system.</p>
<p>Since I decided to switch over to JSON for the source file, I also need to update the C# library to also handle that. At the moment, that code is bit-rotted because I think JSON ended up being a more generic file format for supporting both desk and web clients.</p>
<p>I also need the ability to handle time spans, the time between two calendar instants.</p>
<p>Documentation. Oh, how I hate documenting, but I'm trying to get better at doing so.</p>
<h2>Justification</h2>
<p>The reason I'm working on this library still remains true. It is difficult to find any programming library that handles an arbitrary (one that isn't Gregorian) calendar system. If a fantasy or sci-fi author wants to have a base-10, base-13, or some other system, it is difficult to create something that can be easily parsed or formatted.</p>
<p>With Author Intrusion, I want to be able to have the date and time of a chapter written up in the metadata and have it automatically translate or format it. That way, I can say “sort chapters chronologically” or see how much time is between two chapters.</p>
<h1>Plans</h1>
<p>Finishing up anything is the hard. While I got the basics of the library done, there is still a <em>lot</em> to finish. So, my current plans is to spend the rest of November working on the library before doing a writing project in December.</p>
<p>Not to mention, December is <a href="http://www.frathwiki.com/Lexember">Lexember</a> so I'm doing to be working on building up more works for <a href="https://github.com/dmoonfire/miwafu/">Miwāfu</a>, the constructed language for <a href="https://fedran.com/">my fantasy world</a>.</p>
Author Intrusion 0.1.02015-07-29T05:00:00Zhttps://d.moonfire.us/blog/2015/07/29/node-author-intrusion-0.1.0/Added some functionality to Author Intrusion and bumped up the version to 0.1.0.<p>Over the last few weeks, I did some minor improvements to <a href="/tags/author-intrusion/">Author Intrusion</a>. Since I need to actually use it to write stories and novels, I figured I'd get to a stopping point and update the code.</p>
<h1>Part of Speech</h1>
<p>One of the biggest features I needed was part of speech tagging. This is in the <a href="https://github.com/author-intrusion/node-author-intrusion-pos-tagger">node-author-intrusion-pos-tagger</a> module. This uses the output from the splitter and adds more information about the part of speech, such as “present tense noun” or “adverb”. This lets me identify overuse of adverbs in a short distance (much like before).</p>
<h1>Reworking Echo</h1>
<p>I also significantly reworked the <a href="https://github.com/author-intrusion/node-author-intrusion-pos-tagger">echo</a> plugin to handle the POS tagging, checking for echo words against the stem (which can be used to treat “spit” and “spitting” as the same word).</p>
<p>This was a breaking change, sadly, but hopefully not too many people are using it. I added some unit tests in this (and a number other plugins) to help explore the different functionality.</p>
<h1>Documentation</h1>
<p>I've updated the <a href="https://github.com/author-intrusion/author-intrusion-docs">documentation</a> and the script to check everything out.</p>
<h1>Forums</h1>
<p>I also created a <a href="http://discuss.moonfire.us/c/author-intrusion">category</a> on my forum, if anyone wants to talk about it. It also includes a sub-category for recipes for those who want to talk about how to do some of the analysis.</p>
<h1>Next Steps</h1>
<p>The next biggest step is to get the <em>echo</em> plugin to handle <a href="https://en.wikipedia.org/wiki/N-gram">ngrams</a>. Right now, it works on a token-by-token basis, but I also want it to be able to identify a series of three or four word segments and treat them as higher priority duplicates. For example, to see if the paragraphs starts repeat themselves or sentences have the same beginning.</p>
Mechanical Writing2015-07-20T05:00:00Zhttps://d.moonfire.us/blog/2015/07/20/mechanical-writing/Author Intrusion, though I think it will help my writing, has a risk of getting caught in the objective and mechanical aspects of writing.<p>I work in phases: a bit of writing for a few weeks, a bit of coding, a lot more writing. I'm just coming out of a programming phase where I spent the last two weeks working on a new implementation of <a href="/tags/author-intrusion/">Author Intrusion</a>.</p>
<p>Author Intrusion is my attempt to get around my writing flaws. I've done quite a few attempts at it, but this is both the most extensive and least expansive version I've written. But, it also has the same pitfalls that every other version provided.</p>
<p>The biggest one is the mechanical nature of the analysis. Say I mark things as being overused if it shows up more than ten times in a hundred words. There might be a place where that is actually useful, or important or intentional.</p>
<blockquote>
<p>“If you want to be called Max, Max. Max is what I'll call you.”</p>
</blockquote>
<p>One of my favorite lines (half-forgotten so probably not accurate) from the Norby series by Issac Asmoiv.</p>
<p>I've decided to show close words as warnings and very close words as errors. The programmer in me really wants to get rid of all warnings and errors, but with this tool, it is just highlighting what <em>might</em> need to be fixed, not what <em>must</em> be fixed. Though, the goal is to have the errors become the points I really should fix it.</p>
<blockquote>
<p>“To” is used 7 times in 57 words.</p>
</blockquote>
<p>In the <a href="https://sand-and-blood.fedran.com/chapter-01/">first chapter</a> of <em>Sand and Blood</em>, I overused “to” once. When I read it, it seems okay, but just knowing it is there would have caused me to rework it at least a little.</p>
<p>This is one reason Author Intrusion won't ship with a set of grammar. Because writers should know their own flaws. I know I overuse <a href="http://www.englishpage.com/gerunds/part_1.htm">gerunds</a>, so I'm writing a rule (once I finish coding it) that will highlight occurrences once I get over a certain density. Same with echo phrases and words. These issues might go away, but there will be other ones. If I can at least highlight them, maybe I can improve my writing.</p>
<p>While I won't ship rules with Author Intrusion, that doesn't mean I can't create a <a href="http://discuss.moonfire.us/c/author-intrusion">forum</a> to talk about it, including letting me and others come up with <a href="http://discuss.moonfire.us/c/author-intrusion/recipes">sample ones</a>.</p>
<p>Still having fun with working on this version. It feels better, mainly because it is broken into smaller parts and easier to manage. But, at the same time, I still have to write. So I'm going to use what I've written so far and work on a commission and <a href="/tags/second-hand-dresses/">Second-Hand Dresses</a>, my romance novel, for a little bit.</p>
Author Intrusion for Node.js2015-07-12T05:00:00Zhttps://d.moonfire.us/blog/2015/07/12/node-author-intrusion/Once again, I'm trying to make a version of Author Intrusion that works for me. After a weekend of developing, I have a pretty solid base using a language I just learned: TypeScript and Node.js.<p>The idea of <a href="/tags/author-intrusion/">Author Intrusion</a> has been haunting me for years. I know that I have problems in my writing, little flaws, that make it hard for readers to get into my stories. One reason I need two sets of editors is simply to catch them. But, I also know that it is possible to write a program that could detect many of those flaws, they are mechanical but complicated.</p>
<p>I've written Author Intrusion a number of times now, usually as a full-blown text editor with analysis plugins in the background. Some of them have gotten pretty far, but I always hit some road block that prevented me from going further. Not to mention, it takes <em>time</em> to write a text editor simply to get far enough to write what is really important, the analysis aspect.</p>
<p>After fumbling for a few years, I had almost given up on it. It was suppose to help me, but I just didn't have the skills to do it myself. It hurt, because I know it would help me, but I couldn't manage it.</p>
<p>But, like writing, I couldn't stop. I kept thinking about how to make it work, to change how I viewed the program and write something that actually worked. A few ideas began to gel together over the last few months until they finally came into focus last week. I had to give up what I envisioned Author Intrusion doing and change it into something I could do.</p>
<h1>Command Line Tools</h1>
<p>One of the biggest things is getting rid of the text editor. Yeah, it's fun writing something on screen, but it is also distracting. If I focused on the analysis part as a separate tool, then I could avoid that. Fortunately, <a href="http://atom.io/">Atom</a> had an infrastructure for doing that.</p>
<p>The bulk of the new system is <code>author-intrusion-cli</code>, a command-line tool that takes a Markdown file and produces a standard output from it. The format is pretty simple, an error message for Emacs or the linter JSON for Atom.</p>
<p>It isn't perfect, but it worked.</p>
<h1>Analysis</h1>
<p>There is only one real analyzer now: echo words. It scans the file and looks for the same word used in rapid succession. When that happens (sadly the first chapter of <a href="https://sand-and-blood.fedran.com/">Sand and Blood</a> has some example), then Atom will underline the problem text. When you type and fix it (and save), then they go away and new ones pop up.</p>
<p>There will be more, the hot buttons of my writing, but I feel pretty good about the results. This is something I can extend and build on, hopefully without hitting the roadblocks that previously stopped me.</p>
<img class="img-responsive post-img-link" src="example.jpg" />
<h1>Correctness</h1>
<p>I'm making a point of not making default settings for this. Author Intrusion isn't trying to make a “one true” grammar checking but to allow a writer who knows their flaws to identify them. I want it to let someone keep their voice and style while giving hints on improvement.</p>
<h1>Packages</h1>
<p>I spent a <em>lot</em> of time cleaning up the code and posting it on <a href="https://github.com/orgs/author-intrusion/">Github</a>. I also write a script to install it on a Linux, it's rough, but it seems to work. It also means, in the odd chance that someone is interested, I can actually accept some contributions. I don't expect it, but there was already someone expressing an interest.</p>
<p>I also <a href="https://github.com/author-intrusion/author-intrusion-docs/blob/master/index.markdown">documented</a> a little bit of the process.</p>
<h1>Future Plans</h1>
<p>I can't worry about failing. This may work or it may not, but it feels pretty good to me. I liked how easily everything fell into place, no doubt because of the endless times I've already written it. The code base is pretty solid. Needs changes, but the foundation is pretty simple because I'm doing less so I can focus on the important parts.</p>
<p>I want this to succeed. I think it will help me be a better writer by identifying my problem areas, the places I know are wrong but I can't always see. This tool, both now with only one feature and later with the other things I want, should help me do that.</p>
Setting up a writing project on GitLab2015-05-09T05:00:00Zhttps://d.moonfire.us/blog/2015/05/09/gitlab-projects/One advantage of writing novels in Markdown is that I can use Git to manage to projects. However, having only a local Git on my laptop is risky, so having a remote Git repository helps preventing losing data. Here is how to get a project up on GitLab.<p>As I've mentioned on my blog before, I write novels using <a href="http://daringfireball.net/projects/markdown/syntax">Markdown</a>. I also keep everything in a <a href="http://git-scm.com/">Git</a> repository which helps prevent me from overwriting chapters or giving me a chance to commit changes while at the family cabin.</p>
<p>The biggest advantage of Git is the ability to coordinate writing between two computers (laptop and basement). This requires a third computer, a remote repository, to handle working between the two machines. Until a few months ago, I used a SSH host to handle them.</p>
<p>Then I found out that <a href="https://gitlab.com/">GitLab</a> had much of the features of my favorite (and just about everyone else) host, <a href="https://github.com/">GitHub</a>. The main difference is GitLab is open-sourced and less popular while GitHub a private company, closed-source, but <em>very</em> popular.</p>
<p>I really like GitHub, mainly because of their interface. But they don't allow for private repositories. And I like to keep my novels private until they are released under <a href="http://creativecommons.org/">Creative Commons</a>.</p>
<p>Fortunately, GitLab allows for unlimited private repositories. Also, they have a <a href="https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md">omnibus edition</a> which lets someone like me (somewhat technical) host their own private version outside of a third-party's server. That isn't important for many people, the GitLab hosted version is probably good for many writers.</p>
<h1>Setting up a project</h1>
<p><em>You can click on the images for larger versions.</em></p>
<p><a style="width: 260px; clear: right; float: right; margin-left: 1em; margin-bottom: 1em;" href="sand-and-ash-setup.png"><img class="img-responsive post-img-link" src="sand-and-ash-setup.png" alt="GitLab Setup" height="420" width="244" /></a></p>
<p>Setting up projects is pretty easy. Create an account over at GitLab and select “New Repository”. This will ask some basic information, the only thing I enter is a <em>slug</em>, a name with dashes, that identifies the project. For example, my novel <a href="/tags/sand-and-ash/">Sand and Ash</a> would have a slug of “sand-and-ash”.</p>
<p>Also make sure the project remains as “Private” which means only people you explicitly give access to the project can do so.</p>
<p><a style="width: 260px; clear: right; float: right; margin-left: 1em; margin-bottom: 1em;" href="sand-and-ash-gitlab.png"><img class="img-responsive post-img-link" src="sand-and-ash-gitlab.png" alt="GitLab example #1" height="259" width="118" /></a></p>
<p>Once you set up a project, this brings you to the dashboard. This is where you can list all the files in the project, or view the check-ins. You can also use “merge requests,” which I haven't had a chance to investigate. In theory, you could have an editor work on sections and then push out those changes for you to review.</p>
<p>The other aspects of use are milestones and issues. I use both of these pretty frequently to keep track of things to do before publication (or updates if you happen to need an update).</p>
<h1>Milestones</h1>
<p><a style="width: 260px; clear: right; float: right; margin-left: 1em; margin-bottom: 1em;" href="sand-and-ash-milestones-1.png"><img class="img-responsive post-img-link" src="sand-and-ash-milestone-1.png" alt="GitLab example #2" height="324" width="214"/></a></p>
<p><a style="width: 260px; clear: right; float: right; margin-left: 1em; margin-bottom: 1em;" href="sand-and-ash-milestones-2.png"><img class="img-responsive post-img-link" src="sand-and-ash-milestones-2.png" alt="GitLab example #3" height="420" width="244" /></a></p>
<p>Milestones are significant points in the novel's life. I use <a href="http://semver.org/">Semantic Versioning</a> to identify the versions of the novel with 1.0.0 being the first release. Before that, I start with a number of them that represent the first three rounds of edits (0.1.0, 0.2.0, and 0.3.0), the one-off rounds (say clean up quotations or look for echo words), alpha readers (0.4.0), editor rounds (0.5.0 through 0.8.0), and final beta reading (0.9.0 and higher).</p>
<p>Setting up a milestone is pretty easy. Click on the “Milestones” lists all the milestones and then click on “New Milestone”. From there, you can enter a milestone, you give it a name (“v1.0.0” for the initial release) and a short description. I'd love to have a due date, but I can't publish reliably enough for that so I don't pick one. That way, it won't nag me about failing to meet my self-imposed deadline.</p>
<h1>Issues</h1>
<p><a style="width: 260px; clear: right; float: right; margin-left: 1em; margin-bottom: 1em;" href="sand-and-ash-issues-1.png"><img class="img-responsive post-img-link" src="sand-and-ash-issues-1.png" alt="GitLab example #4" height="327" width="138" /></a></p>
<p><a style="width: 260px; clear: right; float: right; margin-left: 1em; margin-bottom: 1em;" href="sand-and-ash-issues-2.png"><img class="img-responsive post-img-link" src="sand-and-ash-issues-2.png" alt="GitLab example #5" height="328" width="252" /></a></p>
<p><a style="width: 260px; clear: right; float: right; margin-left: 1em; margin-bottom: 1em;" href="sand-and-ash-issues-3.png"><img class="img-responsive post-img-link" src="sand-and-ash-issues-3.png" alt="GitLab example #6" height="324" width="253" /></a></p>
<p><a style="width: 260px; clear: right; float: right; margin-left: 1em; margin-bottom: 1em;" href="sand-and-ash-issues-4.png"><img class="img-responsive post-img-link" src="sand-and-ash-issues-4.png" alt="GitLab example #7" height="326" width="248" /></a></p>
<p><a style="width: 260px; clear: right; float: right; margin-left: 1em; margin-bottom: 1em;" href="sand-and-ash-issues-5.png"><img class="img-responsive post-img-link" src="sand-and-ash-issues-5.png" alt="GitLab example #8" height="326" width="172" /></a></p>
<p><a style="width: 260px; clear: right; float: right; margin-left: 1em; margin-bottom: 1em;" href="sand-and-ash-issues-6.png"><img class="img-responsive post-img-link" src="sand-and-ash-issues-6.png" alt="GitLab example #7" height="326" width="252" /></a></p>
<p>Issues are done the same way as Milestones, but an issue can have a milestone associated with it which is why I do those first. It may not make sense to do issues, but I find them very helpful with the little things that always get lost. The easy issues are “get editing done” and “create a cover”, those can be assigned to a milestone without a problem.</p>
<p>In other times, while I'm working on the story, I'll realize that I need to go through the entire document and fix one specific item. When I realize these things, I'm either in the process of writing something else or I can't focus on it. For these situations, I'll create an issue to go through the book later and then I don't have to worry about forgetting it.</p>
<p>Some examples:</p>
<ul>
<li>Make sure Rutejìmo has his weapon from chapter 18 on</li>
<li>Create the epigraphs at the beginning of the chapters</li>
<li>Enter the epigraphs into the wiki site</li>
<li>Correct a character's name because it was confusing with another one</li>
<li>Back-fill foreshadowing for something</li>
<li>Add details about a city.</li>
</ul>
<p>One really nice feature of GitLab is that one issue can reference another one. If you type an octothorpe ("#"), the system will present a drop-down list of existing issues. There is some search by typing to reduce the list. This comes in handy if you have an issue that depends on another one (updating the wiki requires details being written).</p>
<p>While entering issues, you can use Markdown to create links, bold, italic, and even include images. Most of it is pretty easy if you use the editor bar (images are drag-and-drop).</p>
<p>When I create issues, many times I assign them to a milestone if I know it. For example, “updating the wiki site” is a v1.0.0 milestone. I noticed that the earlier releases will get 10-20 issues assigned to them but the later ones get more precise. They also focus on different things. “Assign ISBN” is a much later task compared to “Make sure he has green eyes.”</p>
<p>One nice thing about tracking tasks and items like this is that I get a list of things that I've done and ones that I have left to do. I found that some evidence of forward momentum helps in those moments when I'm most discouraged about writing ("I'm never going to finish…").</p>
<p>This a technical way of managing books and writing. For me, this works out pretty well because I use the same tools for programming. I'm also easily distracted (e.g., have too many projects), so being able to track the little things that go into writing something like a novel.</p>
Emacs and Multiple Dictionaries2015-04-01T05:00:00Zhttps://d.moonfire.us/blog/2015/04/01/dictionaries/After a few years of struggling with dictionaries while writing, I think I wrote a program that will let me use project, world, and byline dictionaries in writing without shuffling large number of words from file to file.<p>For the last four years, I've been trying to write a program called <a href="/tags/author-intrusion/">Author Intrusion</a>. There were a number of reasons for this, but one of the biggest was that I couldn't find any program that handled dictionaries (really <em>word lists</em>, but a lot of people use the wrong name).</p>
<p>This morning, when I woke up, I ended up doing a random search that took me through a long winding journey that finally gave me an interim solution that is pretty solid until I can get Author Intrusion finished (which may be another four… decades or so).</p>
<h1>The problem</h1>
<p>As with any long-term writing project, I've created a large number of characters, groups, and locations. Most of them are based on a conlang while others just sounded cool. However, when I'm spell-checking my chapters, I need to have those names in the dictionary otherwise they'll continually show up as a typo.</p>
<p>One common solution is to add those names to the program's dictionary. This works out pretty well, until the end of the project. Then, the hundreds of names are not longer relevant for the next series but still show up in suggestions for every project in the future.</p>
<p>My preferred novel-writing editor, <a href="http://en.wikipedia.org/wiki/Emacs">Emacs</a>, has the ability to have <em>per file</em> word lists. This is called “LocalWords”, but it means that I can identify a list of valid words without adding it to my permanent dictionary. Of course, this means I have to keep copying that <em>per file</em> list into each new chapter, which then gets the new words for the characters I've introduced in that chapter. And when I create the chapter after that, it keeps moving and growing.</p>
<p>Because I just finished the draft of <a href="/tags/sand-and-bone/">Sand and Bone</a>, I have built up a three book collection of proper names. This list is in the top of every file, which means I have to scroll down a little to even see the title of the chapter.</p>
<blockquote>
<p>Rutejìmo Chimípu Pidòhu Shimusògo Tateshyúso Pabinkúe Jìmo Mípu Dòhu Pidòhu's Desòchu Sòchu Mapábyo Kechikìma Hyonèku Opōgyo Chimípu's Gemènyo Mènyo Pábyo Kìma Mapábyo's Zotetsūchi Rutejìmo's Hyonèku's Gemènyo's Ryayusúki Wamifuko Nèku Hána Zúchi Mépu Nenemépu Shimusògo's Desòchu's Myunédo Shimusogo Karawàbi Wàbi Tsubàyo Bàyo Tsubàyo's Tejíko Palasaid Markon Tejíko's Mifuníko Yutsupazéso Yutsupazéso's Karawàbi's Nibonyāchu Jyotekàbi Yunujyoraze Byomími nibonyāchu ranuchyahāhi Mípu's shimusogo dépa alchemical dépa's Mifukiga Chobāni Rabedájyo Badenfumi Shigáto Porlin Kamanen Kakasaba Mioshigàma Pabinkue Mikáryo Mikáryo's tazágu Palarin Mistan rikunámi Ryachuikùo Tateshyúso's Nedorómi Chidomifu Kapōra Káryo Chyábi Ganifúma Ralador Markin Kidorīsi Mifúno Mafimára pyābi Mifuno Faríhyo mizonekima chyòre Rolan Madranir Kiríshi Som figaki tòra chyóre's shikāfu Tachìra's Chobìre's Wh Tachìra Monafuma Gidon Kormar Nigímo wabōryo Faríhyo's avian's Ríhyo Gímo ryodifūne Tsudakìmo Myobùshi Funikogo Ganósho Myobùshi's Gidorámi Pyatose myofūne Pyatòse Gichyòbi Higoryo Ríshi Jacin Torabin Kishifín's Makohūni's Tsu Rojikinomi Fimúchi Rojikinòmi Rapinbun Finol Pokīmu Waryōni Nyochizoma clanless Chizoki Miyóna Kyōti Tijikóse Chyobizo Nichikōse Tifukòmi Talsir Shifáni Milifor Krum Opōgyo's banyosiōu kojinōmi kojinōmi's Nyobichóhi Mifúno's helmed Kitópi Piròma Tópi Bakóki Bakóki's Nifùni Byochína Chobìre Midoshina Kafūma Korechyoki Baroshìko Tedoku Nuchikomu Machikimu Garènu Piròma's Kitópi's Nana dépas Kosobyo Kosòbyo nocked Fidochìma Foteramàsu Foteramasu chima Tsupòbi Dimóryo Fùni petabiryōchi Chína Techyomása Mioráshi Kosòbyo's Kidóri Atefómu's Kidóri's ambushers Tikói Menodàka Tateshyuso Kos Ràchyo Záji Gichyòbi's</p>
</blockquote>
<p>That's a lot of names, including a couple that were removed for pacing. Almost every single one of them isn't in the final chapter of <em>Sand and Bone</em>, but they were in one of the hundred or so chapters before it.</p>
<p>There is also no easy way of removing the Miwāfu names and passing them into the next story since those are pretty common across any story I have in the desert.</p>
<p>As far as I could tell, there were only two ways of handling all those names: put it in the permanent dictionary or shovel it along the chapters as I went.</p>
<h1>Vim</h1>
<p>About a year ago, I found out that <a href="http://en.wikipedia.org/wiki/Vim_%28text_editor%29">Vim</a> had a setting that allowed multiple dictionaries, but I didn't want to grok a new writing environment when I had (high) hopes for getting Author Intrusion done.</p>
<h1>The idea</h1>
<p>This morning, I found a random link that led to another. Eventually, I came up with <a href="http://www.emacswiki.org/emacs/WcheckMode">Wcheck</a>. It looked like it had potential for resolving my dictionary problem, so I spent an hour or so trying it out.</p>
<p>In the end, I couldn't get it to work. But, the process of trying gave me a little epiphany on what could work. Instead of changing the library, I decided to write a wrapper around <code>aspell</code> that interrupted checking words and substituted my own lookups instead.</p>
<p>The results fell into place pretty easily. With a <code>local.words</code> file in the same directory as the chapters, my newly created <code>caspell</code> program loads it into memory. When Emacs asks for a word to spell, it checks to see if it knows about the world already and verifies it as <em>passed</em> even if the base dictionary doesn't know about it.</p>
<p>Likewise, adding a word adds it to the <code>local.words</code> file, not the <code>aspell</code> personal dictionary.</p>
<h1>But wait, there's more</h1>
<p>The basic format of the file is pretty simple.</p>
<pre><code>word nibonyāchu
word dépa
</code></pre>
<p>I originally went with “&” as the suggestion used in the pipe, but then I realized I could use readable words without too much of a problem. So, it became “word” and made things a lot easier to process.</p>
<p>Getting the basic lookup was a nice little rush, but then I realized that I could return suggestions. That lead into writing code that gave suggestions for “incorrect” words that I want to expand into real ones.</p>
<pre><code>suggest Shimu = Shimusogo, Shimusògo
suggest shimu = Shimusogo
</code></pre>
<p>There is a certain mindset when things are working. It is easy to move into the next code, though times to the results take longer to develop. In this case, I decided to allow one file to include another. This pulls in the words and suggestions from other files but doesn't merge them together.</p>
<pre><code>command include "../../sand-and-blood/chapters/local.words"
</code></pre>
<p>And then I had it. Dictionaries for <em>per file</em>, <em>per project</em>, <em>per world</em>, and any other combination that I need. I'm planning on creating them over the next couple files, but I think it will let me chain dictionaries so book two will include book one's words. And book three will add book two's which also includes ones. And then <a href="/tags/raging-alone/">Raging Alone</a> includes all three books.</p>
<h1>And then one more</h1>
<p>There was one more thing I ended up doing before I stopped. I used Emacs's <code>abbrev-mode</code> to do auto-corrections while writing. That way, I can type “Rute” and have it expand into “Rutejìmo” complete with accents. Same with various greetings, names, and locations.</p>
<p>As you can guess, I added that feature into the file too.</p>
<pre><code>replace GS = Great Shimusogo
replace GT = Great Tateshyuso
</code></pre>
<p>This feature isn't built in, so I wrote a special mode for the program that takes a <code>local.words</code> and creates a <code>abbrev.el</code> file for the mode.</p>
<pre><code>$ ls
local.words
$ caspell --emacs -p .
$ ls
abbrev.el local.words
</code></pre>
<h1>Full example</h1>
<p>A larger example for the <code>local.words</code> for <em>Raging Alone</em>:</p>
<pre><code>command include "../../sand-and-blood/chapters/local.words"
suggest Shimu = Shimusogo, Shimusògo
suggest shimu = Shimusogo
replace GS = Great Shimusogo
replace GT = Great Tateshyuso
word Badenfumi
word Basamiku
</code></pre>
<p>The entire thing is rewritten whenever I add a word to the dictionary. Each section (except for commands) is sorted so it always produces a consistent order. This makes source control easier to work with (always sort output for that reason, it saves a lot of time later).</p>
<h1>Tying it all together</h1>
<p>Once all the files are created and populated, I had to tell Emacs about the new program and how to hook up the abbrevations. This is done in the <code>.emacs</code> file. I have a hook for text mode that automatically configures what I need.</p>
<pre><code>(defun my-text-hook ()
(setq fill-column 99999)
(setq
abbrev-file-name
(concat (file-name-directory (buffer-file-name)) "abbrev_defs.el"))
(quietly-read-abbrev-file
(concat (file-name-directory (buffer-file-name)) "abbrev.el"))
(setq save-abbrevs nil)
(abbrev-mode 1)
(setq ispell-program-name "caspell")
(setq ispell-personal-dictionary (file-name-directory (buffer-file-name)))
(flyspell-mode 1)
(visual-line-mode)
)
(add-hook 'text-mode-hook 'my-text-hook)
(add-hook 'markdown-mode-hook 'my-text-hook)
</code></pre>
<p>The key parts are the “ispell” lines for hooking up <code>caspell</code>. The “personal dictionary” uses the name of the text file (<code>(buffer-file-name)</code>), figures out the directory, and then passes it into <code>caspell</code> via the <code>-p</code> parameter.</p>
<p>The other bit is the “abbrev” lines to look for <code>abbrev.el</code> in the same directory as the text file and uses it. It seems to work and I'm pretty happy with the results so far.</p>
<h1>Github</h1>
<p>Like almost everything else I write, I threw it up on <a href="https://github.com/dmoonfire/mfgames-writing-perl/">Github</a> along with a few other programs I've been using. I'll document them eventually but the <code>caspell</code> is pretty functional as-is.</p>
Creating a Fictionary2015-03-15T05:00:00Zhttps://d.moonfire.us/blog/2015/03/15/fictionaries/Over the last week, I've been creating a fictionary for Sand and Blood. Here are my experiences of the process.<p>Wandering about the <a href="http://www.reddit.com/">Reddit</a>, I've seen a discussion or two about a fictionary, a specialized ebook dictionary for a book. A fictionary lets a reader click on a name, place, or obscure word and see the meaning. Most of these descriptions are spoiler-free, they are just reminder for the reader. For larger works that spans a dozen books, this could be helpful when it's been a few hundred pages since you last saw a specific character or someone referenced some place.</p>
<p>Five days ago, I came up on a post on Reddit about <a href="http://www.reddit.com/r/Fantasy/comments/2yjrln/fictionaries_for_the_dresden_files_and_the_first/">fictionaries for the Dresden Files</a>. One of the interesting thing is that <a href="http://thefictionary.net/">Fictionary</a> is now providing author services to let any author create a fictionary for their book and have it hosted over at <a href="http://thefictionary.net/">http://thefictionary.net/</a>.</p>
<p>Since I was stalling on my programming project, I figured it would be a fun to try making a fictionary for <a href="https://sand-and-blood.fedran.com/">Sand and Blood</a>. One of the biggest complaints is that I had some difficult names for English readers (e.g., <a href="https://fedran.com/rutej%C3%ACmo/">Rutejìmo</a> and <a href="https://fedran.com/des%C3%B2chu/">Desòchu</a> for example). A fictionary would be a way of possibly relieving that since I'm not planning on changing the character names for world reasons.</p>
<h1>Getting Started</h1>
<p>After a brief discussion on Reddit, I eventually switched over to <a href="http://thefictionary.net/contact/">email</a> to figure out the process. The person running thefictionary.net doesn't have their name anywhere, but let's go with “Fictionary” just for purposes of this conversation.</p>
<p>Fictionary provided me with three files:</p>
<ul>
<li>A template spreadsheet with a number of columns.</li>
<li>A sample version with a single entry.</li>
<li>A sample HTML which is what the dictionary entry will look like.</li>
</ul>
<p>The only thing I, as the author, needed to provide back was the spreadsheet various cells filled in.</p>
<p>For each term, there were the following:</p>
<ul>
<li>Value: This is the name of the dictionary term, a proper phrase or single word that is relevant for the book. For example, “Rutejìmo”.</li>
<li>Additional Matches: This is any additional terms, separated by semicolons, that also point to the same entry. In my case, “Great Shimusogo Rutejìmo” and “Jìmo” are the same guy so they went here.</li>
<li>Key Facts: This is where you can have a key/value pair of entries. Some authors have this, such as date of birth or family members, but I won't for reasons I'll describe below. It is also optional.</li>
<li>Definition: This is the definition of the term. Even though it is a cell, it can be fairly long.</li>
<li>Importance: This is either blank or with a numerical value of how important an entry it. Lower is more important. Fictionary uses this to determine how multiple entries show up on common words, like “Valley” in my case.</li>
<li>Person Indicator: “Yes” or “No”.</li>
</ul>
<p>In the end, it is pretty easy thing to fill out. Copy/paste from your notes (or a pretty form of it) and then send it in.</p>
<p>The hardest part is making sure you get all the terms. One of the services that Fictionary provides (and how they are paying for the free hosting of the dictionary file) is an analysis of the document to find unusual words or phrases. I considered it money worth spent, mainly because it found a single typo (there will be version 1.1.1 or 1.2.0 in the near future because of it). It also gave me a decent list of things to make sure I had properly documented.</p>
<h1>Before that point</h1>
<p>Before I started creating a fictionary, I had already created a <a href="http://github.com/dmoonfire/fedran/">wiki-like</a> and a resulting <a href="https://fedran.com/">website</a> that had information about all the characters, quotes, and details of the world. I figured it wouldn't be too hard to convert that into whatever format Fictionary wanted.</p>
<p>One of the biggest things I knew I needed to do was handle the “spoiler-free” part of the terms. Originally, I just tagged the entire page as having spoilers. However, with my future plans for writing, earlier entries would be helpful to later authors. To work with that, I changed it from tagging the entire page to having a “spoiler” tag for specific works.</p>
<p>This made it interesting for Rutejìmo's page since he already has spoilers for <a href="http://journals.fedran.com/issue-00/raging-alone/">Raging Along</a> and <em>Sand and Blood</em>.</p>
<p>What I did was write a program that includes the entire page until it hits a relevant spoiler tag. Since <em>Sand and Blood</em> happened after <em>Raging Alone</em>, it would include spoilers for <em>Raging Alone</em> but stop as soon as it encountered the <em>Sand and Blood</em> spoiler. When I create a secondary fictionary for <a href="/tags/sand-and-ash/">Sand and Ash</a>, it will include spoilers for <em>Raging Alone</em> and <em>Sand and Blood</em>, but stop before the <em>Sand and Ash</em> spoilers. I think that fits the intent of the fictionary.</p>
<p>It is also why I don't have data elements for characters. Their name, date of birth, date of death, and even who they fall in love with can be a spoiler for a later plot. If I leave those points in the linear narrative, I can cut at the spoilers and still give the relevant information.</p>
<p>Overall, I'm pretty happy with the results.</p>
<h1>Expanding entries</h1>
<p>One thing I realized is that filling in the wiki is an exhausting process. I worked on it for a week or so previously, but as I prepared for the fictionary, I realized it was still missing so much. I took the effort to really expand out the characters, providing a few paragraphs back story for every character who showed up or was referenced in the novel.</p>
<p>The length of the entries isn't an issue. I have a couple thousand word entries and I could probably easily ten times that and it will still work. Not to mention, I could easily expand the entries by a few orders of magnitude and the Fictionary's system will still work.</p>
<p>I saw that Fictionary provides wiki service. Obviously, I'm not using it but I could see why some authors could use it. Paying someone to create a wiki (which is what most fictionaries are based on) has a certain appeal. Hand a copy of the book over and you have a wiki created magically about every character, location, and proper noun in the book.</p>
<p>At least until you get a couple fans willing to do it for you.</p>
<h1>Generating the file</h1>
<p>I was worried that I went the wrong way from the beginning. My intent with the wiki was a <a href="http://wikipedia.org/">Wikipedia</a> type of site. Something that gave the narrative of a character including early life, various significant events, and even follow-up links.</p>
<p>Seeing a small entry in the example, which was more of a list, made me think I was going in the wrong direction. However, after a late-night tweet, Fictionary sent me an email and we talked about it. I'll have to admit, it is really nice talking to someone who knew their stuff and the limitations of their software. We quickly figured out that I could directly generate one of their interim formats and get the same results.</p>
<p>A couple hours later, I basically had the file generated from YAML and Markdown (my source files). Less than a day later, I saw my first fictionary for a book I wrote. It was cool. I found some typos, fixed them, and sent up a new version.</p>
<h1>Going live</h1>
<p>The fictionary should go live in a week or so. I'm sure I'll make an announcement when it does.</p>
<p>However, what did go live was my updates to the site. The <a href="https://fedran.com/">wiki-like</a> now has all the details, including spoilers for at least two works. It also has a fairly detailed <a href="http://fedran.com/sand-and-blood/characters/">character</a> page including a list of every character in or referenced by the book. This also includes all the quotes from before.</p>
<p>One thing I've written is a <a href="https://fedran.com/sand-and-blood/plot/">plot summary</a> of <em>Sand and Blood</em>. I hesitated on this, but in the end I did it for a few reasons. The biggest is that I <em>will</em> have a second and third book. A “previously on” summary is good to have for those who delve into the other books, or just want to know what happens. It's marked with a spoiler, of course.</p>
<h1>Styles</h1>
<p>One of the side effects of doing all this is my three major sites for Fedran have the same style:</p>
<ul>
<li><a href="https://fedran.com/">Fedran</a></li>
<li><a href="https://sand-and-blood.fedran.com/">Sand and Blood</a></li>
<li><a href="https://journals.fedran.com/">Journals of Fedran</a></li>
</ul>
<p>It is a little thing, but I'm beginning to like it. It is quirky, but relatively stylistic with a few subtle things.</p>
MfGames.Culture API - Country Codes2015-03-04T06:00:00Zhttps://d.moonfire.us/blog/2015/03/04/mfgames-culture-api-countries/The third part of MfGames Culture is the implementation of ISO country codes.<p>Continuing my effort to document my new library, this is the second part of a short series on <a href="https://github.com/dmoonifre/mfgames-culture-cil/">MfGames Culture CIL</a>, a C# library for globalization written with the intent of supporting non-standard worlds.</p>
<h1>Links</h1>
<ol>
<li><a href="/blog/2015/02/13/mfgames-culture-api-introduction/">Introduction</a></li>
<li><a href="/blog/2015/02/17/mfgames-culture-api-languages/">Language Codes</a></li>
<li>Country Codes</li>
</ol>
<h1>Singleton</h1>
<p>I haven't had much feedback on this library, but the one I did pointed out that I use a “semi” <a href="http://en.wikipedia.org/wiki/Singleton_pattern">Singleton</a> pattern. Now, singletons have gotten a lot of bad press lately but I don't feel that a blanket statement of “singletons are bad” is the best answer either.</p>
<p>Part of the reason I'm writing this is because <code>System.Globalization</code> is a singleton. There is no way to replace the functionality and I can't easily inject my own codes into the system. The need for that flexibility is one reason I've created this is to get around that limitation.</p>
<p>The biggest different between MfGames Culture CIL and <code>System.Globalization</code> is this:</p>
<pre><code class="language-csharp">var manager = CodeManager.Instance;
CodeManager.Instance = new CodeManager();
CodeManager.Instance.Languages = new CustomLanguageCodeManager();
</code></pre>
<p>While CodeManager is a concrete type, all of the parameters inside the manager are interfaces. This means that the entire implementation can be replaced without breaking the rest of the system, assuming the interface contract is maintained.</p>
<p>I consider this an acceptable use of the Singleton pattern because it allows you to replace the singleton with a mock- or testing-specific implementation.</p>
<p>Also, working with instance members means that one could use <a href="http://en.wikipedia.org/wiki/Dependency_injection">Dependency Injection</a> (DI) to provide the code managers without using the <code>CodeManager</code> class entirely.</p>
<pre><code class="language-csharp">public void SomeProcess(ILanguageCodeManager languages)
{}
</code></pre>
<p>While others could use this:</p>
<pre><code class="language-csharp">public void SomeProcess()
{
languageCode = CodeManager.Languages.GetIsoAlpha3("eng");
}
</code></pre>
<p>So, for those who use DI, they ignore the static instance and inject the code. For those who don't need that flexibility, they can use the static instance. I think this allows for either style of developing without declaring the One True Way™ which I'm not fond of.</p>
<h1>Namespaces</h1>
<p>All of the namespaces for this example are <code>MfGames.Culture</code>.</p>
<pre><code class="language-csharp">using MfGames.Culture;
</code></pre>
<h1>CodeManager</h1>
<p>You may have noticed that I introduced <code>CodeManager</code> while dropping the <code>Instance</code> properties of <code>LanguageCodeManager</code>, <code>CountryCodeManager</code>, and <code>ScriptCodeManager</code>. I had two reasons for this. The first is I was getting a lot more code managers into the system and it was getting overwhelming. The second is <a href="http://en.wikipedia.org/wiki/Single_responsibility_principle">Single Responsibility Principle</a> (SRP). <code>CodeManager</code> is purely to handle singleton management of the code managers; this pulls that logic out of the individual managers and keeps their functions pure.</p>
<h1>ISO 3166</h1>
<p>Like the language codes, I needed country codes for some later functions. I'm also using standard codes, in this case <a href="http://en.wikipedia.org/wiki/ISO_3166">ISO 3166</a> which defines standard two- and three-character abbreviations for countries. For example, the Republic of Turkey has a two-character code of <code>TR</code> and a three-character one of <code>TUR</code>. Country codes are typically uppercase.</p>
<p>In addition, 3166 defines a numerical code for systems that don't have character-based codes.</p>
<p>Country codes are in the <code>CountryCode</code> class and work much like <code>LanguageCode</code> when it comes to case comparison.</p>
<pre><code class="language-csharp">CountryCode turkey1 = new CountryCode("TR", "TUR");
CountryCode turkey2 = new CountryCode("tr", "tur");
Assert.AreEqual(turkey1, turkey2);
</code></pre>
<p>The <code>ToString()</code> method prefers two-character codes over three.</p>
<pre><code class="language-csharp">CountryCode turkey1 = new CountryCode("TR", "TUR");
CountryCode turkey2 = new CountryCode("tr", "tur");
CountryCode turkey3 = new CountryCode(null, "tur");
Assert.AreEqual("TR", turkey1);
Assert.AreEqual("TR", turkey2);
Assert.AreEqual("TUR", turkey3);
</code></pre>
<h1>Country Code Manager</h1>
<p>The real power of the system comes from <code>CountryCodeManager</code> which provides a single place of access for the known country codes. You can get a <code>CountryCodeManager</code> via <code>CodeManager</code> or directly.</p>
<pre><code class="language-csharp">ICountryCodeManager countries1 = CodeManager.Instance.Countries;
ICountryCodeManager countries2 = new CountryCodeManager();
</code></pre>
<p>A default set of countries (basically the known list) is embedded into the DLL as a resource. For the <code>CountryCodeManager</code> loaded in the default <code>CodeManager</code>, it is already loaded but for new instances, the <code>AddDefaults()</code> method needs to be called.</p>
<pre><code class="language-csharp">ICountryCodeManager countries2 = new CountryCodeManager();
countries2.AddDefaults();
</code></pre>
<p>Retrieving codes is pretty simple. The <code>Get</code> method tries all known codes while <code>GetIsoAlpha2</code> only checks the two-character codes. I did this to allow for a generic system (<code>Get</code>) or something being specific. If a code cannot be found, this returns null.</p>
<pre><code class="language-csharp">CountryCode turkey1 = countries.Get("TR");
CountryCode turkey2 = countries.Get("TUR");
CountryCode turkey3 = countries.Get("792");
CountryCode turkey4 = countries.Get(792);
CountryCode turkey5 = countries.GetIsoAlpha2("TR");
CountryCode turkey6 = countries.GetIsoAlpha3("TUR");
CountryCode turkey7 = countries.GetIsoNumeric(792);
</code></pre>
<h1>What's Next</h1>
<p>The following topics are coming up.</p>
<ul>
<li>Script codes, <a href="http://en.wikipedia.org/wiki/ISO_15924">ISO 15924</a>.</li>
<li><a href="http://en.wikipedia.org/wiki/IETF_language_tag">IEFT Language Tags</a> which is build on these three codes.</li>
<li>Language Selector, based on HTTP Accept-Language.</li>
</ul>
What Is Going On Now?2015-02-25T06:00:00Zhttps://d.moonfire.us/blog/2015/02/25/what-is-going-on/It's been a few weeks since I last posted. When I work on projects, I have a tendency to go silent because I could either post or I could work on it. But, here is what is going on.<p>As my lovely spouse as mentioned on more than one occasion, I always have a project going on. Usually two or three or even more. Lately, I feel that I'm starting a lot of projects and not actually finishing anything. With some of my recent projects, that may change since I actually see a finishing point on both <a href="/tags/mfgames-culture-cil/">MfGames Culture CIL</a> and <a href="/tags/sand-and-bone/">Sand and Bone</a>.</p>
<h1>MfGames Culture CIL</h1>
<p>On the programming side, I'm working on a C# library called MfGames Culture CIL (all of my .NET libraries end in “CIL”). This is the fantasy calendar system I've <a href="/blog/2015/02/13/mfgames-culture-api-introduction/">mentioned before</a>. The project is somewhat ambitious for me, which makes it scary, but also something I've been thinking about doing for a number of years. I also think it is something I can actually finish and make into a proper library.</p>
<p>I hit a pretty good point with the library. It now handles cultures, multiple calendars, translations, and a combination of a number of cool things. Everything is hard-coded in C# code, which I'm planning on solving with the next few steps. In the end, it should be completely driven off XML or JSON files.</p>
<p>I'm tracking my “to do” lists toward a 1.0.0 release over on <a href="https://github.com/dmoonfire/mfgames-culture-cil/issues">Github</a>.</p>
<h1>Sand and Bone</h1>
<p>After a number of deaths I've had to deal with in the last few months, I finally got a chance to go back to my writing group. When I had to take some time off at the end of last year, there was a huge group of submissions. Now, not so much. Which means I had a chance to get <em>Sand and Bone</em> finished.</p>
<p>The last fifteen chapters were still very rough since I wrote almost all of them in a single week. I took the last week to do another round of editing against it, cleaning it up and making it ready for the group.</p>
<p>After this week, I have five more submissions and then it should be done with that round of editing. After that, a pair of rounds of professional editors and it will be ready for being published.</p>
<h1>Sand and Ash</h1>
<p>If I mention <em>Sand and Bone</em>, I probably should talk about <a href="/tags/sand-and-ash/">Sand and Ash</a>. I've gotten a number of beta readers for the sequel to <a href="https://sand-and-blood.fedran.com/">Sand and Blood</a> and it has been pretty positive. I'm excited about getting it done, but…</p>
<h1>WisCon</h1>
<p>I had a choice between getting <em>Sand and Ash</em> out or going to <a href="http://wiscon.info/">WisCon</a>. I choose WisCon because it gives me a chance to visit my grandmother (she lives in town), reduces a long haul for a family trip, and gives me a chance to meet up with some friends.</p>
<p>This year I'm going to try doing a reading. We'll see if I get in, but it will be the first time I've read something I've written since I've done poetry readings in Illinois. I'm thinking about doing a chapter from <a href="/tags/flight-of-the-scions/">Flight of the Scions</a>, mainly because the entire world was inspired by panels from that convention.</p>
<p>I'll find out how to get <em>Sand and Ash</em> out this year, hopefully by <a href="http://iowa-icon.com/">ICON</a>. I'm going to sign up for doing a reading there too, probably because I'm hoping to be a lot more comfortable with reading my novel.</p>
<p>I'm also assuming I can <em>pronounce</em> my character names smoothly by then.</p>
<h1>Coming weeks</h1>
<p>I'm hoping to finish MfGames Culture CIL in a week or so. And then I'm sure I'll move on to another project.</p>
<p>There is always another project.</p>
MfGames.Culture API - Language Codes2015-02-17T06:00:00Zhttps://d.moonfire.us/blog/2015/02/17/mfgames-culture-api-languages/The first part of MfGames Culture is the implementation of ISO language codes.<p><em>This is the first part of a short series on the MfGames Culture CIL API. It is currently alpha software, but I'm looking for critiques, opinions, and general feedback. All of my work for this is in the <a href="http://github.com/dmoonfire/mfgames-culture-cil/">Github repository</a> in the <code>drem-0.0.0</code> branch. It is licensed with MIT.</em></p>
<p><em>This page is also a form of documentation by example.</em></p>
<p>When I started working on the culture logic, I decided to hang the code off as many standards as possible. I was very familiar with <a href="http://en.wikipedia.org/wiki/ISO_639">ISO 639</a>. ISO 639 is a standardized list of languages and codes to identify them. You can see these in various programs and places such as <code>en</code> or <code>fr</code> (English and French respectively).</p>
<h1>Links</h1>
<ol>
<li><a href="/blog/2015/02/13/mfgames-culture-api-introduction/">Introduction</a></li>
<li>Language Codes</li>
<li><a href="/blog/2015/03/04/mfgames-culture-api-countries/">Country Codes</a></li>
</ol>
<h1>ISO 639</h1>
<p>There are a few components to the ISO 639 code:</p>
<ul>
<li>A two character code (<code>en</code> and <code>fr</code>).</li>
<li>A three character version (<code>eng</code>).</li>
</ul>
<p>Actually, there are two versions, a <em>bibliographic</em> and a <em>terminologic</em> code. These are known as the <code>B</code> and <code>T</code> codes respectively. The bibliographic code is based on the English translation of the name while the terminologic is based on the language's name for itself.</p>
<p>For example, the bibliographic code for Armenian is <code>arm</code> while the terminologic is <code>hye</code>.</p>
<p>According to Wikipedia, the terminologic is the preferred over the bibliographic.</p>
<p>Also, <code>en</code> and <code>eng</code> are identical codes, but if you treat them simply as a string, they are different.</p>
<h1>System.Globalization</h1>
<p>To my surprise, there is no dedicated object in the base library for C# for ISO 639 codes. There is some properties in <code>System.Globalization</code> on <code>CultureInfo</code>, but nothing that handles the equivalency of <code>en</code> and <code>eng</code>. And I haven't had a lot of success with creating non-standard languages (my <code>xmi</code> for Miwāfu) inside the framework.</p>
<p>There are some enum versions of the ISO code, but they don't have the flexibility to add custom languages.</p>
<p>Unable to find something already there, I created my own ISO 639 class for handling these codes. I called it <code>LanguageCode</code> because I didn't like how <code>Iso639</code> looked. It does ignore the other standards for languages right now, but I was thinking that <code>LanguageCode</code> could handle all of those as separate properties.</p>
<pre><code class="language-csharp">var english1 = new MfGames.Culture.Codes.LanguageCode("eng");
var english2 = new MfGames.Culture.Codes.LanguageCode("eng", "en");
Assert.AreEqual(english1, english2);
</code></pre>
<p>I set it up so the <code>ToString</code> translates into the preferred three-character code.</p>
<pre><code class="language-csharp">var armenian = new LanguageCode("hye", "hy", "arm");
Assert.AreEqual("hye", armenian.IsoAlpha3);
Assert.AreEqual("hye", armenian.IsoAlpha3T);
Assert.AreEqual("arm", armenian.IsoAlpha3B);
Assert.AreEqual("hy", armenian.IsoAlpha2);
Assert.AreEqual("hye", armenian.ToString());
</code></pre>
<p><code>LanguageCode</code> is an immutable object that encapsulates all the properties of an ISO 639 code except for its name. It also compares against the preferred three-character code for equivalency. I also had it intern the strings to avoid memory pressure with larger number of codes.</p>
<h1>Memory</h1>
<p>Memory is something I concern myself with. With a single code, you have:</p>
<ul>
<li>The pointer to the code</li>
<li>The class overhead for LanguageCode</li>
<li>Three pointers to strings</li>
<li>Three strings in memory.</li>
</ul>
<p>Using an interned string for the code means that the three pointers will remain, but at least I won't have a huge number of three- and two-character strings in memory.</p>
<h1>Singleton</h1>
<p>I still wanted to potentially reduce the memory pressure even further. To do this, I created a singleton class <code>LanguageCodeManager</code> which provides a singleton access to the LanguageCode.</p>
<pre><code class="language-csharp">var manager = LanguageCodeManager.Instance;
var english1 = manager.Get("eng");
var english2 = manager.Get("en");
var english3 = manager.GetIsoAlpha3("eng");
var english4 = manager.GetIsoAlpha3T("eng");
</code></pre>
<p>This way, you'll only have one instance of “English” regardless of how many pointers you use. Of course, if you also decide to manually create an English tag, it will continue to compare against the singleton version even though it is a separate object.</p>
<p>I made <code>LanuageCodeManager</code> an injectable singleton to provide for customizations.</p>
<pre><code class="language-csharp">LanguageCodeManager.Instance = new LanguageCodeManager();
LanguageCodeManager.Instance.Add(new LanguageCode("xmi")); // Miwāfu
LanguageCodeManager.Instance.Add("xlo"); // Lorban
Assert.AreEqual(2, LanguageCodeManager.Instance.Count);
foreach (LanguageCode lc in LanguageCodeManager.Instance)
{
Assert.IsNull(lc.IsoAlpha2);
}
</code></pre>
<p>This also means that most methods that use language codes actually take a <code>LanguageCodeManager</code> as a parameter to facilitate testing and isolation. So far, I found that this adds a bit of overhead with many functions but I think it gives the flexibility needed; I'm in the process of converting most of those to argument objects to simplify the process.</p>
<p>The default <code>LanguageCodeManager</code> does not have any of the ISO codes. It is an empty list of codes. To add the ones stored as a manifest resource, you can use <code>AddDefaults()</code> to include them. The initially created <code>LanguageCodeManager</code> has these defaults already added.</p>
<h1>Why a class?</h1>
<p>I decided to make <code>LanguageCode</code> a class despite the overhead of the class mainly to make it easy to pass <code>null</code> in. Also because if I used a struct, then the item would have at least three string pointers everywhere it is used instead of a single one.</p>
<h1>Why not string?</h1>
<p>The main reason I just didn't leave this as a string is because of type-safety. I like passing in a language code when it is suppose to be a language code and not worry that one of the five different strings is suppose to be the three-character code. Or if it is suppose to be a two-character. Or something else.</p>
<pre><code class="language-csharp">var english = LanguageCodeManager.Instance.Get("eng");
var translation = GetTranslation(english, "bob");
</code></pre>
<h1>Special</h1>
<p>There is one <code>LanguageCode</code> that doesn't fit with the ISO standard, “Canonical”. This has a code of <code>*</code> for all of the fields and is used to do the final matching or determine the canonical name of something.</p>
<pre><code class="language-csharp">var canonical = LanguageCode.Canonical;
Assert.AreEqual("*", canonical.IsoAlpha3);
</code></pre>
<h1>Names of Languages</h1>
<p>One aspect of the language code that is not included in the object is the name of the language. This led into one of the more complicated parts of the library, and one of the ones I'm most unsure about, but that requires me to have country codes and language tags to explain.</p>
<h1>Self-review</h1>
<p>An interesting aspect about writing up this page is that I found things wrong with my API. For example, I had <code>LanguageCode.Alpha3</code> when it really should have been <code>LanguageCode.IsoAlpha3</code>. It is a simple change, but writing this was a way of stepping back and looking over it again.</p>
MfGames.Culture API - Introduction2015-02-13T06:00:00Zhttps://d.moonfire.us/blog/2015/02/13/mfgames-culture-api-introduction/An introduction to the alpha design API of MfGames.Culture CIL, a C# library for handling arbitrary cultures.<p>Lately, I've been creating a C# library for handling arbitrary cultures. There is already a built-in system for this (<code>System.Globalization</code>), but I was hitting up against limitations that I couldn't figure out how to work around: I'm writing for cultures that doesn't exist. It is not easy to add a new country, language, or calendar into the API. That makes sense, the designers of C# probably weren't worried about people into world-building or conlangs, or having calendars that don't follow the standard “year, month, day” (YMD) cycle of the modern world.</p>
<p>But, in fantasy, there is a lot more variance from the YMD. Both <a href="http://rpg.wikia.com/wiki/Exalted_Calendar">Exalted's</a> and <a href="http://en.wikipedia.org/wiki/Middle-earth_calendar">Tolkien's</a> have special weeks or days in them. There are calendars like <a href="/blog/2013/09/21/mansupi-tachira-ripochya-solar-calendar/">mine</a> that use fall days instead of leap days. Or you have large cycle calendars, like the <a href="http://en.wikipedia.org/wiki/Aztec_calendar">Aztec</a> with the Long Count. Yeah, the last one isn't fantasy, but it doesn't quite follow the YMD pattern and, therefore, doesn't show up in a lot of programming APIs.</p>
<p>In sci-fi, calendars may be more standardized on the YMD, but the length of the days are different. A day on Mars is forty minutes longer and a day on Jupiter is just under ten hours (thanks Google). I wouldn't be surprised if author's other worlds have similar traits. I want those calendars to also work and still be able to correspond them to a point on time here on Earth.</p>
<p>You also have this in fantasy, such as D&D where time travels differently on the astral and other planes.</p>
<h1>Links</h1>
<ol>
<li>Introduction</li>
<li><a href="/blog/2015/02/17/mfgames-culture-api-languages/">Language Codes</a></li>
<li><a href="/blog/2015/03/04/mfgames-culture-api-countries/">Country Codes</a></li>
</ol>
<h1>My initial goals</h1>
<p>When I write my novels and stories, I have a bunch of metadata up in the header of each chapter. I'm planning on keeping this for <a href="/tags/author-intrusion/">Author Intrusion</a> and it is a system that I'm pretty happy with.</p>
<p>For example, chapter two of <a href="https://sand-and-blood.fedran.com/">Sand and Blood</a> has these entries (among others):</p>
<pre><code>----
title: Confession
when: xmi,1471/3/28
time: 40 min
----
</code></pre>
<p>(There is some formatting issues here, sorry about that.)</p>
<p>The <code>when</code> tag tells me when in the world this event happened. I wrote it in the in-world calendar because I can use that to create a timeline. The <code>time</code> element is for how long the chapter is; this is a rough estimate favoring the long side, but mainly is intended to give me a block of time that I can eventually display on my <a href="https://fedran.com/">website</a>.</p>
<p><a href="http://xkcd.com/1179/"><img class="img-responsive post-img-link" src="http://imgs.xkcd.com/comics/iso_8601.png" alt="XKCD ISO 8601 Dates"/></a></p>
<p>The <code>xmi,</code> prefix on the date is just to avoid any parsing issues with the date. One of the largest problems I've encountered with date formatting is that many cultures use the same separators and numbers in each field, but they have different meanings. For example, in Europe “3/4/2015” means April 3rd but in the US it means March 4th. I'm planning on having this same conflict in my world, so I'm giving myself an out to avoid it in internal situations where I don't want to suffer.</p>
<h1>Using standards</h1>
<p>When I first decided to create a calendar, I seriously considered writing a Fedran-specific one. But, that doesn't seem to be my nature. Instead, I went with trying to use as many existing standards as possible (assuming I was aware of them) to represent my world.</p>
<p>Many readers may be familiar with <code>en-US</code>, <code>en-GB</code>, or <code>fr-CA</code> (U.S. English, British English, and Canadian French). I prefer the three-letter versions (e.g., <code>eng-US</code>) but that is less important. The code itself is an <a href="http://en.wikipedia.org/wiki/IETF_language_tag">IEFT Language Tag</a> which is based on both the <a href="http://en.wikipedia.org/wiki/ISO_639">ISO 639 Language Code</a> and the <a href="http://en.wikipedia.org/wiki/ISO_3166">ISO 3166 Country Codes</a>.</p>
<p>The specification for all three are pretty well known, but more importantly, they support non-standard versions. That way, I can say the language of <a href="/tags/miwafu/">Miwāfu</a> is <code>xmi</code> and the country of Kyōti is <code>QKY</code>.</p>
<h1>How its used</h1>
<p>It's a small thing, but I use these codes in my writing.</p>
<pre><code>Kanéko looked over at Garèo, struggling with the words in her head before she spoke. "t-xmi: Are you `eng: sure`?".
</code></pre>
<p>When it is rendered by the formatter, I get this:</p>
<blockquote>
<p>Kanéko looked over at Garèo, struggling with the words in her head before she spoke. "<em>Are you</em> sure?".</p>
</blockquote>
<p>Now, this is a violation of the IEFT tag. It should be <code>en-t-xmi</code> or <code>eng-t-xmi</code> to say “Miwāfu translated into English” or notionally translated. I'm going to support <code>t-xmi</code> simply for typing reasons just as I'm using <code>*</code> to indicate the native language of the book.</p>
<p>If she was actually speaking the language and the POV character didn't understand it, I'd have this:</p>
<pre><code>Mioráshi muttered, "xmi: oe nyozerìku," before stalking away.
</code></pre>
<p>Which will still render as:</p>
<blockquote>
<p>Mioráshi muttered, “<em>oe nyozerìku</em>,” before stalking away.</p>
</blockquote>
<p>But it would let me see where I write in the conlang and translate it based on characters. It also lets me use the language to figure out formatting since I use guillemets for telepathy.</p>
<h1>The library</h1>
<p>I'm probably over-engineering this, but it ties into my work with Author Intrusion, this library, and my ebook formatting. So, it fits my overall goals and I don't think there is an open-sourced library (or a <code>System</code> namespace) that quite fits my uses of it (the ability to add unknown languages and countries into the system).</p>
<p><a href="http://xkcd.com/974/"><img class="img-responsive post-img-link" src="http://imgs.xkcd.com/comics/the_general_problem.png" alt="XKCD The General Problem"/></a></p>
<p>Right now, the library consists of:</p>
<ul>
<li>ISO Language Codes</li>
<li>IEFT Language Tags (missing a lot)</li>
<li>ISO Country Codes</li>
<li>Language Selectors (based on the HTTP Accept-Language header)</li>
<li>Translation Management</li>
<li>Arbitrary Calendars</li>
</ul>
<p>Things I still have left:</p>
<ul>
<li>XML and JSON reading</li>
<li>Handling Multiple Culture</li>
<li>CLI Tool</li>
</ul>
<h1>Critiques</h1>
<p>An interesting aspect about being both a writer and programmer is that I see things from different angles. One of the differences is asking for critiques. In writing, you can beg friends to do it, post on website, go to a writing group, or a number of other things. There isn't the same thing for programming and creating APIs. I haven't found a good place to see if the signatures make sense, the organization is insane, or there was already a library that does everything and more but I just missed it.</p>
<p>In the end, I'm going to write a short series of posts about the library, not only to let search engines know what I'm trying but also to see if I can get opinions about implementation, API surface, and the little things that I may have missed while I'm trying to get a calendar functionality built up.</p>
<p>All of my work for this is in the <a href="http://github.com/dmoonfire/mfgames-culture-cil/">Github repository</a> in the <code>drem-0.0.0</code> branch. It is licensed with MIT and is all alpha quality, it works and I'm doing TDD for a bulk of the writing.</p>
Meandering This Week2015-01-26T06:00:00Zhttps://d.moonfire.us/blog/2015/01/26/meandering/I spent the last few weeks getting things off my to do list so I could focus on Author Intrusion, but it looks like I need another week or so before that happens.<p>Over the last few weeks, I've been trying to knock things off my to do list so I could focus on <a href="/tags/author-intrusion/">Author Intrusion</a>. I thought I was ready to switch over, but then I realized I needed to focus on something big this week at work and I'm stick. So, it would be better to work with smaller items for the time begin. So, this week is another to do week.</p>
<h1>Sick Again</h1>
<p>I was perfectly fine when I headed out Thursday night. By the time I got home, I wasn't. In a few short hours, I got nailed by a nasty head cold which has made everything difficult in general. Thankfully, I have a wonderful spouse who lets me sleep, but I still had to get other things done.</p>
<h1>SSL</h1>
<p>I managed to switch the <a href="https://journals.fedran.com/">Journals of Fedran</a> website over to SSL thanks to <a href="http://startssl.com/">StartSSL.com</a>. I know that there isn't nearly enough there for privacy concerns, but I believe that most sites should be SSL these days and thankfully, I don't have to pay a couple hundred a year to get it.</p>
<h1>Editing</h1>
<p>Probably the best thing that came out of last week was going over edits for <a href="https://journals.fedran.com/issue-00/under-the-streets/">Under the Streets</a> and <a href="https://journals.fedran.com/issue-00/raging-alone/">Raging Alone (Part 1)</a>. Both of these ended up being rather drastic reworking, mainly because of a beta reader pointing out a few flaws.</p>
<p>I've uploaded new versions to the website, but I don't really expect anyone to read them. Hope, yes, but until I get these edited, I'm not really pushing them.</p>
<p>A question came up during the edit, but I'm going to make that a different post.</p>
<h1>MfGames Culture</h1>
<p>Since I can't focus for <em>Author Intrusion</em> (including the Markdown parser), I decided to work on little things. One of them is to hopefully figure out a system for managing dates in my little <a href="https://fedran.com/">fantasy world</a>.</p>
<p>Dates and calendars are a nasty little thing, mainly because so much code is written to handle the Gregorian calendar but not other calendars (Islamic for example) or a fictional one. I, on the other hand, use different cultures when I write and I'd like the ability to tag various elements with the “in world” date but still have the information to put them on a time line or figure out when one chapter is before another.</p>
<p>I started <a href="/tags/mfgames-culture/">MfGames Culture</a> quite a few years ago with the idea of writing a library for that, but it was never a high priority and it never went anywhere. Eventually, I planned on using it in <em>Author Intrusion</em> so I should the difference of times between chapters without faking it via the Gregorian calendar.</p>
<p>Most of the system is based on Julian Days (JD) as a generic baseline (it is just a decimal, so its easy to work with). The plan is just to get the <a href="/blog/2015/01/01/tarsan-standard-calendar/">two</a> <a href="/blog/2013/09/21/mansupi-tachira-ripochya-solar-calendar/">calendars</a> written up and capable of parsing the formats into JD and back again.</p>
Journals of Fedran covers2014-12-17T06:00:00Zhttps://d.moonfire.us/blog/2014/12/17/journals-covers/One of the hardest parts is coming up with a visual style for something. But, using the inspiration for Journals, I have come up with the covers for the issues and the individual stories.<p>When it comes to book covers, I lean toward typographic and abstract designs. Something that hints at the contents instead of painting an illustration or using a photograph.</p>
<p>This is a contrast to the bulk of the books in my library which are the opposite and why <a href="https://sand-and-blood.fedran.com/">Sand and Blood</a> has the cover that it does. That and I know that my severe style for covers doesn't appear to be that popular.</p>
<p>For <a href="/tags/journals-of-fedran/">Journals of Fedran</a>, I'm going with a typographic cover based on newspapers. While I wanted to do the entire thing like a newspaper, I don't think reading a four thousand word story in nine point font in four columns would be that appealing. Not to mention that ebooks can't be formatted that way, so it would be a moot point. A cover, on the other hand, can give the impression of being a newspaper and still get that “feel” that I'm going for.</p>
<h1>The Issue Cover</h1>
<p><a href="issue-00.jpg"><img class="post-img-link" style="float: left; margin-right: 1em;" src="thumb-issue-00.jpg" height="256" width="180" /></a></p>
<p>The first thing I worked on was the issue cover. I used a program called <a href="http://www.xelatex.org/">XeLaTeX</a> which is a programming language for typesetting books and documents. It can also do some pretty impressive things when it comes to laying out PDFs in general.</p>
<p>Even though most of the examples of newspapers from the 1800s didn't have teasers on the front page, I started with lead-ins for the pieces, but that didn't work. Eventually, I ended up on the first couple of paragraphs of each story. It ends up being an impromptu table of contents.</p>
<p>I'm planning on putting some images on the page, mainly a little advertising, to help with thumbnail recognition, but that's going to be a later step.</p>
<p>About half way through making the cover, I switched from hand-writing LaTeX to writing a program to pull in the paragraphs and sections. Written in Perl, this scans the source file for the project, pulls out the ones that are significant (based on metadata in the YAML markdown), and then inserts them into place. The advantage of this is that I don't have to update the cover manually when I edit the pieces. Just type <code>make</code> and the covers will be refreshed.</p>
<p>Overall, I think it looks pretty good for what I'm aiming for.</p>
<h1>Individual Stories</h1>
<p><a href="farimons-revelation.jpg"><img class="post-img-link" style="float: left; margin-right: 1em;" src="thumb-farimons-revelation.jpg" height="256" width="180" /></a></p>
<p>One of the things I'm toying with is allowing individual downloads of the stories. If I do that, then I need a distinctive cover for each one without getting an illustrator. Since I have the issue format, I took bits and pieces of that and created something more specific.</p>
<p>With the story, the lead-in worked out pretty well. I seem to have consolidated down to eight lines of text, all proclaiming the contents. Overall, when lined up, I think they look pretty cool.</p>
<p>I did keep the header and footer in there (though the author can change). That way, all of the issue zeros will be tied together.</p>
<p>Like the issue, this is based on a Perl program for generating. That keeps it up to date if I have to make changes in the first <em>X</em> paragraphs at the bottom, but also lets me update them in a single shot. Most of the leading data is just YAML at the beginning.</p>
<pre><code>---
Author: D. Moonfire
cover:
leading:
- A barbarian child
- Burning with rage!
- A helpless brother
- on the brink of death!
- Abandoned in the desert
- And left to die!
- See the barbaric rituals
- Of a foreign land!
Title: Raging Alone
---
</code></pre>
<p>Putting it in the metadata allows me to change the lead-in text, remove the capitals, or other minor changes. It also keeps the data for the story in a single file instead of spreading it to a control program.</p>
<p><a href="raging-alone.jpg"><img class="post-img-link post-img-link-tile" src="thumb-raging-alone.jpg" height="256" width="180" /></a>
<a href="the-cross-gambit.jpg"><img class="post-img-link post-img-link-tile" src="thumb-the-cros-gambit.jpg" height="256" width="180" /></a>
<a href="samus-and-the-savage-slasher.jpg"><img class="post-img-link post-img-link-tile" src="thumb-samus-and-the-savage-slasher.jpg" height="256" width="180" /></a>
<a href="under-the-streets.jpg"><img class="post-img-link post-img-link-tile" src="thumb-under-the-streets.jpg" height="256" width="180" /></a>
<a href="a-cup-of-soup.jpg"><img class="post-img-link post-img-link-tile" src="thumb-a-cup-of-soup.jpg" height="256" width="180" /></a>
<a href="second-hand-dresses.jpg"><img class="post-img-link post-img-link-tile" src="thumb-second-hand-dresses.jpg" height="256" width="180" /></a>
<a href="simple-goren.jpg"><img class="post-img-link post-img-link-last" src="thumb-simple-goren.jpg" height="256" width="180" /></a></p>
<h1>Series</h1>
<p>There are pair of series pieces in here (<em>Raging Alone</em> and <em>Second-Hand Dresses</em>). These will eventually have their own ebooks and downloads, with a cover that doesn't reference a single issue but will probably still have the same lead-in and title as the others.</p>
<h1>Licensing</h1>
<p>Like when I was about to release <em>Sand and Blood</em>, I'm considering <a href="http://creativecommons.org/">Creative Commons</a> for this. It is a scary idea, but part of me has been asking to make something CC-licensed for years. I already get so much from the Commons and I want to contribute something back.</p>
<p>The other reason is <a href="http://craphound.com/littlebrother/about/">Cory Doctrow's</a> discussion about his book, Little Brother:</p>
<blockquote>
<p>Giving away ebooks gives me artistic, moral and commercial satisfaction. The commercial question is the one that comes up most often: how can you give away free ebooks and still make money?</p>
<p>For me – for pretty much every writer – the big problem isn't piracy, it's obscurity (thanks to Tim O'Reilly for this great aphorism). Of all the people who failed to buy this book today, the majority did so because they never heard of it, not because someone gave them a free copy. [..]</p>
</blockquote>
<p>Those two paragraphs has been hanging around me since he spoke at ICON a few years ago. And I think <em>Journals</em> might be a good fit for that. At least for a few issues.</p>
<p>Cory sells print versions of the book even though he gives away the ebook. I'm thinking about doing the same, make a print version and throw up a donation link, but otherwise just let the stories go wild and see what happens.</p>
<p>That led into the effort to make individual covers for the stories.</p>
Messing with Markdown2014-11-23T06:00:00Zhttps://d.moonfire.us/blog/2014/11/23/messing-with-markdown/I spent the last few weeks working on the beginning of an event-based Markdown parser. I know why, but I've fallen into a rabbit hole that threatens to swallow me.<p>I have a lot of projects in my head, which is one reason why I try to avoid starting new things while working on others. It got too easy to start a project and then never finish it when I got bored or just moved on.</p>
<p>At the same time, many of my ideas don't “stick.” They seem like they will, but then it either doesn't work out, the ideas don't really gel, or simply I lose the passion. Some of them, like my <a href="https://mfgames.com/comics/glorious-saber/">old Exalted webcomic</a>, haunt me for years with the nagging voice of “finish me” but I keep not doing it. Others end up being bigger projects, but suffer through constant revisions as I try to figure it out.</p>
<p>I do have some major projects that I haven continued to update and maintain quite a few years after I start them. One of the biggest is <a href="https://github.com/dmoonfire/mfgames-writing-python">MfGames Writing Python</a> which I started in 2010 when my father said I'd love Python. Even though I've long-since decided that <a href="/blog/2012/10/17/two-years-of-python/">I don't like Python</a>, I've still been writing updates to the tools in support of my own writing efforts.</p>
<p>Like <em>Glorious Saber</em>, I've been thinking about converting my Python writing tools over to C#, but never really got to it. I had a couple stabs at it but nothing really “stuck.”</p>
<h1>The Side Project</h1>
<p>After ICON, something <a href="http://www.jimchines.com/">Jim C. Hines</a> and <a href="http://www.scottlynch.us/">Scott Lynch</a> said during their “Beyond SF 101” panel was still echoing in my head.</p>
<blockquote>
<p>Write more words and don't be a dick.</p>
</blockquote>
<p>Now, <a href="/tags/sand-and-ash/">Sand and Ash</a> is still stuck in limbo, so I couldn't really do anything with that. So I had this idea for a writing project that would be complimentary to my <a href="https://fedran.com/">Fedran</a> world and let me get different ideas out. It involved smaller pieces, short stories and essays and lessons, so I was trying to figure out how to pull it all together.</p>
<p>For some reason, my Python tools just choked. They are optimized toward writing novels and single-file <a href="http://www.docbook.org/tdg5/en/html/docbook.html">DocBook 5</a> files, but not a multitude of smaller DocBook XML files which would these individual pieces.</p>
<p>I started to look into my C# version, which had a different “gather” utility and, once again, realized that I probably wasn't <em>that</em> far off from getting the C# version working with its flaws and maybe move away from the Python.</p>
<p><em>One of my major difficulties with Python is that it doesn't handle UTF-8 characters natively. I seem to have a lot of non-ANSI characters in my desert world (macros are a killer) and it kept choking on them.</em></p>
<h1>MfGames Writing CIL</h1>
<p>So I shifted from my side project to <a href="https://github.com/dmoonfire/mfgames-writing-cil">MfGames Writing CIL</a> which is the C# version of the Python tools “plus” additional functionality. The biggest is that I wrote the Python around <a href="http://www.wikicreole.org/">Creole</a> instead of <a href="http://en.wikipedia.org/wiki/Markdown">Markdown</a>, which is the markup language I've migrated my writing to.</p>
<p>While I was working with writing up a Markup conversion utility for the tools, I came upon <a href="http://commonmark.org/">CommonMark</a> which is an attempt at a well-documented specification. I figured I could use that to help guide my effort on the conversion utility.</p>
<p>It didn't take long before I realized that I was duplicating my work with <a href="/tags/author-intrusion/">Author Intrusion</a> for handling Markdown. Usually when that happens, I figure I should start up a new project to handle the common logic and write it once.</p>
<h1>MfGames Text Markup CIL</h1>
<p>And then I moved from <em>MfGames Writing CIL</em> to <a href="https://github.com/dmoonfire/mfgames-text-markup-cil">MfGames Text Markup CIL</a>. This is an attempt to create a single, centralized reader (and eventually writer) of markup languages in general and Markdown in specific.</p>
<p>There were a couple reasons I went this project instead of another library:</p>
<ul>
<li>Most Markdown libraries only convert to HTML, which meant I was parsing HTML to get into Author Intrusion or DocBook format. I wanted something that had an intermediate output that was ideal for converting to other formats.</li>
<li>Again, most libraries seem to load the entire Markdown file into memory at once (mainly because of the deferred links) and then writing it out. I haven't tested this completely, but I already know that I have 640k word series that have to be parsed; I do not want to have this loaded into memory. This implies a callback interface (SAX verses DOM).</li>
</ul>
<p>I decided to write this in a similar style to C#'s XmlReader. Instead of loading everything into memory and then writing out the results, it just translates the Markdown file into element types.</p>
<pre><code>// Loop through the Markdown and process each one.
while (markdown.Read())
{
switch (markdown.ElementType)
{
case MarkupElementType.BeginDocument:
this.WriteBeginDocument(xml);
break;
case MarkupElementType.EndDocument:
xml.WriteEndElement();
xml.WriteEndDocument();
break;
case MarkupElementType.BeginMetadata:
case MarkupElementType.EndMetadata:
case MarkupElementType.BeginContent:
case MarkupElementType.EndContent:
break;
case MarkupElementType.BeginCodeSpan:
this.WriteForeignPhrase(markdown, xml);
break;
</code></pre>
<p>The system seemed to work out pretty well for my test cases, but when I started to throw “real” chapters at it, it started to crumble. Not from the foundation, but simply because I didn't write a good enough Markdown parser to translate them.</p>
<h1>CommonMark</h1>
<p>And here is where I started down the rabbit hole. The callback system worked great, but I needed to get my parser to be competent enough to handle what I wrote. Once I get that, converting to DocBook is trivial (as the above example probably shows).</p>
<p>A few days ago, I noticed that the <a href="https://github.com/jgm/CommonMark/blob/master/spec.txt">CommonMark spec</a> was a Markdown file with some magic for handling the input/output examples. Well, I could write a bunch of unit tests or… I could write a program that converted the 500+ examples into unit tests for me.</p>
<p><em>Why write things out by hand when I can write a program to do it for me?</em></p>
<p>Last night, I finished converting most of the unit tests over. Now, I just have to solve 508 unit tests, or at least 300 of them.</p>
<h1>Rabbit Holes</h1>
<p>I'm pretty sure this is going to be overwhelming but I think it will still further my goal of finishing up my writing tools and get back to my side project. We'll see how it ends up, but I still have a mountain to climb.</p>
<p>This is also the reason I haven't really posted for a few weeks to. I was lost in a rabbit hole.</p>
Reorganizing my Git writing repository2014-10-05T05:00:00Zhttps://d.moonfire.us/blog/2014/10/05/reorganization-git-story-repo/Last night, I completely reorganized my writing repository to get around some limitations of Git.<p>As some of you may know, I use <a href="http://git-scm.com/">Git</a> to organize my writing. After years of accidentally overwriting a good chapter with an old one or trying to coordinate changes from two separate machines, I got into source control for writing; it worked for programming, why not my novels?</p>
<h1>The submodule approach</h1>
<p>Well, I've had a couple iterations of trying to get the “perfect” Git setup for my writing. Earlier this year, I broke apart the novels into submodules but left the bulk of my writing in the main repository (called <code>stories</code>). This meant I had a <code>sand-and-blood</code>, <code>sand-and-ash</code>, and <code>sand-and-bone</code> repository as submodules in the appropriate location of the <code>stories</code> repo (dmoonfire/fedran, if you are curious).</p>
<p>My reasons came while I was working on <span class="missing-link" data-path="/tags/sand-and-blood">Sand and Blood</span> covers. Since I checked in as I went, the size of the repository quickly became too large for my website to handle. I could download up to 50 MiB repo without too much trouble, but when it got into the 900 MiB range, I couldn't clone the repository anymore.</p>
<p>I had already worked with <a href="http://git-scm.com/book/en/Git-Tools-Submodules">submodules</a> before, so I thought they would be a perfect thing for the novels. I spent a pair of nights pulling out the five current WIP novels into a submodule, mainly by cloning the repo and using various commands to carve them out. It also took a while because I have a <em>lot</em> of project branches (41 beyond <code>master</code>) which represent every work-in-progress or semi-completed work I've done. Pulling out binaries from every branch was a painful process to say the least.</p>
<p>The submodule approach worked out fairly well, but I quickly found out some of its limitations. Because of how Git implements submodules, its inevitably shows up in other branches. It also has additional work.</p>
<p>To give an example. Assume I'm on my <code>sand-and-ash</code> branch and I'm happily working in the <code>dmoonfire/fedran/sand-and-ash</code> directory making changes. When I'm done, I've committed them and pushed up.</p>
<p>When I got up a level, to <code>dmoonfire/fedran</code>, I have to do a second commit to commit the submodule's position in the <code>stories</code> repository. It was a little extra work, but it kept the two isolated.</p>
<p>The real problem came when I switched to the <code>sand-and-blood</code> branch. The directory <code>dmoonfire/fedran/sand-and-ash</code> is still there and pointing to a repoistory (the <code>sand-and-ash</code> one), but I have to tell the <code>sand-and-blood</code> branch about it, otherwise it will show as an untracked file.</p>
<p>My two choices were to either add the <code>dmoonfire/fedran/sand-and-ash</code> directory to the <code>.gitignore</code> file of the <code>sand-and-blood</code> branch. (Okay, there are a lot of filenames in this post, sorry about that.)</p>
<p>The other approach is to add the submodule to the other branches so they didn't show as changes. Which worked until I made another change to the submodule and then I had to update it on <em>every</em> other branch to reflect the changes.</p>
<h1>Isolating covers instead</h1>
<p>Last night, I got tired of jumping through the hoops of submodules. I realized the entire reason I wanted to isolate the novels was to handle the covers. So, I decided to make a <code>covers</code> repository instead, put it into the root of the <code>stories</code> working directory and then add it to the <code>.gitignore</code>. This means that the <code>stories</code> repository doesn't officially know about the <code>covers</code> repository, but I can still reference it via soft links into <code>covers</code>.</p>
<p>The advantage of this approach is all the writing (actual words) are still managed in the same repository. This means when I switch branches, the stuff in <code>sand-and-ash</code> branch (not repo now) goes away until I go back. There isn't any cruft that drags on between the individual branches that has nothing to do with the current branch.</p>
<p>It isn't very elegant to have covers separated, but I only need covers when I'm formatting ebooks.</p>
<h1>Losing history</h1>
<p>One of the side effects of breaking apart the repository and pulling them back together is that I'm losing history data. I kept most of the commit histories intact, but now I can't really do a graph of total words written over a month or time. Since I can't tell if anyone actual read my posts when I documented them, I decided to accept that lose.</p>
<h1>BFG</h1>
<p>I mentioned that splitting apart the repositories was a lot of work. When I combined them back together, I was preparing myself for a lot of work. Then, I found <a href="http://rtyley.github.io/bfg-repo-cleaner/">BFG Repo Cleaner</a>. This is a Scala (a language I don't know) tool that works better than <code>git filter-branch</code> in a lot of ways.</p>
<p>I ended up using BFG to remove most of the cover images from the repository along with the large files. This let me trim the final <code>stories</code> repository from 1.9 GiB to 20 MiB. The <code>covers</code> repository is at a nice 419 MiB, but that is also acceptable since I use it so infrequently.</p>
<p>If you have to remove files, directories, or large objects from your repository, it looks like BFG is something to seriously consider.</p>
Creating slippery maps2014-09-13T05:00:00Zhttps://d.moonfire.us/blog/2014/09/13/making-slippery-maps/Someone recently asked me how to create an online map of a created world. I decided to use this opportunity to update my Exalted online map to a newer library while writing a tutorials.<p>Recently, I <a href="http://www.reddit.com/r/worldbuilding/comments/2g3u7y/is_there_a_program_with_an_interactive_map_style/ckfdza1">answered a question</a> about creating an online map of someone's fantasy world. I mentioned <a href="http://wiki.openstreetmap.org/wiki/Slippy_Map">slippery maps</a> which is the generic name for maps like Google Maps or MapQuest where you can drag around the map, zoom into it, and maybe set up bookmarks.</p>
<p><em>If you just want to look at the results, head over <a href="http://exalted-map.mfgames.com/">to the website</a>.</em></p>
<p>Now, a <em>lot</em> of things changed since 2006 when I last created a map. Then, I needed to create some PHP code to manage it and spent better part of a week trying to jam Google Map's API into something that would work.</p>
<p>Today, things are a lot different and a lot easier to create something that looks good. It also doesn't require a PHP host or a database to run. But, to answer the question I gave in more detail, this is a short tutorials on how to create a slippery map of a fantasy world.</p>
<h1>Zoom Levels and Tiles</h1>
<p>Probably the best way to understand slippery maps is to understand zoom levels. The lowest detail, highest area is zoom zero (Z0). This represents the entire world as a single 256 pixel square tile.</p>
<p>A zoom level starts with zero (Z0) which represents the entire world in a single tile.</p>
<img class="img-responsive" src="x0y0.png" />
<p>Each <em>zoom</em> level higher doubles the image size. So Z1 is a 512 pixel square image and Z2 is a 1,024 pixel square image. Google Maps allows up to Z18 which is an image 67,108,864 pixels on a side. At that level, we can see the driveway leading into most houses but the image requires at least 16 petabytes to load into memory at once.</p>
<p>I don't about you, but I'd love to have a 16 PiB RAM machine, but that isn't going to happen. Not to mention, you'd have to download all 16 PiB to actually see it.</p>
<p>So, to break it into smaller parts that are easier to download, display, and manage, slippery maps use 256 pixel square tiles. Each of those images are broken down into these consistent-sized tiles which are then downloaded as the library needs it.</p>
<p>Since we aren't using a full image, we have to know <em>where</em> in the bigger image we need to retrieve these images. And this introduces the coordinate system for slipper maps. The basic coordinate is a (z, x, y) where <em>z</em> is the zoom level. Both <em>x</em> and <em>y</em> are the zero-based index from the upper-left of the image.</p>
<p>Z0 would only have a single image (0, 0, 0). Z1 would have four: (1, 0, 0), (1, 0, 1), (1, 1, 0), and (1, 1, 1). Z2 would have sixteen and so on.</p>
<p>When a slippery map needs to retrieve the image, it maps the coordinate system into a URL of some sort. For example: <a href="http://exalted-map.mfgames.com/creation/z0/x0y0.png">http://exalted-map.mfgames.com/creation/z0/x0y0.png</a>.</p>
<p>The final component of a slippery map is a <em>layer</em>. This can be raster data, such as satellite or a political map. It can also be vector data, such as a map of all streets in a tile. In the above example, the layer is actually <code>creation</code>. There is also <code>creation-geo</code> which is a second raster layer that doesn't have cities, geomancy, or boundaries on it.</p>
<p>The only limitation on how far you allow zooming or which layers is time. It does take a fair amount of time to create and manage tiles. The higher the acceptable zoom level, the more work and disk space is required to keep those tiles and to make them look good.</p>
<h1>Starting points</h1>
<p>Because the Javascript can't create the tiles for me, and I'm not using PHP, I need to pre-render the individual tiles at all the zoom levels I need which will then be uploaded to my website.</p>
<p>For me, the best place to start is with a relatively large map in <a href="http://en.wikipedia.org/wiki/Mercator_projection">Mercator project</a>.</p>
<p><em>I'm not going to use a proper Mercator projected map since I'm going to put my <a href="https://mfgames.com/exalted-map/">Exalted Map of Creation</a> back online and I already had most of the work done. This is a flat map without projection, so most of the coordinates are hacked to get things to work.</em></p>
<p>I decided to focus on a zoom level five for the map which requires me to start with a 8,192 pixel square. Fortunately, SVG scales very well so I have a relatively sharp image at that level.</p>
<p><em>What doesn't work is that my map of Creation isn't square. I'm cheating and just putting black bars on the top and bottom, but I highly recommend you use a properly projected map that fills the tile completely.</em></p>
<p>I do <a href="http://exalted-map.mfgames.com/creation/map-z5.png">host a full version of the map</a> on my website (there is also z0 through z4, there is also creation-geo for maps without markers). Feel free to download or use that, or even hotlink. It's there until I run out of bandwidth or something goes wrong.</p>
<p>The next step is to create 256 pixel square tiles from the image and give them a consistent name. Now, the tedious approach is to manually slice out each one and save it to the disk. For the larger map, that can take hours.</p>
<p>I happen to be lazy, I don't do tedious things like this when I can find a tool to do it for me. Fortunately <a href="http://www.imagemagick.org/">ImageMagick</a> has all the tools I need. This runs under Windows (you have to double up on the <code>%</code> though for the DOS command prompt) also, for those who don't use Linux.</p>
<pre><code class="language-shell">$ convert map-z5.png -crop 256x256 -set filename:tile "x%[fx:page.x/256]y%[fx:page.y/256]" +repage +adjoin "z5/%[filename:tile].png"
$
</code></pre>
<p>This basically creates 1024 tile images in <code>exalted-map/z5</code> where the name is <code>xMyN.png</code> where <em>M</em> and <em>N</em> are numbers from 0 to 31. This means that you can get a tile just from the middle by using something like <code>exalted-map/z5/x15y15.png</code>.</p>
<p>Now, if you can't guess from the above path, I'm putting all the zoom levels above five (four through zero) in the same structure.</p>
<pre><code class="language-shell">$ convert -scale 50% map-z5.png map-z4.png
$ convert map-z4.png -crop 256x256 -set filename:tile "x%[fx:page.x/256]y%[fx:page.y/256]" +repage +adjoin "z4/%[filename:tile].png"
$
</code></pre>
<p>This is all in a <a href="http://exalted-map.mfgames.com/creation/Makefile">makefile</a> so I can just type <code>make</code> and have it generate all six zoom levels in a single command. Now, if you have a custom map for a different zoom level, then don't scale it from the higher level (z5 to z4). I'm only going to have the one for each layer, so I'm just scaling the entire thing from Z5.</p>
<h1>Putting it up</h1>
<p>Now that I have the tiles at all the zoom levels (Z0 through Z5) generated, I need to put them somewhere that can be accessed via a HTTP request. If you can't guess, I have a host at <a href="http://exalted-map.mfgames.com/">http://exalted-map.mfgames.com/</a>.</p>
<p>The tiles have to have a consistent name for the slippery map to work. I decided to use the pattern:</p>
<pre><code>http://exalted-map.mfgames.com/{layer}/z{zoom}/x{x}y{y}.png
</code></pre>
<p>At the moment, I'm allow providing two layers: <code>creation</code> and <code>creation-geo</code>.</p>
<p>The above URL is important for when we hook up a library to our URL, but the important part is that you can go to any image on that map and have it produce the correct results.</p>
<h1>Creating the HTML</h1>
<p>I decided to use <a href="http://openlayers.org/">OpenLayers 3</a> as my Javascript library. This is a pretty solid library, though there are others that can do much of the same functionality.</p>
<p>On the <a href="http://openlayers.org/en/v3.0.0/doc/quickstart.html">quickstart</a> directions, they have a very basic HTML page that uses OpenLayers 3 and has the basic page. Take that HTML file and throw it up on a website (maybe at the <a href="http://exalted-map.mfgames.com/">root above the layers</a>).</p>
<p>If you load the page, you should see MapQuest map. But we don't really want to see that, we want to see our own map. To do that, we change the layer:</p>
<pre><code class="language-javascript">layers: [
new ol.layer.Tile({
source: new ol.source.XYZ({
layer: 'creation-geo',
url: 'http://exalted-map.mfgames.com/creation-geo/z{z}/x{x}y{y}.png'
})
})
],
</code></pre>
<p>Refresh the page and you have a basic slippery map.</p>
<h1>Icons and Locations</h1>
<p>Getting locations to show up in the map is a <em>bit</em> harder. In OpenLayers 3, there are a lot of ways of doing it and I haven't quite found a perfect way. But, a “good enough” approach uses KML and a bit of magic.</p>
<p>This is the basic KML file. It just lists points on a map along with their icons. If you follow the <a href="http://exalted-map.mfgames.com/icons/">icon links</a>, you can find the icon images, they aren't that impressive (though I did make them in Inkscape).</p>
<pre><code class="language-xml"><kml>
<Document>
<name>Exalted Canon Locations</name>
<description>Created from stephenls map data.</description>
<Folder>
<name>Locations</name>
<description>Locations</description>
<Style id="style-canon-gate">
<IconStyle>
<scale>1.0</scale>
<Icon>
<href>http://exalted-map.mfgames.com/icons/canon-gate.png</href>
</Icon>
</IconStyle>
</Style>
<Style id="style-canon">
<IconStyle>
<scale>1.0</scale>
<Icon>
<href>http://exalted-map.mfgames.com/icons/canon.png</href>
</Icon>
</IconStyle>
</Style>
<Folder>
<name>POI</name>
<Placemark>
<name>Imperial Mountain</name>
<styleUrl>#style-canon</styleUrl>
<Point>
<coordinates>-20, 12</coordinates>
</Point>
</Placemark>
</Folder>
<Folder>
<name>Gates</name>
<description>Celestial Gates</description>
<Placemark>
<name>Celestial Gate #10</name>
<styleUrl>#style-canon-gate</styleUrl>
<Point>
<coordinates>22.7673, 8</coordinates>
</Point>
</Placemark>
</Folder>
</Folder>
</Document>
</kml>
</code></pre>
<p>I uploaded the KML file to the site under <a href="http://exalted-map.mfgames.com/canon.kml">http://exalted-map.mfgames.com/canon.kml</a>. If you can't guess, the library needs to download it to use it.</p>
<p>Adding the KML file to the map is pretty easy, you do it as a second layer.</p>
<pre><code class="language-javascript">var creationGeography = new ol.layer.Tile({
source: new ol.source.XYZ({
// attributes: Always give credits
url: "http://exalted-map.mfgames.com/creation-geo/z{z}/x{x}y{y}.png",
}),
});
var canonLocations = new ol.layer.Vector({
source: new ol.source.KML({
projection: "EPSG:3857",
url: "canon.kml",
}),
});
var map = new ol.Map({
target: "map",
layers: [creationGeography, canonLocations],
view: new ol.View({
center: [0, 0],
zoom: 3,
}),
});
</code></pre>
<p>You'll notice I refactored the geography stuff out into a variable to make it easier to read. I do that a lot as I'm working and puzzling through things.</p>
<p>When you upload and refresh, you'll see pretty icons for all of the canon locations. The reason I'm using ‘canon’ is because I had a secondary layer called ‘sepia’ for my Sepia Throne campaign. We had a tendency to create new locations, destroy cities, and we also used 108 celestial gates instead of the 56 in the canon world. (The 108 came from the 108 Stars of Destiny.)</p>
<h1>Clickable labels</h1>
<p>And the final part. It isn't obvious from the icons which gate it is, so it would be nice if you could click on it and get a bit more information. To do that, we are going to show a bootstrap popup of the KML name, which should be enough.</p>
<p>This needs three parts. The first is to add Bootstrap and jQuery to your webpage. View source at the map to find examples if you don't know how.</p>
<p>The second is to add a popup element in the map.</p>
<pre><code class="language-html"><div id="map" class="map">
<div id="popup"></div>
</div>
</code></pre>
<p>And finally, a bit more Javascript.</p>
<pre><code class="language-javascript">var element = document.getElementById("popup");
var popup = new ol.Overlay({
element: element,
positioning: "bottom-center",
stopEvent: false,
});
map.addOverlay(popup);
// display popup on click
map.on("click", function (evt) {
var feature = map.forEachFeatureAtPixel(
evt.pixel,
function (feature, layer) {
return feature;
}
);
if (feature) {
var geometry = feature.getGeometry();
var coord = geometry.getCoordinates();
popup.setPosition(coord);
$(element).popover("destroy");
$(element).popover({
placement: "auto",
html: true,
content: feature.get("name").replace(/ /g, "&#160;"),
});
$(element).popover("show");
} else {
$(element).popover("destroy");
}
feature = null;
});
</code></pre>
<h1>Conclusions</h1>
<p>There are a few more things I'd like to do, but… it's good enough for now. If you want to see a full version, check out the source of <a href="http://exalted-map.mfgames.com/index.html">http://exalted-map.mfgames.com/index.html</a> and <a href="http://exalted-map.mfgames.com/map.js">http://exalted-map.mfgames.com/map.js</a>. That should give you a rough idea of how to do it, along with poor comments on what we're doing.</p>
<p>I don't like using the KML approach, but I couldn't easily figure out how to do it via JSON or creating the features manually. For some reason, <code>ol.geom.Point</code> wasn't working with a <code>ol.Feature</code> and I didn't know why. This works for now, but if I was going to add the rest of the points, I'd find a cleaner way of doing it.</p>
<p>The main reason I don't want to use KML is because I want to have a link to a wiki site of some sort, or maybe some additional text. These are things easily done via JSON but not KML.</p>
Mailing lists and social networking2014-08-08T05:00:00Zhttps://d.moonfire.us/blog/2014/08/08/mailing-lists/I realized that I haven't been obvious everywhere I hang my hat on social networks. Here is a list along with what I post where.<p>A few days ago, in my random wandering of the Internet, I came up to a post on <a href="http://www.indiesunlimited.com/2014/08/06/august-blog-fest/">Indies Unlimited</a> about blog sharing. I threw my two cents in, but I don't write that much about getting published, so I figured I wouldn't get a lot of posts.</p>
<p>It wasn't until I saw other people's responses (not to me) that I realized I don't really have a way for people to “follow” me. More importantly, the ways I did have, I didn't actually <em>post</em> on my website for anyone to follow me.</p>
<h1>RSS/Atom Feeds</h1>
<p>My preferred method of paying attention to any website (including authors and comics) is a RSS feed. I use <a href="https://newsblur.com/">NewsBlur</a> for the bulk of my reading because it is a small developer, it replaced Google Reader nicely, and has most of the features I want.</p>
<p>Naturally, this is the one that I had set up from the beginning. I did break it into multiple feeds, so anyone who just wants to see my writing or programming posts can do so. Or look at everything.</p>
<ul>
<li><a href="http://d.moonfire.us/atom.xml">Atom/RSS feed of all posts</a></li>
<li><a href="http://d.moonfire.us/categories/writing.xml">Atom/RSS feed for writing only</a></li>
<li><a href="http://d.moonfire.us/categories/programming.xml">Atom/RSS feed for programming only</a></li>
</ul>
<p>I haven't quite created feeds for every tag (so you could follow only <a href="/tags/sand-and-blood/">Sand and Blood</a> posts for example). I will, eventually, or if someone asks.</p>
<h1>Mailing Lists</h1>
<p>Well, mailing lists is kind of a stretch, but I don't care for MailChimp. I do, however, use Google Groups for a couple of categories. It does the important part, sends emails, but doesn't quite have the social and Big Brother views. For some reason, I don't like when the link from a mass-mailing doesn't show the actual link; Google Groups doesn't mess with the links.</p>
<ul>
<li><a href="https://groups.google.com/forum/#!forum/dmoonfire-announce">dmoonfire-announce</a> (read only, writing announcements only, very low traffic since I don't write that much)</li>
<li><a href="https://groups.google.com/forum/#!forum/dmoonfire-blog">dmoonfire-blog</a> (read only, all blog posts)</li>
<li><a href="https://groups.google.com/forum/#!forum/dmoonfire">dmoonfire</a> (read/write, has all blog posts but you can respond and chat)</li>
</ul>
<h1>Twitter</h1>
<p>The other social network I use is <a href="https://twitter.com/dmoonfire">Twitter</a>. This happens to be my favorite one to use at the moment, mainly because it fits the way I think and interact. There are some Twitter-clones I'll use eventually, but they aren't there yet.</p>
<ul>
<li><a href="https://twitter.com/dmoonfire">@dmoonfire</a> (random saying, conversations, and a link to all blog posts)</li>
</ul>
<h1>Facebook</h1>
<p>I don't use Facebook a lot, but I do have a fan site. It should be set up to automatically mirror any blog posts about writing to it.</p>
<ul>
<li><a href="https://www.facebook.com/dmoonfirefans">D. Moonfire Fans</a> (writing posts only)</li>
</ul>
<h1>LiveJournal</h1>
<p>I know that LiveJournal appears to be dead for most people, I still maintain a mirror of it. I think it is for a single person, but I just don't have the heart to shut it down yet.</p>
<ul>
<li><a href="http://dmoonfire.livejournal.com/">dmoonfire</a> (all blog posts)</li>
</ul>
<h1>Google+</h1>
<p>I just added this yesterday because I found a way of mirroring things from my blog. If you can't guess, I prefer to have cross-posters for the bulk of the writing but respond individually on each network. Google+ has always been a messy thing, mainly because of the cross-posting, but I think I found a way.</p>
<ul>
<li><a href="https://plus.google.com/110772252513189006556/about">+D. Moonfire Fans</a> (writing posts only)</li>
</ul>
<h1>Diaspora</h1>
<p>I love the idea of Diaspora, but sadly it really doesn't have the market share for social networking. Not to mention, no cross-posting that I can set up which means I have to manually enter stuff… which means I don't always do it.</p>
<ul>
<li><a href="https://joindiaspora.com/people/4d1a56bf2c17436aa4006515">dmoonfire@joindiaspora.com</a></li>
</ul>
<h1>Everywhere</h1>
<p>I love to answer questions and chat. It doesn't happen much, but regardless of the network, I love feedback, interaction, and just connecting. I think it is one of the important parts of being a fan and a creator.</p>
Author Intrusion - Names2014-08-07T05:00:00Zhttps://d.moonfire.us/blog/2014/08/07/author-intrusion-names/One of the complexities of authors is their names. I want to avoid making an assumption of how they are displayed and used. At the same time, I want to reuse the logic to handle character names within the story.<p>I was slowly making my way through the Markdown and DocBook file processing when I hit an interesting snag: names.</p>
<p>Author names are a strange thing. It would be easy to just stick with the Western approach, which is what Amazon uses in their KDP program: first and last name. Smashwords does the same but they add suffix. No mention of title, which some people really hang up on (I almost went through a Ph.D. program just so I could be Dr. Moonfire).</p>
<h1>Complexity of names</h1>
<p>The problem is that I work with Japanese people who have family name first and given name second. There are others too, I just happen to know Japanese culture a little better than most others. Most websites I know don't actually handle this because they assume the first/last name approach.</p>
<p>You have some cultures where many people have two last names, one from the mother and one from the father.</p>
<p>The other interesting problem comes in with single name authors, let's go with Madonna or Teller for examples. Some websites allow you to go without both names, but there are a lot that don't. Google, for example, insists on something and NFN (No First Name) looks horrible when they say "Hello, NFN!".</p>
<p>An interesting one is Goodreads. Goodreads lets you have an account with only one name, but you can't sign up for a raffle without both names filled out.</p>
<p>There are also names that start with a lowercase character or have an apostrophe in the middle. Or a slew of other Unicode characters that most developers make assumptions that aren't true.</p>
<p>This bothers me, but that shouldn't be a surprise. One of the principals of Author Intrusion is “Author Know Best”. It means that the author knows what they are doing and we don't tell them how to work. This includes how format their own name. If they want a Unicode symbol, that's fine. I'll probably draw the line at images for authors, simply because I don't want to code that. I'm all for it, it's just low in my priority list.</p>
<h1>Solving the problem</h1>
<p>Normally, I use DocBook for inspiration on handling things. I figured after so many decades, they've solved most of the problems. And, in this case, I think <a href="http://www.docbook.org/tdg5/en/html/personname.html">they have</a>.</p>
<pre><code class="language-xml"><author>
<personname>Madonna</personname>
</author>
<author>
<personname>
<honorific>Not-Dr</honorific>
<firstname>Dylan</firstname>
<othername role="middle">Robert Evans</othername>
<surname>Moonfire</surname>
</personname>
</author>
</code></pre>
<p>There are no specific conventions for <code>othername</code>, but it's better than most. It also lets me enter full names for the author and figure out how to format them later.</p>
<p>In my Markdown files, my author tag is usually a preferred name.</p>
<pre><code>---
Title: Amazon Piece
Author: D. Moonfire
---
It was a dark and stormy tea cup...
</code></pre>
<p>This caused me some trouble with formatting ebooks because I occasionally need to know first name and last name to figure out sorting.</p>
<h1>Reusing concepts</h1>
<p>While I was in the process of figuring out names, I realized that I'm going to have the same problem with one of my future plans: characters. One of the reasons I'm writing this is to highlight character names. All of them, including possessives and nicknames. There is some other ideas around this, but I needed to handle a highly flexible approach there also.</p>
<p>From there, it was easy to figure out that I'm going to use the same class to represent an author name as well as character (location, objects, etc) names. And then I came up with <code>NameInfo</code>. This is a <code>Dictionary<string, string></code> that uses interned key strings.</p>
<pre><code class="language-csharp">NameInfo name = new NameInfo();
name.PreferredName = "Shimusogo Rutejìmo";
// or name["Preferred Name"] = "Shimusogo Rutejìmo";
name["First Name"] = "Rutejìmo";
// or name.FirstName = "Rutejì";
name["Clan"] = "Shimusìgo;
name["Surname"] = "Shimusogo;
name["Nickname"] = "Jìmo";
name["Possessive"] = "Rutejìmo's";
name["Possessive Nickname"] = "Jìmo's";
NameInfo author = new NameInfo() { PreferredName = "D. Moonfire" };
</code></pre>
<p>When I write these out in DocBook, it will become a full XML block.</p>
<pre><code class="language-xml"><personname>
<firstname>Rutejìmo</firstname>
<surname>Shimusogo</surname>
<othername role="Clan">Shimusì/othername>
<othername role="Preferred Name">Shimusogo Rutejìmo</othername>
<othername role="Nickname">Jìmo</othername>
<othername role="Possessive">Rutejìmo's</othername>
<othername role="Possessive Nickname">Jìmo's</othername>
</personname>
<personname>D. Moonfire</personname>
</code></pre>
<p>In the Markdown formatter, I get this:</p>
<pre><code>---
name:
PreferredName: "Shimusogo Rutejìmo"
FirstName: "Rutejìmo"
Clan: "Shimusìgo"
Surname: "Shimusogo"
Nickname: "Jìmo"
Possessive: "Rutejìmo's"
PossessiveNickname: "Jìmo's"
author: D. Moonfire
---
</code></pre>
<h1>Conclusion</h1>
<p>Overall, this seems to work. If you have a simple name, as some authors do, it will create a neat output file and they just have to enter the name. For those who are a bit pickier about names, it will let them have a complex name or set of names. It also means that when I put in name highlighting and refactoring (I rename characters a lot, but I want to also change possessive and nicknames at the same time), I'll have the infrastructure to do so.</p>
Author Intrusion - A black triangle CLI2014-08-02T05:00:00Zhttps://d.moonfire.us/blog/2014/08/02/author-intrusion-cli-and-black-triangle/I reached the first visible moment in Author Intrusion with command-line tool that does one thing, convert Markdown into DocBook formats.<p>Over the last week, I've been working on the current version of <a href="/tags/author-intrusion/">Author Intrusion</a>. It can get pretty discouraging since there is so much to do, so I'm focusing on a single task at a time and building out along the way so I will <em>hopefully</em> get a complete project by the end.</p>
<h1>Formatting</h1>
<p>One of the hardest things about coding is the GUI. There are so many things for getting events hooked up and the interface running smoothly. Being that GUIs are my weakest area of code, plus the state of the cross-platform .NET UI libraries is still a little weak, I'm going to work on other things.</p>
<p>I ended up deciding on project IO. For those who have read my blog in the post, you probably remember that I spend a lot of time working on automated the creation of ebooks and print from my source documents. Not to mention a lot of time working on creating a format that works the way I like to work.</p>
<p>That lead into my first goal: transforming a source project into another format. Eventually, this is going to lead into system used to create ODT, Microsoft Word, EPUB, MOBI, and everything else. Not to mention importing from other formats.</p>
<p>While I'm going to eventually have a GUI interface for it, I can also write a command-line interface (CLI) version which lets me get the functionality written without getting distracted by the GUI.</p>
<h1>Progress</h1>
<p>This weekend, I actually saw the beginning of my ideas show up. The initial framework for converting a <a href="http://daringfireball.net/projects/markdown/">Markdown</a> file into <a href="http://www.docbook.org/tdg5/en/html/docbook.html">DocBook 5</a> worked via the CLI.</p>
<pre><code class="language-shell">$ aicli transform sample.markdown sample.xml
</code></pre>
<p>I went with <code>aicli</code> (Author Intrusion CLI) for the tool. The interface is inspired by <a href="http://git-scm.com/">Git</a> with sub-commands and a somewhat flexible way of adding new tools.</p>
<p>The cool part is that the above command actually worked. It doesn't look like much, but there weeks of effort to get to that point. It's a <a href="http://philosophistry.com/archives/2009/01/what-is-a-black.html">black triangle</a> moment in this development.</p>
<h1>Formats</h1>
<p>I arranged the file formats into, creatively enough, <code>IBufferFormat</code> objects. They are discovered via an <a href="http://en.wikipedia.org/wiki/Inversion_of_control">IoC</a> container (<a href="http://docs.structuremap.net/">StructureMap</a>) and represent a single type of input or output.</p>
<p>The initial ones to get the above code working were:</p>
<ul>
<li>Markdown (input only)</li>
<li>DocBook (output only)</li>
</ul>
<p>I'll get these to be both input and output, that way anyone can use either Markdown or DocBook files for their primary project format. I'll also use these for the output formats:</p>
<ul>
<li>EPUB</li>
<li>MOBI</li>
<li>Libreoffice (ODT)</li>
<li>Microsoft Word (DOCX)</li>
</ul>
<h1>And it has options too!</h1>
<p>Yeah, the other bit I like is the single configurable option.</p>
<pre><code class="language-shell">$ aicli transform sample.markdown sample.xml -ORootElement=chapter
</code></pre>
<p>There may be more, but that puts me one step closer to replacing <code>mfgames-creole docbook</code> that I originally wrote for <a href="https://github.com/dmoonfire/mfgames-writing-python">MfGames Writing Python</a>.</p>
Stupid obsessions2014-07-30T05:00:00Zhttps://d.moonfire.us/blog/2014/07/30/stupid-obsessions/For the last two days, I've gotten obsessed about fixing a minor thing on my website.<p>For the last few years, I've been pretty good about only focusing on projects that advance my long-term plans. However, there are occasionally little things that don't really <em>help</em> me. The last few days was one of those projects. Recently, I switched over to <a href="http://jekyllrb.com/">Jekyll</a> because WordPress was giving me a hard time. I worked on it for a few days, but stopped because I really needed to work on other things.</p>
<p>And then two days ago… oh, that was not the smartest thing to do. I wanted to fix a couple minor things, mainly breadcrumbs and navigation for posts. I figured it'd be a couple hours effort and I'd be done.</p>
<p>I managed to get breadcrumbs back in on the posts, navigation, and everything else that I planned on doing it. However, I had to write a <em>lot</em> of other programs and reinvent the wheel to get everything working together.</p>
<h1>Obsession</h1>
<p>Somewhere in the middle of last night, I knew that I should have just let it go and used the old version. I have writing and programming to do, not obsess about getting annual and month archives on the site.</p>
<p>Around midnight, I even said “just put it down,” but I couldn't. It was haunting me. I kept telling myself “just a few seconds until I'm done.” Of course, I knew that wasn't true, but I had so much trouble putting it down.</p>
<p>This evening, I finally finished the coding on the site. I only fixed a few minor things, ones that no one will probably notice, but now I'm happier with the site. And I can let it go.</p>
<h1>Jekyll and Ruby</h1>
<p>I'll admit, some of the obsession comes from solving problems. I don't know Ruby very well (the language Jekyll is written in) and I was trying to just “get it working” which means I use Perl to prototype. Yeah, it isn't the <em>right</em> way of doing it, but it's the way I think and the time I was willing to put into it.</p>
<p>Sooner or later, I'll have to learn how to write some of these features in Ruby and proper Jekyll plugins, but the <a href="https://github.com/dmoonfire/mfgames-jekyll-perl">Perl version</a> seems to work for what I want, including all those breadcrumbs, collection gathering, and archives.</p>
<p>Now, I can do what I was suppose to do.</p>
Author Intrusion Tokens2014-07-27T05:00:00Zhttps://d.moonfire.us/blog/2014/07/27/author-intrusion-tokens/A brief discussion why I'm using tokens for this implementation of Author Intrusion and the issues of memory pressure and editing.<p>Given that my ideas for Author Intrusion are so large, there is always a danger that I'll get discouraged or “lost in the forest” from my efforts. I'm thinking about doing the occasional post of what I'm trying and why.</p>
<p>This entire thing started with an idle thought a month ago. Even though I haven't touched AI in months, I still think about it as I work through the problems that caused me the stop or lose track of my efforts. Yeah, I was working on a novel, but AI's development still stalled.</p>
<p>I was trying to figure out how to load my 600k word weekly serial into the editor. This is one thing that AI is suppose to handle: I've not had a lot of luck with single documents over ten thousand words much less something that large; IDEs in general can't handle a single document of that size either.</p>
<p>I'm not planning on loading it all into memory either, but I'm going to have processes that will look across the entire piece which means I'm going to load and unload in the background as I go.</p>
<h1>Background Processing</h1>
<p>One of the biggest features of AI is going to be the background processing. For programs like Microsoft Word, you might occasionally notice that spell-checking happens a few seconds after you type the word. In Visual Studio, it can take a while for ReSharper or NCrunch to finish what it was doing. I'm planning on doing the same thing, but it will be more complicated than spell-checking and probably on par with code analysis that ReSharper does.</p>
<p>The problem that I had was updating the UI. The previous C# implementation of AI tried to use locking to handle the updating, but it didn't work well. Actually, it crumbled under the weight of even lightweight locking. With this version, I'm going to use the idle threads to update the UI that was calculated in a Task.</p>
<p>That will handle the UI updating issue, but introduces a different problem. Every time the writer changes the line, all the processes are invalidated. Even things like highlighting character names or spell-checking. So, to avoid redoing thing, I jumped through a lot of hoops to update the string in place, which… ended up being more difficult that I imagined.</p>
<h1>Tokens</h1>
<p>My idea came down to breaking the lines into tokens, individual words or components. So, the line “You are a cheese head.” would become ten tokens: “You”, " ", “are”, " ", “a”, " ", “cheese”, " ", “head”, ".".</p>
<p>With tokens, I can have the background process say “update line 23, token 4” and it would be easier to update from a background process using an idle thread.</p>
<h1>Memory</h1>
<p>The problem of breaking apart tokens is memory pressure.</p>
<p>Assuming a 32-bit platform, a pointer to a string is 4 bytes. According to <a href="http://www.dotnetperls.com/string-memory">this page</a>, the size of the string is 20 plus double the length.</p>
<p>“You are a cheese head.” requires 68 bytes (4 for the pointer, 20 for the overhead, and 44 for the length).</p>
<p>If I broke apart the line, the token version requires: 284 bytes (30, 26, 30, 26, 26, 26, 36, 26, 32, and 26 bytes for a single pointer to the individual tokens respectively).</p>
<p>Fortunately, the text can be shared (interned) so I don't have to allocate four copies of " ", which lowers the cost of the tokenized version to 218 bytes (since I don't need three of the 22 byte allocations).</p>
<p>While 218 bytes is much larger than 68, I think it will still work. If I use my weekly serial as an example, in 600k words, the most common tokens are:</p>
<ul>
<li>and: 18786</li>
<li>she: 19143</li>
<li>her: 25504</li>
<li>,: 29788</li>
<li>the: 31865</li>
<li>.: 50501</li>
</ul>
<p>This means that allocating 50k “.” would take 1,282 kB but interning that data would be 197 kB. And that's for a single character. Obviously, I'm not planning on loading the entire thing in memory, but I'm going to have to load and unload chapters while processing and this would reduce the memory pressure.</p>
<h1>Editing</h1>
<p>The other complexity of tokens is editing. It gets more difficult to insert and delete text in the middle of the token, but it is possible. Most of the early effort of the UI will be handling that, but I think I can work through that using unit tests and a proper MVC approach.</p>
<h1>Complexity</h1>
<p>This may not work. Writing an IDE or a text editor is not a simple problem and I can find very little discussions on <em>how</em> to write one. Also, I haven't found discussions on working with very large documents since most the editors that provide code and discussion load the entire thing into memory. I could be wrong, but I'm fumbling through this as I go, but it seems to be working out pretty well at this point.</p>
Author Intrusion for a little while2014-07-20T05:00:00Zhttps://d.moonfire.us/blog/2014/07/20/switching-to-author-intrusion/Going to work on Author Intrusion a little while waiting on my writing projects. Maybe I can finally get the writing environment I'm looking for.<p>Yesterday, I finished the first draft of the last of my writing obligations. It's taken me a few months, mainly because my output has dropped significantly with me breaking a leg and BAM coming into our lives. It also means that the new few months are probably going to continue to trend and I'm going to be frequently interrupted and not given a lot of contiguous hours to write.</p>
<p>I could work on <a href="/tags/sand-and-bone/">Sand and Bone</a>, the third and final book of the series. I'm about three months from finishing that project.</p>
<p><a href="/tags/sand-and-ash/">Sand and Ash</a> is currently at the editor, but I honestly don't expect to get it back for another four months (probably my largest frustration at this point).</p>
<p>The problem comes down to these longer projects. When I write, I keep hoping for features that no program seems to have. My project dictionary has gotten rather cumbersome, which makes it harder since I have to copy/paste it into each new chapter. When it was thirty or so words, it wasn't so bad. At three hundred words (including names, locations, conlang, etc), it is a significant hunk of each document. There are also other little things that would help with writing, but Emacs (or Scrivner or Word) just don't seem to handle.</p>
<p>Since <em>Ash</em> is in limbo, I decided to work on Author Intrusion for a month or so. Actually, I <a href="https://github.com/dmoonfire/author-intrusion-dev/">started over again</a> because the last five attempts have ended in logical dead-ends. I'm not going to say failures, because they aren't. I learned a lot with each iteration, but when I got far enough into it, I found places where they broke down.</p>
<p>In the last few months, I actually tried seeing if there was something I can do with another editor, including looking into expanding Emacs or Sublime. Sadly, I don't think I have the skill to do either of those, mainly because of the infrastructure and their associated scripting languages (Lisp and Python respectively).</p>
<p>I've gone back to a C# implementation, but with a different foundation for the editor. I'm also leaving it a glorious mass of single projects until the 1.0.0 release instead of the first checkpoint release. I think I split the projects too quickly and spent more time shuffling the DLLs around then actually coding.</p>
<p>I'm also trying to focus on staying with unit tests for longer. I think I know what Gtk# can do at this point. I'm planning on stealing most of the Gtk# GUI code for the project, but integrating the code with the GUI adds a lot of complexity to solving projects. If I can resolve most of them via TDD, then it would be easier to integrate later.</p>
<p><em>I still plan on writing a WPF and a Mac-specific front end, but Gtk# plays better across the two platforms I use daily.</em></p>
<p>Sadly, I was hoping Gtk# or Qt# would stabilize after a half year, but they haven't. I found ways of working around them for now, so I'm going to move forward and figure out those problems when I get there.</p>
<p>I have no clue if this is going to work. But, I'm still going to try. The problem comes down to complexity. A lot of developers can talk about how to create a game, a to do application, or a slew of other applications. But, very few talk about creating an IDE. I think I know why, it is a complex problem. And I'm doing this on my own, so I fully expect there to be stumbling along the way.</p>
Uploading ebooks into WooCommerce and WordPress2014-07-04T05:00:00Zhttps://d.moonfire.us/blog/2014/07/04/enabling-epub-and-mobi-uploads-to-wordpress-and-woocommerce/How to enable MOBI and EPUB uploads into WooCommerce.<p>I've helped set up a number of WordPress-based commerce sites for authors. One of the first things that always gets in the way is the inability to upload MOBI or EPUB file into WooCommerce to actually set the books.</p>
<p>To get around this is a pretty simple process.</p>
<ol>
<li>In <code>wp-content/plugins</code>, create a directory like <code>local-upload-ebooks</code>.</li>
<li>Create <code>local-upload-ebooks/local-upload-eboooks.php</code> and edit it.</li>
<li>Put the text below into it and save it.</li>
<li>Go into your Plugin Manager and enable the plugin.</li>
</ol>
<pre><code class="language-php"><?php
/*
Plugin Name: Upload Ebooks
Version: 0.0.0
Author: D. Moonfire
Description: Enables uploading ebooks into WordPress and WooCommerce.
*/
// Add in a hook to add the taxonomies and categories.
add_action('init', 'upload_ebook_init', 0);
function upload_ebook_init()
{
// This allows up to add the MIME types that can be uploaded.
add_filter('upload_mimes', 'upload_ebook_upload_mimes');
// These are completely arbitrary and probably too big.
ini_set('upload_max_size','100M');
ini_set('post_max_size','99M');
// Ebooks sometimes take a little longer to upload.
ini_set('max_execution_time','300');
}
function upload_ebook_upload_mimes($mimes)
{
$mimes = array_merge($mimes, array(
'epub' => 'application/epub+zip',
'mobi' => 'application/x-mobipocket-ebook'
));
return $mimes;
}
?>
</code></pre>
<p>I hope it helps anyone who gets stuck by this. I'm sure there is already a plugin in WordPress's catalog, I just couldn't find it.</p>