WordPress mass uploading

I'm a lazy person. I can't really describe it any other way. I don't like doing the same thing over and over again. It is tedious and my time is better spent on other things. However, I'm always changing. It might be a new technology, file format, or organization. For example, in the process of analyzing how many words I've written over the years I touched on the problems of repeatedly changing file formats . I want to keep them consistent instead of in a ton of different formats. My previous analysis would have been a lot harder if I had five different formats. And I'm not going to change this part of me. If I decided to Creole is a horrible format to write in (actually, I want to move to Markdown since that is the most popular format right now), I will translate every story, novel, and chapter to the new format.

The only reason I haven't switched over is because I'm in the middle of a serial and I thought it would be cool to show the progression of the serial over time and I don't want to worry about renames when I do it.

Changing websites is one of those things. With the DokuWiki site, I had the Creole parser plugin so I just made a couple changes and uploaded them directly into the file host. This is actually one of DokuWiki's strengths. When I had a new story or chapter, all I had to do was scp the file up and visit the site. It was there and it took no effort.

Using WordPress is a much harder. I can't just scp the files up and have them show up. I need to go through the website (or at least the API) and create the page. And then copy/paste it in and change the formatting (I rarely use italics but I do use them). And that is fine for the 5-6 stories I have on http://d.moonfire.us/. For the other byline, I would have hundreds of files to upload.

And I'm too lazy to do that.

Fortunately, I had a lot of things working for me

  • WordPress has a XML-RPC API which lets me create and update pages. It also supports custom taxonomies such as the ones I use.
  • My Git repository is organized like my website.
  • I have a build process (hacky at the moment) that takes my .txt, .md, and .creole files and puts them into a properly organized build/ directory as DocBook files.
  • The stories that go on the website are marked with a DocBook <subjectset/> which also controls the build process.

Given that, I want to upload '/local/build/directory/fiction/thomas-the-baker.xml' to http://d.moonfire.us/fiction/thomas-the-baker. I dropped extensions years ago, mainly so I can change formats, but also because I think it creates a cleaner looking URL.

The main part is creating the pages. I'm not going to post a lot of source code this time, mainly because I rolled all of the code into MfGames Writing in the brand new mfgames-docbook-wordpress utility.

$ cd /local/build/directory
$ mfgames-docbook-writing upload --username=myuser --password=mypass --blog=myblogid fiction/thomas-the-baker.xml
... output

This works because '/fiction' was already created. There is a fair amount that is happening here, but each step is relatively simple. One of the first things mfgames-docbook-wordpress does is download a list of all the pages already there. It also grabs the taxonomies that applie to the post type we are uploaded (in this case, it is hard-coded to "page").

For each file, we create a WordPress-specific version of the file (via XSLT) that doesn't include the common headers and footer elements. We also calculate the SHA256 on the source document.

From there, it figures out the path for the file. If the page doesn't exist on the WordPress installation, it creates a new pages and uploads it. If it already exists, it checks the hash to see if it changed. If it has, then it uploads the new version. Otherwise, it skips it.

You'll notice I had to add a --username and --password options to my file. I expanded my MfGames Python Tools to get around that. Now, it can check for an INI file in a variety of locations and pre-populate the values. In this case, I have just a few key ones.


Now, I can be inside a directory and just run the command and it will create the proper path.

$ cd fiction
$ mfgames-docbook-wordpress upload thomas-the-baker.xml

In the end, it means that when I finish a chapter on my serial, I just have to go into the proper directory and type make which then converts the files to DocBook XML and then tries to upload the entire website. Only pages that changed will upload and they are live right away. So, it took a few days to get to this point, but it simplifies things going forward.

$ make
... magic happens

... at least until I change everything again.