There are a number of valuable checks that I should do every time I make a change to this site, things like checking for broken links and misspelled words, catching build warnings, locating orphaned pages (those that exist on disk but aren’t linked) and confirming that my build environment is correctly versioned. I’m lazy (or busy, or both) so while those things might be valuable, unless someone or something does them for me, they’re unlikely to be done regularly. Fortunately the nature of these tasks makes them great candidates for automation.

Recently I wrote about switching my wordspeak deployment scripts to fabric and I’ve been gradually extending my fabric fabfile to include these validation checks. Here’s the output from my most recent deployment, showing:

• the build, checking for Nikola warnings
• spell checking using pyenchant (actually, this isn’t shown, but it does happen)
• versioning the modules used in my build environment
• confirming I’ve version controlled my posts and build environment
• the sync to the staging site (on my local machine)
• broken link checking using linkchecker
• checking for orphaned files (I’m still working out what to do with the gallery, and I haven’t finished writing the background for the other page)
• the sync to the main site
• a push to the github repository to make sure it’s current

(wordspeak)Mercury:wordspeak.org esteele$ fab deploy
[localhost] local: /Users/esteele/.virtualenvs/wordspeak/bin/nikola build
0 actions performed
[localhost] local: pip freeze > requirements.txt
[localhost] local: git status --porcelain
[localhost] local: rsync --delete-after -a output/ /Users/esteele/Sites/staging.wordspeak.org
[localhost] local: rsync --delete-after -a output/.htaccess /Users/esteele/Sites/staging.wordspeak.org
[localhost] local: linkchecker --config linkcheckerrc http://staging.wordspeak.org
That's it. 182 links checked. 0 warnings found. 0 errors found.
[localhost] local: linkchecker --config linkcheckerrc --verbose --file-output=csv/linkchecker-output.csv --no-status --ignore-url '!(staging.wordspeak.org)' http://staging.wordspeak.org
Orphaned file: categories/index.html
Orphaned file: pages/d3/d3-nt-sla-map.html
Orphaned html files exist (are on disk but aren't linked). Continue? [Y/n] y
Push to live site? [Y/n] y
[localhost] local: rsync --delete-after -a output/ wordspeak.org:/users/home/esteele/web/public
[localhost] local: rsync --delete-after -a output/.htaccess wordspeak.org:/users/home/esteele/web/public
[localhost] local: git push
Counting objects: 17, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (13/13), done.
Writing objects: 100% (13/13), 2.17 KiB, done.
Total 13 (delta 9), reused 0 (delta 0)
To git@github.com:edwinsteele/wordspeak.org.git
39b8464..c81b8b1  master -> master
Done.
(wordspeak)Mercury:wordspeak.org esteele$

You can see that Fabric is doing all the donkey work, and I’m left to confirm a few (temporary) oddities and make a final choice to send the change onto the live site. It’s very satisfying. At some point I’ll probably extend it to do W3C validation of the HTML and RSS and as I don’t know a way to detect JavaScript errors, I could use it as an excuse to learn how to write selenium tests. I regularly trigger warnings even in its current state, though, so it’s value is clear.

I consider these sort of tasks to be the unit tests and static validations for my site, and apply the same sort of mindset as one would with other checks of this type. Namely:

• Make them repeatable
• make the checks run quickly (mine run in around 30 seconds)
• make it harder to deploy the site without running them than with them (I do this by making my deploy command run all the tests)
• white-list the known, but accepted, violations so you don’t have to think about whether the warning is known

The source for my fabfile is on github

It’s amazing how much there is to read, and the pace at which great content is replaced by more great content. Here is some of the gold from February:

• Apollo 13 - we have a solution - IEEE Spectrum. All the fascinating detail that couldn’t be shown in the movie. Best read of the month.
• The cold hard facts of freezing to death - Outdoor Adventure Magazine. Well written narrative about the effects of cold on the human body.
• At Facebook, zero-day exploits, backdoor code bring war games drill to life - Ars Technica. Insight into how the Facebook IT team handle major security incidents. Interesting throughout.
• What Every Programmer Should Know About Memory [pdf] - Ulrich Drepper. Incredibly detailed explanation of how CPUs use memory and cache during execution. It took me several months to read but has irreversibly changed my perspective on software performance. Ignore the fact that it’s a few years old - it’s still incredibly relevant and valuable.
• 3:10 to Yuma (1957): Had deeper character development and fewer distractions from the main storyline than the 2007 version. I found it very engaging.

• And I will remember my covenant - Charles Spurgeon. The work of salvation was completed when Jesus died for my sin. There’s nothing that I can add, and nothing that I need to add.

  My looking to Jesus brings me joy and peace, but it is God’s looking to Jesus which secures my salvation and that of all His elect, since it is impossible for our God to look at Christ, our bleeding Surety, and then to be angry with us for sins already punished in Him.

Fabric has caught my eye as a deployment tool several times over the last year or so. Each time I’ve seen it, my thought has been that it’s a attempt to replace something that doesn’t need replacing, namely the venerable unix shell script.

Fabric came up again this week in an article by the NPR application team ¹ and on this occasion I saw that they were solving a similar problem to one that I recently solved with a shell script. The authors seemed clear-headed and not prone to fads so I thought I’d take the opportunity to duplicate my shell script functionality in a fabric fabfile.

I must say that I was impressed. The fabric tutorial is very well structured and allowed me to get results quickly. Within a few hours I’d git-rm’ed my shell script and the build-version_control-test-deploy cycle for this site is now handled by a fabfile. While the fabfile is the same length as the shell script, I think it’s far easier to read and allows partial deployments (reminding me of make). I particularly like the innovative use of context managers as it removes one of the key problems with deployment scripts, that inadvertently modifying global state can cause unexpected downstream side-effects. Well done, Fabric developers.

I don’t plan to replace all my shell scripts with fabfiles, but I’ll seriously consider it next time I need to write a script.

The state of my workspace affects my state of mind. Perhaps I shouldn’t find it hard to write code when there’s clutter, but until I find a way to cope with it, I’ll continue to have occasional cleaning frenzies and purges. One such event happened last week and I’m pretty happy with the result.

Most items in my workspace are like old pairs of jeans. They don’t look as good as those in the shops, and maybe aren’t as flash, but they’re comfortable and do the job well. I use a decade-old mouse, a keyboard from the mid 90’s, a monitor from 2005 and an aging MacBook air and while they’re functional, they leave something to be desired in the cable clutter department. As I’m not going to replace them, I need to deal with the clutter in another way.

Enter stationary supplies:

• Bulldog clips (so cables don’t fall down the back of the desk when they’re not in use)

• Sticky tape (so my headphone cable doesn’t run across my desk)

• Cardboard (so I don’t run out of sticky tape stashing a USB hub and long cables)

Now I have a clutter-free desk, without even visiting Ikea.

Last month I moved from having a dynamic WordPress blog to having a statically built blog generated by Nikola. I wanted to preserve the WordPress-style post and RSS URLs when I migrated and while I could have manually created redirections within Nikola, I thought I could do better through web server directives.

My WordPress installation generated posts URLs in the form http://wordspeak.org/yyyy/mm/dd/some_post_name. After using the WordPress importer in Nikola, that post now exists at http://wordspeak.org/posts/yyyymmddsome_post_name.html. I noticed that Apache RewriteRules didn’t play nicely with the relative paths that Nikola generates but I found I could achieve what I wanted with the Redirect family of directives.

Here’s a snippet of my Apache .htaccess file to handle these redirections.

RedirectMatch permanent ^/([0-9]{4})/([0-9]{2})/([0-9]{2})/([a-z][-a-z0-9]*)$ /posts/$1$2$3$4.html
# /feed /feed/ /feed/rss /feed?=http://wordspeak.org
RedirectMatch permanent ^/feed /rss.xml
RedirectPermanent /rss /rss.xml

Hopefully someone will find this valuable.