wrangling octopress

I wanted somewhere easy to dump technical notes that weren’t really suitable for this blog. I wanted a static HTML generator type of blog because the place to dump my notes (people.canonical.com) isn’t really set up to run anything complex for a multitude of reasons, such as security.

I also didn’t want to just do it 1990s style and throw up plain ASCII README files (the way I used to) because I envision embedding images and possibly movies in my notes here. At the same time, the closer I can get to a README the better, and so that seems to imply markdown.

After a brief fling with blacksmith where absolutely nothing worked because of a magical web 2.0 fix-everything-but-the-zillions-of-pages-of-existing-docs rewrite, I wiped the blood and puke from my mouth and settled on octopress.

Octopress was much better, but it was still a struggle. It’s a strange state of affairs that deploying wordpress on a hosted site is actually *less* difficult than configuring what *should* be a simple static HTML generator. Oh well.

Here are some notes to make life easier for the next person to come along.

Deploying to a subdir, fully explained
One wrinkle of hosting on a shared server using Apache conventions is that your filesystem path for hosting files will probably get rewritten by the web server and displayed differently.

That is:

    unix filesystem path                 =>  address displayed in url bar
    /home/achiang/public_html/technotes  =>  http://people.canonical.com/~achiang/technotes

The subdir deployment docs talk about how to do this, but the only way I could get it to work is by issuing: rake set_root_dir[~achiang/technotes] first. So the proper sequence is:

rake set_root_dir[~achiang/technotes]

vi Rakefile	# and change:
	-ssh_user       = "user@domain.com"
	+ssh_user       = "achiang@people.canonical.com"
	-document_root  = "~/website.com/"
	+document_root  = "~/public_html/technotes"

vi _config.yml	# and change:
	-url: http://yoursite.com
	+url: http://people.canonical.com/~achiang/technotes

rake install
rake generate
rake deploy	# assuming you've setup rsync deploy properly

Once you’ve tested this is working, then optionally set rsync_delete = true. But don’t make the same mistake I made and set that option too soon, or else you will delete files you didn’t want to delete.

Finally, once you have this working, the test address for your local machine using the `rake preview` command is http://localhost:4000/~achiang/technotes.

Video tag gotchas
One nice feature of Octopress is the video plugin it uses to allow embeddable H.264 movies. I discovered that unlike the image tag which apparently allows for local paths to images, the video tag seems to require an actual URL starting with http://.

Therefore:

    {% video /images/movie.mp4 %}	# BROKEN!

However, this works:

    {% video http://people.canonical.com/~achiang/images/movie.mp4 %}

I’ll work up a patch for this at some point.

Misc gotchas
The final thing I tripped over was https://github.com/imathis/octopress/pull/1438.

I’ll update here if upstream takes the patch, but if not, then you’ll want the one-liner in the pull request above.

Summary
After the initial fiddly bits, Octopress is good enough. I can efficiently write technical content using $EDITOR, the output looks modern and stylish, and it all works on a fairly constrained, bog-standard Apache install without opening any security holes in my company’s infrastructure.

sbuild vs pbuild

Today I spent a little bit of time playing with sbuild and after an hour or so, decided I hated it. Tried to figure out why people recommend it, and it seems like the best answer is, “it’s the closest to what the buildds use”. I guess that’s a fair answer, but out of the box, sbuild feels clunky to me.

Luckily, Michael Terry is jawesome and wrote these really great pbuilder wrapper scripts and now they’ve landed in Quantal.

If you want to know why I ♥ them so, check out my contra answer on askubuntu:

Why use sbuild over pbuilder?

And if you want to speed up your pbuilder even moAR, then check out PbuilderHowto.

Maybe I don’t know what I’m doing so if you have tips or corrections, add them over there. If you see mterry out somewhere, buy him a beer!

“local” telephone numbers

Many apartment buildings have some sort of buzzer system for the front door so that residents can conveniently allow their guests to enter without have to (gasp!) physically walk somewhere to (gasp!) manually open the door. (for the record, I heartily approve of this feature)

There are several variations of buzzer systems ranging from the archaic hard-wired type that is connected to a dedicated intercom unit in the apartment to more modern types with a telephone “directory” of sorts and end up calling a phone number.

My building has the latter, and after having lived here for a few months I decided it was finally time to hook my phone up to the system. There was one little catch: the buzzer system actually makes real phone calls and it only has a local calling plan, meaning anyone with a long distance number is out of luck.

This is the part where we cheer for Google Voice, which is the best thing for phones since Caller ID.

Using Google Voice, you can essentially buy a phone number anywhere in the country and route it wherever you want including your existing mobile phone. So all I had to do was to buy a number that the buzzer system considered to be local and I’d be laughing.

Except… what, exactly, is a local telephone number these days? In olden times, area codes were pretty reliable indicators. But then people started getting business lines and fax lines and cell phones and pretty soon the phone companies started running out of numbers and we got split plans and overlay plans and the much hated and feared ten digit dialing but lucky for us the world didn’t end. [nb, my childhood incorporated a split plan, having grown up in 201 for my entire childhood, we were abruptly shoved into the far inferior 973 which was mildly traumatizing at the time, but I’ve since worked out all those issues with my therapist]

So anyway, once we were dialing 10 digits for all calls, consumers no longer had a clear signal as to what was a local call and what was not… except we all got cell phones a few years later and pretty much stopped caring about that archaic concept known as “long distance”.

Save for pesky little things like door buzzers with el cheapo local dialing only. My building manager gave me some dramatic warnings about how not all Google Voice numbers were considered local to our door buzzer and sent me off on somewhat of a wild goose chase that ended up with me on one of the most deliciously geeky websites on the entire intarwebs: local calling guide.

It’s kinda hard to know what to do with this page when you first see it, but after some digging around, it turns out to just be awesome. Start by pondering list of NPA codes. Normal human beings will refer to these things as “area codes” but hey, now that we’ve learned the precise terminology, we should adopt it, right? Let them eat “numbering plan areas”!

From there, you will need to find the NPA-NXX-X of both the buzzer system and the potential Google Voice number you might want to acquire. I haven’t actually been able to figure out what NXX-X stands for (if anything), but the practical piece of information is:

201-555-1234
NPA-NXX-X...

So 201 is your NPA, then 555-1 is your NXX-X. Once you have identifed the NXX-X of both the buzzer system’s phone number and whatever number you might get from Google Voice, you simply check the LATA (local access transport area) column. If the two NXX-X numbers are in the same LATA, then you can dance a jig because it means the buzzer will consider a call to your GV number to be local.

A few extra tips though regarding the GV numbers. You can see that are registered and owned by bandwidth.com. Note the column labelled “eff. date”, which is when the number went into service. If the date is very new, then it could be possible that the back end system your door buzzer hooks into might potentially not know about the new NXX-X and consider it to be in a different LATA, and hence refuse to make calls. I chose an NXX-X that was registered in 2009 just to be safe.

[bonus fun fact: most of northern Colorado shares the 970 area code, but not all of the NPA-NXX-X are in the same LATA. This explains why calling from Ft. Collins to Grand Junction is indeed local but calling South Baggs is not]

And once you get the buzzer hooked up to GV, well… all sorts of interesting things can happen, since GV numbers happen to be quite programmable. The fun stuff you can do with them is left as an exercise to the reader.

Be sure to check out the SAQ or seldom-asked questions. I totally ♥ the extreme nerd factor on this site.

bufferbloat for the impatient

After wandering around for a bit, I’ve settled back in San Francisco on a more or less permanent basis. Part of the moving process was finding an ISP and it seems like Comcast is the best option (for my situation). I signed up for their standard residential service, and remote teleworking continued on quite merrily… except for one tiny wart.
Stores don’t always advertise their extra online deals, such as free shipping or 20 percent off your entire purchase, but coupon services have these codes. Even 10 percent off your purchase is worth the extra five minutes it will take you to search

We use Google Plus hangouts quite extensively on my team including a daily standup with attendance that hovers between 5 to 10 people. The first time I tried a hangout with my new Comcast service, it was unusable with extreme lag everywhere, connection timeouts, and general unhappiness.

I had a strong hunch that I was suffering from bufferbloat, and a quick ping test confirmed it (more on that later). Obviously I wanted to fix the problem, but there is a lot of text to digest for someone that just wants to make the problem go away.

After a bit of irc whingeing and generous help from people smarter than me, here are my bufferbloat notes for the impatient.

background
Bufferbloat is a complex topic, go read the wiki page for excruciating detail.

But the basic conceptual outline is:

  • a too large buffer on your upstream may cause latency for sensitive applications like video chat
  • you must manage your upstream bandwidth to reduce latency (which typically means you intentionally reduce upstream bandwidth)
  • use QoS in your router to globally reduce upstream bandwidth (not for traffic shaping!)

diagnosis
Ensure your internet connection is idle. Then, start pinging google.com. Observe the “time” field, which will give you a value in ms. Watch this long enough to get an intuitive feel for what is a normal amount of latency on your link. For me, it hovered consistently around 20ms, with some intermittent spikes. You don’t need to be exact. If the values swing wildly, then you’ve got other problems that need to be fixed first. Stop reading this blog and call your ISP.

While the ping is running, visit http://testmy.net/upload and kick off a large upload, say 15MB or more.

If your ping times increase by an order of magnitude and stay there (like mine did to around 300ms), then you have bufferbloat.

This isn’t as rigorous as setting up smokeping and making pretty graphs, but trust me, it’s a lot faster and way easier. Thanks to Alex Williamson for this tip.

mitigation
You will need a router that can do QoS.

The easiest solution is to spend $100 and buy a Netgear WNDR3700 which is capable of running CeroWRT. Get that going and presumably you’re done, although I can’t verify it since I am el cheapo.

I didn’t want to spend $100 and I had an old Linksys WRT54GL lying around. Install Tomato onto it. (Big thanks to Paul Bame for helping me (remotely!!) recover a semi-bricked router.) Now it’s time to tune QoS.

In the Tomato admin interface, navigate to QoS => Basic Settings. Check the “Enable QoS” box and for the “Default class” dropdown list, change it to “highest”.

Figure out your maximum upload speed. You should be able to obtain this number after a few upload tests at testmy.net that you did in the previous step. Enter your max upload speed into the “Outbound Rate / Limit” => “Max Bandwidth” field. Make sure you use the right units, kbits/s please!

Finally, in the “Highest” QoS setting under Outbound, set your lower and upper bounds. I started with 50% as a lower bound and 60% as an upper bound.

Put a large fake number in for “Inbound Limit” and change all the settings there to “None”. These settings don’t seem to affect latency.

Click “save” at the bottom of the page — you do not need to reboot your router.

Re-run the google.com ping test + large upload test at testmy.net. Your ping times under load should remain relatively unchanged vs. an idle line. Congrats, you’ve solved your bufferbloat problem to 80%.

Update (7/29/2012): Thanks to John Taggart for pointing out a more rigorous page on QoS tuning for tomato.

Now you can experiment with increasing the lower and upper bounds of your QoS settings to get more upstream bandwidth. As always, make a change, save, re-run the ping + upload test, and check the results. Remember, the goal is to keep latency under load about equal to what it is on an idle line.

Now your colleagues will thank you for the increased smoothness of your video chats, although remembering to brush your teeth and put pants on is the “last mile” problem I can’t solve for you.