pagerduty and sentry email integration

When adding a new generic email integration into PagerDuty, you are greeted with the following defaults:

Screenshot from 2015-12-15 13:54:18

If you are integrating with Sentry, then this default behavior will quickly annoy you because you will get a brand new PagerDuty incident each time someone makes a comment on the Sentry report.

Luckily, you can easily write rudimentary email filters in PagerDuty, but what to filter on?

It turns out that Sentry includes a plaintext section in the mails they send out, and there seems to be at least one regularly occurring pattern in new Sentry event mails, namely the string:

A new event has been recorded in Sentry:

Armed with this knowledge, we can write a simple PagerDuty filter so that new Sentry events create new PagerDuty incidents, but comments on existing Sentry events are ignored.

Screenshot from 2015-12-15 10:51:51

Things that make me sad:

  • no pre-canned Sentry integration in PagerDuty
  • PagerDuty email filters only work on subject line, body, and sender; why can’t I filter on the message headers?
  • Sentry includes an “ID” + a unique event identifier in their email, but only in the mail on the first event; on subsequent mails, the unique identifier is no longer present, which prevents me from writing a better filter in PagerDuty that could theoretically find the unique identifier and link followup emails together

So, I guess we are still stuck with “parsing email bodies as an API” 1990s Linux-landia. Won’t some nice VC throw a few hundred $million at this sure-to-be-a-unicorn market?

a love letter to instacart

I recently signed up for Instacart Express, which starts you off with a free two-week trial. After last night, I cancelled my subscription before they billed me.

Instacart sent me an automated email asking why, and this was my response:

I really wanted to love Instacart. I thought it was going to change my life.

But my experience during your two week trial has been a clown show.

The first order that was screwed up was XXXXXXX. We ordered shallots and the shopper gave us garlic instead. If you’ve ever cooked anything more complex than a Hot Pocket, you’d know that those items are not substitutes for each other.

On top of this, you charged an incredible $3.69 per shallot. I can only assume these shallots were eaten by civets, crapped out, and then harvested by the only migrant workers in America making a fair living wage.

To your credit, you did fix the issue by re-delivering these magical shallots for free the next morning, and in the end, it was a very minor incident after all.

But tonight was a very special experience indeed. And by “special” I mean, “not special”.

On order XXXXXXX, I placed it quite early in the day with a delivery time for 7 — 8pm.

At 8:30pm, after receiving no update from Instacart, I sent a ping to the shopper asking if there were any issues with the order, and received this text in response:


Pretty brilliant, I must say.

So, in summary, thank you for the free two week trial of your service. I learned that I would have been signing up for a year of disappointment for $99, and for that price, I could donate money to the Trump campaign and gotten far more entertainment value out of it.

Maybe you can make me feel better. But seriously. The Donald. Gotta be better than The Donald.

To Instacart’s credit, they responded quickly, offering me another free month of Instacart Express and a $25 coupon. So, I guess we’ll continue with the experiment, as I really want this service to work for us.

Thanks Michael P!

4th gen dell xps13 + ubuntu

My employer Angaza Design (we’re hiring!) recently bought me a new 4th gen Dell XPS13 developer edition. Naturally, I opted for the version with Ubuntu 14.04 preloaded and what Dell calls a “QHD+ (3200 x 1800) InfinityEdge Touch Display”.

Initially I had some mild annoyance since overall HiDPI on Linux is not totally ready for prime time. The biggest issue was getting a non-4K external monitor working properly, but after a few days of struggle, things are working reasonably well enough to describe for others.

Habits from my old life as an Ubuntu distro developer remain, and my goal was to do the least amount of customization possible to be productive, since in my experience, hardware dies at random and inopportune times, and being too tied to a highly customized OS installation means you waste too much of your life tweaking stupid things.

Thus, the minimal set of tips to make the machine usable (for my setup) are:

Finally, to fix the mismatched DPI settings between the HiDPI built-in display and your external monitor:

xrandr –output eDP1 –auto –output HDMI1 –auto –panning 3840×2400+3200+0 –scale 2×2 –right-of eDP1

I saved that to a script and run it manually whenever I hotplug my external monitor, although certainly you could hook it up to a udev rule so it fires automatically. Your exact settings may be different, so check this Arch wiki page for more details.

Between those few updates, and my #ubuntugolf score of 10 (initially forgot to mention stgit), the XPS13 is a great developer’s machine, and you don’t have to carry a Mac like all the other web 2.0 hipsters.

Nice work to the respective Dell and Ubuntu teams.

[there’s also a github project to track xps13 issues; most of them I have not personally been affected by]

Collinsium ciliosumpus


Gotta hand it to the CBS headline writers: Spiky armored worm had 30 legs, will haunt your dreams.

A spiky, wormlike creature with 30 legs — 18 clawed rear legs and 12 featherlike front legs that likely helped it filter food from the water […] is one of the first known animals on Earth to develop protective armor and to sport specialized limbs that likely helped it catch food.

I guess as humans, we’ve managed to develop Under Armor and the opposable thumb, but somehow, we gave up after only 2 legs, none of which are clawed or featherlike. Sad times.

Happy Platypus Friday!


Long-beaked echidna (Zaglossus bruijni) crossing water, Papua New Guinea , endemic, New Guinea highland forests

Long-beaked echidna (Zaglossus bruijni) crossing water, Papua New Guinea , endemic, New Guinea highland forests

Thanks to the fine folks at Wired for running their own feature, “absurd creature of the week”. They recently featured the echidna which:

lay eggs like reptiles and detect electrical fields like sharks. They’ve got the mug of an anteater and the spines of a porcupine. And of course the males have four-headed penises like … well, maybe something out of hentai?

But even more fascinating is their link to synapsids.

Synapsid tree

You can see where we diverged from the reptiles a Very Long Time Ago, but the super cool part is that the creature below is more closely related to us than to a lizard. Somewhat mind blowing.

Dimetrodon skeleton

Happy Platypus Freedom Day!

margins of safety

It was a bluebird Sunday and I was a dead weight, stuck and swinging stupidly in space, 7 feet away from the nearest solid surface in any direction. My rappel was in the midst of going awry.

Continuing down would’ve resulted in guaranteed major injury if not outright death. Swinging inward 7 feet to the rock face, was literally impossible, and only Adam West and Burt Ward would’ve been able to pull themselves up the 25 feet back to the roof lip, underneath which I was dangling.

batman and robin

It was typically windy up high near the top of Donner Pass, and shouting back up to my three partners was a lost cause. They couldn’t see me nor I them; all they knew was that the rappel rope was still taut after an abnormally long time. All I knew was that the problem solving was not going particularly well.

We’d just finished climbing One Hand Clapping as two parties of 2, so we had two 70m ropes between us. The plan was to rap off the Lizard Ledge in two separate 35m rappels. I’d go first on the blue rope while bringing the yellow rope with me, so that I could set up the second rap, all the name of efficiently getting 4 people to the ground with minimal fuss.

E coiled the yellow rope into the standard butterfly backpack configuration and handed it to me. She made a joke about coiling it for her petite size and that I was too fat for it to fit properly. I tied it around my back and indeed the tails were on the short side; I tied them into the standard square knot but didn’t have enough tail to finish them off with overhand backups. But it was a beautiful day and we were already thinking about cheeseburgers.

I ran the blue rope through my tubular rappel device and slapped on my autoblock, which is a short 8″ loop of webbing, clipped onto my harness’s leg loop with a locking biner.


I asked E if she had an autoblock setup; she said “no, do you always use one?” Ever the witty one, I joked, “well, I value my life so yeah, I always use one”. Talk about your Vertical Limit levels of foreshadowing.

vertical limit cam

Before I started lowering, I asked another party of 2 who had just completed Touch and Go where the next set of anchors were. “Directly below you” was the response.

I was confused because I had just watched them climb up from a left angled ramp below me, say, 7 o’clock, and I was standing on the edge of a ledge. Directly below me at 6 o’clock was the underside of the ledge, which is to say, an overhang.

“You sure it’s not just down that ramp to my left, where you guys just came up?”

“Directly below you dude.”

“Hmm… ok.”

Dubious, I started rappelling down the ramp they’d just come up in the 7 o’clock direction, and after about 30 feet, I saw some rap chains to my right. They were indeed directly underneath the giant ledge, at 6 o’clock from my starting point. I guess brah was right, so I pushed off the ramp and flipped the blue rope over the nose of the overhang so I was hanging in a plumb line below my rappel anchor.

That was the beginning of the fuckery.

Find the poo

Click to enlarge

Click on the photo above. Zoom in and look for the poo just to the left of “7. Touch & go 3rd pitch .10a” That’s where I was, 20 feet below the roof.

As I bounced into free space, the janky square knot holding the yellow rope on my back came undone and the rope started sliding off into oblivion, where “oblivion” was defined as “the red circle with two black Xes just to the right ‘4’ where the orange and magenta lines intersect, stopping there only long enough to bounce off the rock towards the jaunty ‘1’ very far away where the green line starts”.

I caught the yellow rope with one hand, and after checking my autoblock a few times to ensure that I myself would not go shooting off to the aforementioned “oblivion” place, cautiously retied yellow. A little tighter this time.

The roof was large enough, and the wall beneath it steep enough, that I was at least 7 feet away from the vertical surface, if not more. Vainly, gamely, I attempted to swing my body a few times to try and make contact with the vertical wall but I never moved more than 6″ off vertical. Height-wise, the chains were at my eye-level, but they may as well have been in Istanbul.

So to recap, the roof was 20′ above. The chains were impossible to gain. And down we’ve already defined as “oblivion”.

I thought briefly about tying yellow to blue and continuing to rap down even further towards oblivion, but then decided that was an idiotic plan because it involved a huge unknown, that being complete lack of knowledge whether there were more chains below me and how far away oblivion was. Another party was on the ground, just starting up One Hand Clapping and I also entertained the thought of asking them for help, but they were 2 pitches down and unlikely to reach me any time soon.

Up was the only way. It took me about 5 minutes of faffing about to reach this conclusion.

I had a single length runner made of 1/2″ tubular webbing and a second single length runner made of fancy tech webbing, probably Dyneema.

The idea was to tie both runners into prusik knots above me, clip one into my belay loop, and the other was to be used as a foot step. I could step in the lower one, stand up on it, then slide the upper one higher. I’d sit in the upper one and slide the lower one higher. Wash, rinse, repeat.

Here’s what actually happened.

First, I tied the brake side of my rappel into a figure 8 on a bight, below my autoblock. I was about to introduce perturbations into my system and although I had reasonable amounts of confidence in the procedure upon which I was about to embark, it still didn’t feel great to contemplate the idea of my prusiks failing and also my autoblock failing and then that whole oblivion thing.

I used the 1/2″ tubular webbing as the upper sling first and discovered the hard way that I couldn’t get enough friction. When I weighted that prusik, it kinda slid down the rope a bit before reluctantly catching. After making 3 or 4 attempts at “push up, then slide back down on something that Really Shouldn’t Slide”, I decided this was an undesirable feature for my ascending system.

It turns out the skinny Dyneema webbing gripped the rope much more securely, so I reconfigured that sling to be the upper one.

Next lesson was that ascending with 2x single length slings is hard, because the foot sling doesn’t reach down anywhere near your feet and it is tricky to get enough height standing in it to move the upper sling any reasonable distance.

Additionally, the rope was still running through my rap device and my autoblock, and moving rope through that entire system was … strenuous.

A bit of experimentation revealed that the trick was to commit mentally to ascending. This meant flipping upside down in my rappel position so I could get my foot into the foot sling. Every time I did that, I checked in as the Mayor of Sucktown, because all the gear strapped to me including my friend the yellow rope kept shifting around and also because going upside down in your harness is not a natural position for a climber.

After getting the high foot, the other technique was to stand up very hard while pushing my foot out in front of me. This movement allowed me to push my waist higher, and thus slide the upper sling a longer distance every repetition.

Each rep rewarded me between 2 and 3 feet of height. The most annoying part was feeding the newly gained slack through my rap device and autoblock. Perhaps it would have been easier had I taken the autoblock off, but at that point in my life, that did not seem like a wise course of action.

Every 6 feet, I retied the 8-on-a-bight below my autoblock. Safety first, y’know?

In this manner, I was able to ascend 15 feet in the 20 minutes since I’d last departed the upper rap anchor. My plan was to gain the roof, give my screaming abs a break, and then figure out part 2 of the plan.

I was about 5 feet away from the roof when the party who’d come up Touch and Go rapped down the ramp on their rope. It turns out there were chains in the ramp after all, and that’s what they meant when they said “directly below me”.

After a brief bit of discussion, since the tail ends of my blue rope were actually in the ramp, we decided that he would grab those tails and pull me in towards the ramp. I’d lower back down on rappel and be guided into him, where he was clipped into the proper anchor.

Undoing my ascension system took another small effort, but soon enough I was going back down again, this time with tension on the tail ends of my rope so that I could get back into the damn ramp. Finally, I was able to clip the proper anchor and take myself off rap. It had been an annoying 20 or 30 minutes, and I was glad for it to be over.

Bringing the rest of our party down on the blue rope, and then to the ground afterwards from the 2nd rap station proved to be mostly uneventful, and we rewarded ourselves with bacon Sriracha jerky back at our packs.


To me, the biggest takeaway is that adventures (which later blossom into epics) always stem from errors in judgement.

I had seen that party climb up the ramp with my own eyes, and I knew that rappelling off an overhanging roof is not an action to be done cavalierly. Yet for some reason, I chose not to continue down the ramp and instead opted to flip my rope over the roof into a free hanging rappel.

That decision went against my instincts, and yet I did it anyway because I chose to listen to confusing words from a 3rd party rather than trusting my own judgement. I have no excuse for my brain wandering off into stupid-landia. I’m just glad it didn’t kill me.

That said, if you’re going to make stupid decisions, at least bring enough tools to extract yourself from said decisions.

In the context of rappelling, to me, this means ALWAYS using a backup knot of some sort. Whether it’s a prusik or autoblock or whatever else, the point is that rappelling is dangerous because it is committing. Minimally, you are committing yourself to your anchor, and if you are free hanging in air, you are committed to the rest of your system if something goes south. Your 5.12 climbing skills are irrelevant if you can’t touch the actual rock, and whatever your next move is, doing it without a backup means that you have a much smaller margin of safety.

In the past, I have not always rappelled with the two minimal slings I’d need to ascend back up the rope. After this incident, I think I’m just going to start doing so. In a trad climbing situation, you probably have enough slings on you anyway. It’s harder to remember in a sport climbing situation, but I’ve also had minor excitement rapping from a 10 pitch route in Potrero Chico where all you typically need is a rope and quickdraws, and we were lucky that we had at least one trad sling with us at the time.

For those keeping track, this means that I’m now bringing an autoblock cord and two slings for any non-trivial rap.

Next, the knowledge and skills to use your rescue gear. I was lucky in that there was a clear picture in my mind on how to go up the rope, and that I have used variations of the technique in the past, although never in a situation where I was hanging free and 100% committed to the ascension system. Even so, it took me a few valuable minutes to remember the optimal way to do it.

Not a problem on a sunny, cloudless Sunday, but not something you want to be trying to remember if thunderclouds are rolling in fast and your hair is crackling in your helmet. Do yourself a favor and practice at least once in a safe situation.

Not included in this discussion are the myriad of other ways that rapping can kill you, but on a final note, every multipitch climber really needs to learn the Munter hitch, which is the friction knot you’d use to both belay and rap with if you drop your belay device. If you don’t know this knot, you are going to be a very sad panda some day and we will be your sad panda friends.

Climbing is fun. Complacency kills.