san francisco santacon, 2011

I’m happy to announce that a few packages I’ve been working on over the past year have finally landed in Ubuntu Precise[1].

If you have a 3G USB modem, and it currently doesn’t work well (or at all) in Debian or Ubuntu, you should check this list of modems[2]. If it listed, then you may be a candidate to try an alternative 3G networking stack.

$ sudo apt-get install wader-core

This command will remove ModemManager and install wader-core. It should be an entirely transparent operation to you, except that after you reboot, your modem should appear as a connection option in the network manager applet.

Yay!

###
1: naturally, I was a good boy and uploaded the packages to Debian unstable first
2: this list is predominantly composed of Vodafone-branded modems, but there are others in there as well.

Thanks to the Debian python team for mentoring me and to Al Stone and dann frazier for even more mentoring in addition to sponsoring me.

I’m giving up on social networks. Between G+, Flickr, Twitter, Path, LinkedIn, and whatever else, it’s just too damn hard to manage them all. It feels like work, which is exactly opposite the point of being connected to my friends.

Whither Facebook? No, sorry. They don’t give you control over your own stuff. When computer nerds talk about “bla bla bla privacy”, just replace it all with the following question — do you have control over your stuff? Facebook doesn’t give it to you, at least not as of today, in the amount that you deserve. So no matter how much other shiny stuff Facebook might tempt you with, it’s not worth it because you’re lending them your things without the promise you can get them back. If they change in the future, great — I’ll happily change my mind. But until then, they can go fuck off.

So, what to do about photos? Because every superfluous service ditched is a step closer to sanity. After wasting two hours of my life, I’ve discovered it is possible to use Google Plus Photos to mostly replace Flickr for my primary use case, which is hosting images for my WordPress blog.

Why Google Plus?

• Lots of storage space. And it’s free too, which is a bonus. (I pay for Flickr Pro currently; it’s been worth it, but I’m just getting fatigued. Sorry, Flickr!)
• You get to choose who sees your photos.
• You can download all your photos in a batch. You can let others do so too. This is great if you want to take your ball and go home if Google makes you mad, and it also allows Auntie Susana to grab all your photos to make a calendar or mug or whatever with.
• You can embed the photos in your blog.
• It’s super duper easy to upload photos from your phone. In fact, you can turn on automatic uploads from your phone.
• You can specify a Creative Commons license for your photos. This is important to people like me who care about the open flow of information and data.
• You can get stats via Google Analytics.

So what’s the catch?

The catch is that you can’t actually do those things from the Google Plus interface today. You need to actually go into Picasa Web to manage your photos and.

With Picasa Web, you can:

• arrange the photos in an album in the order you want
• set a cover photo for the album
• grab the link to an individual photo to embed elsewhere, like a blog
• set up analytics tracking

Don’t even bother trying to figure out how to do those things in Google Plus Photos. You can’t, at least not today. Luckily, Picasa Web does see all the photos you’ve uploaded into Google Plus; you can manage them in Picasa, then flip back over to G+ to share with your circles if you want.

Feature for feature, Flickr Pro is moderately better. But in the bigger picture of “too damn many social networks”, something’s gotta give. Again, I’m sorry Flickr. I loved you for a while, and will always remember you fondly. But it’s time to consolidate and simplify.

For my normal friends who don’t keep up with understandably nerdy topics such as this, the summary is that:

• many smartphones include a nasty piece of code called Carrier IQ that can track all your SMS, your calls, your every keystroke, and upload that to your wireless provider
• Carrier IQ is on both Android and Apple phones, although reportedly it has been removed from recent iOS builds
• Carrier IQ is confirmed to be on Sprint and AT&T Android phones

It is completely unclear what data is being recorded and sent back to Big Brother. You just have to trust Sprint when they claim they are being responsible stewards of your data.

Excuse me while I shit dysenteric diarrhea (and sprinkle in extra amoebas for dramatic effect) into a gallon-size freezer-safe Ziploc bag and mail it to my good friends in Overland Park, KS.

I mean, the unremovable, botched abortion apps they force onto you, such as the NASCAR wart, the crashy Sprint TV dung pile, the oozing pus of a football app, were bad enough, but at least you could mostly ignore them (except when you got mysterious random notifications that the Sprint TV app crashed again without you ever touching it).

And to lump another piece of garbage like Carrier IQ in there… I would pretend to be shocked that a cellular carrier would show such a level of disrespect for its customers, but why bother.

Anyhow, enough of the ranting. Here’s how to completely re-install your phone with something called CyanogenMod which is Android without the crap. Let’s not delude ourselves — using a cell phone means you’re sending data to Big Brother, but at least with CyanogenMod, you can get a bit of a choice in what you send.

Update: Slate asks if you should even care and Manjoo comes to the same conclusion: it’s the lack of transparency that rankles.

Update 2:: Dan Rosenberg does more research on what CIQ can and cannot do. Great analysis, but as he notes, it only applies to one handset (Samsung Epic 4G Touch) on one carrier (unnamed). Carriers can apparently make CIQ do different things, so who knows what was happening on my HTC Evo 4G on Sprint? We simply don’t know. To reiterate: it’s the lack of transparency that alarms me.

Anyhow… These instructions assume Linux.

1. Download and run revolutionary. Be sure to run it with sudo, else your phone will reboot into fastboot (as expected) but your computer will just sit there waiting for something (unexpectedly). When revolutionary (on the PC side) prompts you whether you want to install ClockWorkMod, say yes.
2. revolutionary will automatically complete on the PC side, and your phone will be sitting there with a white screen and some menu options. On your PC, download the Superuser app. Do not unzip it; just leave it there for now.
3. On your phone use the volume key to navigate to the “bootloader” menu item, then select “recovery”. Your phone will reboot into the ClockWorkMod screen, which has a dark background.
4. In the ClockWorkMod menu, select the “mounts & storage” menu item, then “mount USB”. This will make your phone appear as a drive on your PC
5. Copy the Superuser zip file from above onto your phone. You can just drop it into the root directory, you don’t need to put it into a folder.
6. Unmount the phone from the PC
7. On the phone, go back to the top level menu, and then select “install zip from sdcard”. With some intelligent guessing, you should eventually be able to select the Superuser zip file on the phone; select it and install it.
8. After this is complete, you can reboot your phone.

You have now gained root on your phone, which gives you special magical powers. It is time to use your new magical powers to install CyanogenMod. But first, back your phone up. Some recommended apps are:

• Titanium Backup
• SMS Backup & Restore
• Call Backup & Restore

I also had my phone set to sync all my contacts with my Google account.

Once you’re reasonably confident you have your data backed up, you’re ready to install CyanogenMod. This will obviously wipe out your entire phone and void your warranty, but hey you don’t have to be annoyed at the infuriatingly uninstallable NASCAR app on your phone anymore.

1. Install CyanogenMod on your phone. You can find it in the Market.
2. Download latest image onto your PC
3. Copy the zip file onto your sdcard again, by mounting your phone in your PC
4. Launch the CyanogenMod app, and find the zip file.
5. After some intelligent menu guessing, your phone should reboot and install CyanogenMod.
6. Important: be sure to select “wipe data/factory reset” and “wipe cache” in the install, else your phone will hang on reboot

You’re done. Hopefully you can restore your data from backups.

After playing with it for an afternoon, Cyanogen is actually a lot nicer than the HTC Sense interface.

And, you get USB internet tethering for free. Woo!

If you run into errors, ask for help in either #revolutionary or #cyanogenmod on freenode. Good luck!

Last January through April, I pretty much fell off the face of the earth, in real life as well as online. For those that asked, I alluded to some long hours at work, but of course couldn’t say much publicly.

Well, we finally launched, and I’m quite proud of all our team accomplished.

Without question, this was the hardest project I’ve taken on in my career to date. But I was part of a great team, and we pulled together to ship.

We’re bringing free, open software to the world. This is the mission I signed up for.

Some links for your reading pleasure, take with a grain of salt:

• Vodacom press release
• OMG! Ubuntu mentions it
• TechCrunch adds commentary
• Engadget rehash
• Tom’s Hardware article

The easiest workaround I could figure out was the following:

$ cat ~/.pbuilderrc-debian
COMPONENTS="main contrib non-free"
DEBOOTSTRAPOPTS=("${DEBOOTSTRAPOPTS[@]}" "--keyring=/usr/share/keyrings/debian-archive-keyring.gpg")

And then you can issue:

sudo pbuilder create --basetgz /var/cache/pbuilder/sid.tgz --distribution sid --mirror http://mirrors.kernel.org/debian --configfile ~/.pbuilderrc-debian

The better way to fix this of course, would be to fix above bug. But this works for now.

It is also a fact that email is a skill, and there are varying levels of proficiency out there.

So, it is only a matter of time before you find yourself on the annoying end of an email thread gone awry. Perhaps it is a discussion on the wrong mailing list, or perhaps it is the infamous 1 grillion people in the To: or Cc: fields problem.

Before long, a “take me off this list” / “stop replying to all” storm ensues, and then something horrible like Facebook gets invented to “solve” this “problem”.

Of course mail filters can be deployed, shielding you from the idiocy. But what if you want to be more proactive? Is there a way to stop the insanity without having to hax0r into the mail server and just start BOFH‘ing luser accounts?

Yes, there is an easy solution that works most (but not all) of the time.

Put all the unintended recipients in the Bcc: field. Put the correct recipients in the To: field.

In the case of discussion on the wrong mailing list, this is easy; just put the correct list in the To: field. Include a note in the mail body, such as “Redirecting to foo list, which is more appropriate.” Respondents will then typically automatically respond to the correct list.

In the case of “too many Cc:s”, there’s no easy answer. You could move all the Cc: to Bcc:, and then put something like none@none.invalid in the To: address. You will get a single bounce, but then so will everyone else who attempts to respond to you. This trick only works because the people who tend to cause the problem also tend to be lazy and just respond to the last mail received. They can’t spam everyone else because their addresses are obfuscated via the Bcc:. If you feel brave, you could socially engineer the recipients by writing something inflammatory, in order to entice them to respond to you, rather than other mails in the thread, which will then result in a bounce.

Hope this helps.

[nb, .invalid is actually a reserved domain, read rfc2606 for more details.]

Sometimes though, I want to read old logs, either to remind myself of a conversation I had with someone, or to dig out a URL, or whatever. bip keeps these logs around, but they can be annoying to read.

Here’s an example of how bip stores the logs:

achiang@complete:~/.bip$ ls -R
.:
bip.conf  bip.pid  logs

./logs:
bip.log  canonical  freenode  oftc  sekrit

./logs/freenode:
2011-03  2011-04  2011-05  2011-06  2011-07

./logs/freenode/2011-03:
achiang.30.log     #coherellc.31.log      #ubuntu-devel.31.log
chanserv.30.log    #ubuntu-motu.30.log
chanserv.31.log    #ubuntu-motu.31.log
#coherellc.30.log  #ubuntu-devel.30.log

They’re just free form text, which is good, because you can then use normal tools like grep on them. Unfortunately, they’re also full of long, noisy lines that look like:

achiang@complete:~/.bip$ head ./logs/freenode/2011-03/#ubuntu-devel.31.log 
31-03-2011 00:01:48 -!- zeeshan313!~zeeshan@119.153.35.115 has joined #ubuntu-devel
31-03-2011 00:03:05 -!- T0rCh__!~T0rCh_rao@187.104.99.84 has quit [Remote host closed the connection]
31-03-2011 00:06:23 -!- holstein!~holstein@unaffiliated/mikeh789 has quit [Ping timeout: 240 seconds]
31-03-2011 00:08:23 -!- m_3!~m_3@cpe-72-179-48-240.austin.res.rr.com has quit [Ping timeout: 276 seconds]
31-03-2011 00:08:48 -!- abhinav-!~abhinav@122.161.12.85 has joined #ubuntu-devel
31-03-2011 00:12:02 -!- holstein!~holstein@71-90-232-189.dhcp.gnvl.sc.charter.com has joined #ubuntu-devel
31-03-2011 00:12:03 -!- holstein!~holstein@71-90-232-189.dhcp.gnvl.sc.charter.com has quit [Changing host]
31-03-2011 00:12:03 -!- holstein!~holstein@unaffiliated/mikeh789 has joined #ubuntu-devel
31-03-2011 00:15:35 -!- andreasn!~andreas@117.192.217.128 has joined #ubuntu-devel
31-03-2011 00:20:32 -!- TeTeT!~spindler@178-26-84-220-dynip.superkabel.de has joined #ubuntu-devel

So just viewing them in an editor can be annoying.

And that was a rather long intro to describe what is one of the world’s most trivial scripts (which has at least one known bug :-/ ). But anyway, I call the snippet below “bipcat”:

#!/bin/sh

cat $1 	| grep -v "has quit" 		\
	| grep -v "is now known as" 	\
	| grep -v "has joined" 		\
	| grep -v "has left" 		\
	| sed 's/!.*:/:/' 		\
	| cut -f 2- -d' '

And now, you can get much more sensible output:

achiang@complete:~/.bip$ bipcat ./logs/freenode/2011-03/#ubuntu-devel.31.log | head
00:54:45 < dholbach: good morning
01:05:16 < pitti: Good morning
01:58:29 < pitti: should bug 685682 be closed with the new fglrx that we landed yesterday?
01:58:32 < ubottu://launchpad.net/bugs/685682
01:59:08 < didrocks: it seems that cnd still have that issue with the driver and workarounded compiz
01:59:41 < didrocks: anyway, there is still a need for a compiz upload which will come with other fixes (probably Monday)
01:59:48 < pitti: ah, thanks
02:00:09 < tseliot: the fix should be available in the next upload of compiz (it's already available in a daily PPA)
02:00:28 < pitti: so I guess for now the fglrx tasks should be closed then?
02:00:30 < didrocks: yeah, but as told, it's not working on cnd's machine, I asked him to check with you and jay

Much nicer!

[the bug is that the sed line does a greedy match, so it replaces everything up to the last ':', which is clearly not the right thing to do if someone actually typed in a ':'. suggestions for improvement welcome]

The classic example is if you are in charge of delivering software to a group of machines, but do not have source code to the software. Maybe you just have a pre-compiled library you want installed somewhere.

You could ask your users to:

$ sudo cp mylib.so /usr/lib

But then what if you need to update mylib.so somehow? I can see the nightmare a-comin’ from all the way over here.

So then you think to yourself, gee, I have a very nice package management system; why don’t I use it? Which means, you’re going to try and teach yourself the bare minimum debian packaging skills needed to do such a thing, to which I say, good luck.

Perhaps there are easy examples out there [and if so, let me know and I'll update this post]; in the meantime, this is the bare minimum that I could come up with.

Hope it helps.

The directory layout follows:

achiang@aspen:~/Projects/example$ ls -RF
.:
debian/  usr/

./debian:
changelog  compat  control*  copyright  install  rules*  source/

./debian/source:
format

./usr:
lib/

./usr/lib:
mylib.so

Of course, we have the debian/ directory, which is where the magic happens. But the other top-level directory, usr/ in our case, is what we want to install on the target system.

The easiest thing to do is to re-create your target system’s filesystem layout as a top-level directory, and then put your files in the appropriate spot. Here, we want to install mylib.so into /usr/lib on the target system, so you can see that I’ve recreated it above.

If you also wanted to install, an upstart script, say, you’d also create:

$ ls -RF
.:
debian/  etc/  usr/

./etc:
init/

./etc/init:
myjob.conf

Ok, next let’s look at the stuff in debian/:

achiang@aspen:~/Projects/example/debian$ cat rules
#!/usr/bin/make -f
%:
	dh $@

That’s pretty easy. How about the control file?

achiang@aspen:~/Projects/example/debian$ cat control
Source: example
Section: libs
Priority: extra
Maintainer: Alex Chiang 
Build-Depends: debhelper (>= 7)
Standards-Version: 3.9.1

Package: example
Architecture: any
Depends: ${misc:Depends}
Description: A skeleton installation deb
 An example, minimal package that installs files into the filesystem, without
 any processing from the packaging system.

Ok, one more interesting file, the ‘install’ file:

achiang@aspen:~/Projects/example/debian$ cat install
usr/

The usr/ entry in ‘install’ maps to the usr/ directory you created above. Again, if you also wanted to install something into etc/, you’d add the corresponding line into ‘install’. Extend this concept to any other directories/files you’ve created.

The rest of the files are more or less boilerplate. I’ll display them for completeness’ sake:

achiang@aspen:~/Projects/example/debian$ cat compat
7
achiang@aspen:~/Projects/example/debian$ cat copyright
This package was debianized by Alex Chiang  on
Mon Jul 11 14:30:17 MDT 2011

Copyright:

    Copyright (C) 2011 Alex Chiang

License:

    This program is free software: you can redistribute it and/or modify it
    under the terms of the the GNU General Public License version 3, as
    published by the Free Software Foundation.

    This program is distributed in the hope that it will be useful, but
    WITHOUT ANY WARRANTY; without even the implied warranties of
    MERCHANTABILITY, SATISFACTORY QUALITY or FITNESS FOR A PARTICULAR
    PURPOSE.  See the applicable version of the GNU Lesser General Public
    License for more details.

On Debian systems, the complete text of the GNU General Public License
can be found in `/usr/share/common-licenses/GPL-3'
achiang@aspen:~/Projects/example/debian$ cat source/format
3.0 (native)

So, there you have it. A pretty trivial example on how to package a binary inside of a debian source package. Of course, you could do this with text files, PDFs, whatever.

Feedback appreciated.

Let’s say you’re a sysadmin deploying or a developer writing a twisted app. It’s likely the app is writing a bunch of logs somewhere. The problem is that the default logging settings for twistd are set to write an unbounded number of log files, so over time, it’s possible to fill up your filesystem with old logs.

developers
As a lazy developer (or more likely lazy bug fixer who is just trying to fix something and move on with life without having to learn yet another huge framework because let’s face it, you have gtk, glib, firefox, and kernel bugs that need fixing), you google and find the quick basic help on logging in twisted. You read:

If you are using twistd to run your daemon, it will take care of calling startLogging for you, and will also rotate log files.

So you think “top banana”, and merrily invoke your program with:

from twisted.python.release import sh
sh("/usr/bin/twistd --nodaemon --logfile=%s" % "myapp.log")

And you’re done and it’s time to go to the pub. Not so fast, Speedracer. Let’s take a look at the defaults (I checked twisted 10.0 and 11.0):

137	class LogFile(BaseLogFile):
138	    """
139	    A log file that can be rotated.
140
141	    A rotateLength of None disables automatic log rotation.
142	    """
143	    def __init__(self, name, directory, rotateLength=1000000, defaultMode=None,
144	                 maxRotatedFiles=None):
...
234	class DailyLogFile(BaseLogFile):
235	    """A log file that is rotated daily (at or after midnight localtime)
236	    """

See that maxRotatedFiles=None? It means you will eventually hit -ENOSPC, and pandas will be sad.

A little more digging, and reading through the twisted application framework howto, you get the hint on how to modify the default logging behavior. The example says:

The logging behavior can be customized through an API accessible from .tac files. The ILogObserver component can be set on an Application in order to customize the default log observer that twistd will use.

Ok, so you look at the example, and then you say to yourself, that’s pretty good, but Sidnei da Silva could improve it, and then you bug Sidnei for a few hours, asking him rudimentary questions about whether the whitespace in Python really matters or not, and then he politely gives you the following rune (all the ugliness below comes from me, anything elegant comes from Sidnei):

from twisted.application.service import Application
from twisted.python import log
from twisted.python.logfile import DailyLogFile

application = Application("foo")
log.startLogging(DailyLogFile(maxRotatedFiles=1).fromFullPath("/var/log/myapp.log"))

And that kinda works without any fancy observers or anything, but then you realize that your codebase has the following pattern repeated across many hundreds of source files:

from twisted.python import log
...
log.msg("unique message 1 out of 47890478932423987234")

[This is the point where I just gave up trying to learn twisted in 45 minutes or less, and also wanted to stress eat a cheeseburger that is 180% beef, 120% bacon while forgetting everything I thought I knew about how percentages worked.]

You think to yourself, “this is Linux, and I wouldn’t be using this OS unless my stubbornness rating was +8″ so you go bash around on google for a while before remembering that Linux has a tool to take care of all this for you already called logrotate. So you stop trying to wrestle with twisted and beg your downstream packagers to fix it for you. Example below.

sysadmins
Luckily, twisted’s built-in log rotation mechanism matches up with what logrotate expects, namely, that when it rotates log files, it will do things like:

foo.log -> foo.log.1
foo.log.1 -> foo.log.2

So, all you have to do is teach logrotate about your application’s logfiles, and you are done. The beauty is, all the standard logrotate commands Just Work^TM.

On debian and Ubuntu, you can drop a new file into /etc/logrotate.d/. Not sure how the other distros package it.

Here’s an example file:

# cat /etc/logrotate.d/myapp
/var/log/myapp.log {
        rotate 0
        nocreate
        copytruncate
        missingok
}

This says:

1. Pay attention to this file, /var/log/myapp.log. twisted will try and rotate it to /var/log/myapp.log.1, myapp.log.2, etc.
2. We don’t want many log files around. In fact, we will only keep around myapp.log and myapp.log.1. Anything older gets auto-deleted
3. Don’t try to create myapp.log. Let the myapp create it on its own
4. Important! Don’t rename myapp.log to myapp.log.0. Rather, just copy it to myapp.log.1, and then clear out myapp.log so it’s empty. This is good because myapp might assume that its file descriptor is still valid and continue trying to write to it. Without this command, myapp might lose data because it’ll assume the original log file is still around and continue to write to it, even though it’s gone. It’s a little dangerous because it’s possible that you might lose some logged data between the time that logrotate copies the new log file and truncates the old file. Caveat administrator.
5. Dear logrotate, please do not puke if you do not find myapp.log, it’s ok, I’ll give you a hug.

And now, twisted and logrotate are playing nicely with each other. On Ubuntu, logrotate runs as a daily cronjob, so twisted shouldn’t get too far ahead by creating too many extra log files. Of course, if it does, you can just create an hourly cronjob or something even more special, but probably the real answer is to discover why myapp is creating more than 1MB of logs so quickly.

One last tip, you can experiment with the behaviors above by playing with twisted and logrotate at the command line, without needing a reboot.

To force twisted to rotate its log files:

# kill -SIGUSR1

To check what logrotate will then do:

# logrotate -f -d /etc/logrotate.d/myapp

Beware, the above doesn’t actually do anything, it just pretends. So to make it actually do stuff, but with verbose output:

# logrotate -f -v /etc/logrotate.d/myapp

And that, friends, is a thousand words on how to solve what should be a pretty simple problem but turns out to be way harder than necessary because you don’t know yet another framework. Good luck. As always, I recommend moAR bacon.

Innocent English-as-a-second-language typo? Or wry self-referential pun? You decide.