alex chiang: web 6.0

Thursday started off with a keynote by Stormy Peters titled “Would You Do It Again For Free?” Some interesting things came out of this talk. First, open source developers are mainly motivated by internal factors. Second, studies show that giving external rewards to people who are doing an activity and later removing the rewards tends to lead to cessation of the activity. Put the two concepts together, and all of a sudden, the idea of, “are companies killing open source by paying for it?” whacks you across the face like a 2×4.

Unfortunately, Stormy shifted focus slightly during this talk, and started talking about how company processes tend to squash developers’ creativity, thus demotivating them, which is still an interesting topic, but not the one she started with. The idea here is that companies need to figure out how to balance traditional corporate process with the creativity and individualism needed to develop software.

I did shoot off an email to Stormy about the topic shift and we ended up chatting for a bit later that evening at the networking event, and we agreed that both topics were interesting, but she was really focusing on the latter.

I skipped the tutorial sessions because I couldn’t build a kernel for Ubuntu that had lguest and wireless both working at the same time. I only mention lguest because that’s the session I’d wanted to attend; I’m sure my personal problems were all contained in ipw2200.

During the afternoon, I went to Nick Piggin’s NUMA pagecache replication talk, and came away with mixed impressions. In a nutshell, it’s a performance feature that replicates a cache coherency algorithm in the operating system rather than in the hardware, and at the page granularity rather than cacheline granularity. So not a groundbreaking concept, but it does take serious hacking-fu to implement it correctly (in other words, please don’t think that I’m trying to detract anything from Nick at all).

My impression was that getting it all working correctly was really hairy and complicated, and that scared me. I’m not entirely convinced that it will actually lead to performance gains at least not in the world I typically inhabit, that being large ia64 SMP machines, but may help on smaller x86_64 machines. Nick doesn’t have any performance numbers yet, so it’s all somewhat up in the air as to how well it’s all going to work, and it’ll be interesting to see where it all ends up.

Next was HP’s own Doug Chapman (drc) doing a kickass demo of his autonomous rockhopper robot to a packed room. All sorts of cool hackery was demonstrated, including streaming live video from his workshop in New Hampshire to Melbourne (with an assist from his neighbor Paul). At least one audience member remarked that it was perhaps the best talk of the conference that he’d seen. Cool beans!

I went to davem’s “Linux on Sun logical domains” talk but didn’t get anything out of it. The fault was mine, not his, but in any case, I checked out kinda quickly.

Last talk of the day was conceptually the most mind-blowing for me. Vik Olliver presented his RepRap which is essentially an open-source rapid prototyping machine, aka a 3-d printer. The printer itself was kinda cool, but more interesting were the concepts around the ability for basically everyone to own a universal fabricator. Think about it — all of a sudden, a lot of problems go away if you can just make stuff on demand, on location.

No more holding inventory, no more transporting finished goods around, different tax implications (people aren’t buying stuff anymore, they’re making stuff for themselves), no ability to embargo or prohibit goods, etc. etc. etc. The implications are powerful.

One last note on why LCA is seriously cool — my G1G1 OLPC was having some problems with the control key getting stuck (a known issue) so Jim Gettys just swapped mine out for a new one. Schweet.

A few notes on Wednesday…

Bruce Schneier keynote address today. Cool stuff, although nothing groundbreaking (due to years of reading Crypto-Gram and his blog). Some important security concepts

• security is best viewed via an economic lens; specifically, we are in a lemons market, aka information asymmetry.
• in this market, we are all security consumers, making economic tradeoffs (should I wear this bullet-proof vest today? no, it’s kinda hot, and besides, it would clash with my belt)
• the concept of feeling secure vs. being secure is quite important; divergence between the two is where all of our security problems come from essentially
• society (broad term to include social conventions, technology, etc.) is evolving faster than our species. human brains aren’t equipped to be good judges of risk anymore.
• the only way to get out of the current mess is by disseminating more information, and reducing the information asymmetry

I went to Jonathan Oxer’s Second Life talk, and was a bit disappointed, although this was entirely my own fault. I was hoping to get an explanation of why a sane person would care about Second Life. Jonathan’s talk was predicated upon the assumption that one already cared about Second Life and he explained how to do insane things with it.

I should have read his abstract closer, because it was all there. Regardless, it was still quite an entertaining talk (Jonathan is an excellent speaker) and amazingly his demo worked on the first try! Yowza!

Oh, you want to know exactly what insane things he was talking about, probably. Well, the talk mainly focused on attaching a programmable Arduino board to various appliances to your house so that when a real life switch is flipped, it triggers an event on the Arduino which sends some http request to a web server somewhere that then kicks off a Second Life script that then makes a Second Life object do something.

For instance, Jonathan has hooked a switch up to his real life snail mailbox so that when letters are placed inside, lots of magic happens, and then in his Second Life world, his fake mailbox’s flag raises up and it looks like it’s stuffed with snail mail. Someone from the audience suggested hooking up a scanner to the mailbox so he could actually read the mail in Second Life, and of course this idea was extended to then pipe the scanned text through some OCR tool and then run a spam filter on the scanned text so you could actually do automated Bayesian filtering on your real life snail mail.

Like I said: insane.

Next talk was Jonathan Corbett’s Linux World News “State of the Kernel” talk. Jonathan is a good speaker too, but I really shouldn’t have gone to this talk considering I already read lkml, and I was the wrong audience. Oh well, my fault again. I did notice that HP was not one of the top 20 organizations contributing to the kernel (as measured in changesets). Boo.

I attended Jim Gettys’ OLPC talk. Again, poor talk selection on my part because he was preaching to the choir (me). It was interesting to hear the emphasis that the number one inhibitor for laptops in the third world is power. They’re exploring the use of solar panels to replace the hand crank (he had demos of both), but power issues dominate the technical difficulties for actually deploying these things.

By the way, have you noticed that power consumption happens to be a huge issue in “normal” laptops right now? Oh, and in the data center too. As jwz would say, intertwingularity! It sounds completely ridiculous, but theoretically, a motivated kernel hacker who sits down with PowerTOP could do more to save the world than Al Gore ever could. A geek can dream, right?

Last talk of the day was the best. HP’s own Bdale Garbee gave a great talk on peace, love, and rockets. It isn’t every day that you get to hear a CTO of a $100B company talk about the difficulties in getting JTAG working on the ARM chip which he surface-mount soldered onto a two-layer PCB that he designed himself. Fascinating stuff, and I’m pretty sure that as a company, HP most definitely wins the “My CTO has a bigger um, dongle, than your CTO.”

Ok, enough! This entry has dragged on long enough, and I need to sleep.

The nerdlinger portion of Tuesday concluded with me wandering back over to the kernel mini-conf to catch the lightning talks and the kernel panel.

Nothing super interesting at the lightning talks (not to detract from them, but they were basically plugs of upcoming LCA events or brief overviews of patch sets or upcoming changes, so nothing earth shattering).

The kernel panel was slightly better, with two very interesting problems and one very humorous moment.

First problem — is it reasonable to ask a user to perform a git-bisect to help a kernel hacker debug a problem? Lots of debate back and forth in the room here, and my personal take is, “probably not”. Joe Average doesn’t have multiple computers — he has his one machine that he wants to play Quake on, and your kernel bug is preventing his machine from booting and fragging or gibbing (or whatever the hell they call it) his opponents. Each git-bisect is going to take two reboots, and it usually takes, what, 7 tries on average to find out what commit broke his machine? Add on top of that the overhead of installing git, cloning a kernel, building said kernel, installing it and the initrd correctly, bla bla bla… man, that makes me tired just thinking about it.

Perhaps some work can be done in this space to write some tools to help automate this process. It would be a win-win, in my opinion, as kernel hackers are certainly going to save time since they’re doing bisects all day long, and Joe Average won’t be intimidated by the arduous process and will get back to his happy state of blowing up professional Korean Quake players all the sooner.

Second problem — how to track incoming bugs? A thousand bugzillas dot the landscape, and it’s difficult to get them to talk to each other. The fundamental problem is that the underlying database schemas are often different, so it’s not just a simple matter of forwarding bugs to one master bugzilla. I really believe that the distros need to show leadership here and work together to create some set of common schema that they can all agree on to solve this problem. It’s not a technical issue — it’s a human issue. Of course, other developers hate bugzilla, prefer to do everything out of email, and that’s fine too. But for those people using tools, let us, as a community, come to some agreement on the best way to use those tools.

Funny ha ha — question was raised about upstream acceptance of kernel debuggers. Some talk in the room ensued, and at one point, someone on the panel suggested slipping the patches in now, since Linus might not be paying so much attention. In a beautiful moment of comedic timing, willy says, “oh, hi there, we were, um, just talking about ice cream!” just as Linus himself had snuck in the back of the auditorium. Linus seemed to just wave and smile, but I think the “wave” was really that Darth Vader “use the dark side of the Force to mentally death choke” David Miller from across the auditorium. I saw davem twitch a little bit when Linus raised his hand, so it’s the only reasonable conclusion.

Final mini-conf talk of Tuesday for me was tbm’s “Release Management in Large Free Software Projects”. The most interesting part of this talk for me was seeing all the parallels between shipping free software and shipping proprietary software.

It turns out that while free software development is radically different from proprietary software development, trying to ship the end result is remarkably similar, due to the foibles of human nature.

In the end, what large free software projects strive for is… wait for it… yes! predictability! woo! Hey, that’s basically what proprietary software products try to provide too. So fundamentally, the problem space is equivalent.

tbm’s research showed that many large free software projects have settled on time-based releases to try and achieve predictability, and then gave a bunch of examples. Again, I won’t repeat his talk here; go to his website to read tbm’s entire thesis.

The interesting implication for me is that management at proprietary software companies shouldn’t despair about the growing momentum in free software development. Rather, they should view it as an opportunity — management at top-notch software companies know how to ship software, so they ought to be thinking of ways to marry their management expertise while leveraging the benefits of open source development models.

Perhaps there’s a business model in there somewhere, where a strong management team can make money by shipping open source software to interested buyers with some level of predictability and responsibility. And yes, you’re reading me correctly if you think that I’m implying that this management team startup idea is marketing both to developers (”hey, let us manage your project and sell it to customers!”) and buyers (”hey, let us sell you this excellent piece of software!”).

Food for thought, anyhow.

I got away from the kernel mini-conf for the early part of the afternoon to check out the distro mini-conf. HP’s own Bob Gobeille gave a talk on FOSSology that turned out to be quite entertaining.

The basic problem they’re trying to solve can be described as: what content, exactly, is in a file, and does that content resemble anything that we might care about?

Buh?

Ok, some examples might help. Let’s say both Fedora and Debian are shipping a library, zlib say, and you run ldconfig on both distros and both times, ldconfig says v1.2.3. Cool, they’re shipping exactly the same library right?

Erm, maybe. Who knows what patch levels each distro might be shipping, etc. Source level inspection is pretty much the only way to figure this out.

Let’s say you’re some poor schlub teaching assistant at Uni because the brochure tricked you into thinking that you’d go to some prestigious institution to get a great education, but in reality, you have to teach data structures 500 indifferent undergraduate halfwits. The half clever bit of wit that all those undergrads have is to figure out who the nerd in the class is that actually understands what a pointer is, promise to get him some girl’s phone number, copy his program, but only hand it in after cleverly changing all the variable names. Genius! The same algorithm used to analyze zlib above can be applied to help detect exactly this sort of cheating independent of variable names. Zing!

Oh, and you can use the algorithm to figure out such mundane things as what free software license any given source file might be distributed under. And it turns out that’s actually the reason HP wrote this tool, but the general underlying principle is extremely powerful because it’s so simple.

The grand vision behind FOSSology is to create a huge repo of source files, analyzing them for… whatever. Licenses, code reuse/originality, dependencies, etc. And HP gets it, which is really cool — we’re going to hand this repo over to the Linux foundation and allow anyone to poke at it.

Good job guys, and good job bobg for the entertaining talk.

The second talk I attended on Tuesday was “Cache Efficient Data Structures” given by Joern Engel. Personally, I found this talk to be very interesting on several levels, but primarily because it focused on the nexus between theory and implementation that happens to excite me (yes, I am a dork. A good-lookng dork.).

The key takeaway from Joern’s talk, IMO, was that computer science theory is dandy in the classroom, but when it comes to actually implementing stuff, theory is insufficient.

The classic data structure that computer scientists prefer using to store information is the hash table, because it can be proven mathematically that operations such as insertions, removals, and lookups all have the perfect tradeoff in performance.

Unfortunately, in real life, it turns out that once a data set grows large enough, it doesn’t really matter what you do with it so much as how you access it. In other words, the penalty incurred from a cache miss completely dominates the theoretical running time of data structure accesses.

This means that while a computer scientist is happy to use a hash table to store a million objects, a kernel hacker realizes that the number of cache misses incurred when inserting / removing objects is going to be sufficiently large (as you can only fit one hash table object per cache line) to the point where performance will suffer. You’re going to be spending all your time flushing your caches instead of doing useful work. And may the gods help you if you’re on some sort of multi-cpu cache-coherent system with concurrent readers and writers. Egads.

More importantly for a kernel hacker, that memory doesn’t even belong to you; it’s the user’s memory! Bad hacker! Bad!

The solution is to use more exotic data structures like radix trees or perhaps a B tree variant. The memory characteristics of these structures happen to be much more cache friendly (lower memory overhead and higher number of objects per cacheline), and the gains are worth the increased complexity when managing these structures.

Two quick final notes (watch the video if you’re interested; I’ll not repeat it all here): willy mentioned a crazy data structure called an “xor list” which is basically an optimization trick on a doubly-linked list where you xor the forward and backward pointers together to save yourself one pointer’s worth of overhead, and second, Judy hashes (or whatever they’re called) could be cool if normal humans could understand them. As it is, probably only the author of the library is the only person on this planet that actually understands how they work. The verbatim quote from the slide:

Judy trees: !#@$$%@##%^&& ????

Might be an interesting project for the motivated individual, to rewrite this library with a usable API.

Started off Tuesday at the kernel mini-conf. Unfortunately, due to a typo in the printed program, I missed a talk I was kinda interested in seeing (How not to invent kernel interfaces by Arnd Bergmann), and didn’t get there until hch’s talk on writing a PCI driver using qemu.

hch showed a lot of boilerplate code saying that much of it was “trivial, trivial”, and it certainly did seem like it, even to a PCI newbie such as YT. On the qemu side, it was some more “trivial boilerplate code” and again, it all seemed pretty straightforward. He mentioned that the RealTek 8139 would be a good example on the qemu side to try and learn from. He also pointed out that qemu was written by a former winner of the Obfuscated C contest, so the only way to really understand it is to get the swords out and start cutting your way through line after line of uncommented code. Good luck.

Unfortunately for me, I wasn’t smart enough to understand the implication of *why* one would want to use qemu to write a fake piece of hardware and then use it to write a device driver. hch pointed out that it was much easier to simulate hardware using C and qemu compared to writing out VHDL (which I certainly agree with), but I didn’t have any “ah ha” moments from a kernel guy perspective. Oh well, I’m certainly not as smart as hch, so maybe this is something useful for him, but I just didn’t get it.

On a side note, hch seems pretty personable to me; surprising, given his kinda bad guy reputation on lkml. Before his talk, we happened to be sitting outside the lecture hall, and he had no problems watching my bag whilst I got a coffee. Maybe the secret to making him like you is by not writing any SCSI code and certainly not showing it to him if you did.

LCA is about letting nerds be nerds, and what’s more nerdy than a GPG keysigning party? (If there exists such a beast, I sure don’t know about it.)

The original plan was to use the Sassaman projected method, but the projector wasn’t so great (projected the everything mirrored, which made it rather hard to read the text on the IDs), so after a little bit of confusion, the decision was made to do it the old fashioned way: by standing in a circular queue, checking the IDs of the person in front of you, and rotating. To be honest, I kinda liked that method a bit better because it was much more personal than the Sassaman method.

madduck pulled his “Transnational Republic” trick again, and I guess I fell for it. I asked for his ID, he showed it to me and said, “this is from the Transnational Republic” and I said, “ok, looks good to me”. Hey, I’m just a dumb ‘merkin, I figured it was some crazy Eastern European Slavic country I’d never heard of. Not my fault, says I.

A short break to recuperate, and then I tagged along on a small, ad hoc dinner that ended up as a mix of HP (bobg, tpot, tbm, achiang) and Debian developers (DDs: pasc, madduck, and tbm (tbm wears two hats)). Didn’t talk about anything groundbreaking; mostly revolved around the possibilities and challenges of debian in the corporate world, and things HP might be able to do to help improve on that front (we could start by providing non-broken m11y .debs!).

It was interesting for me to see the discussion evolve into something resembling corporate Linux guys vs keepers-of-the-flame, with bobg and YT taking the lead for $MEGACORP and madduck on point for the hippies. What made it interesting was the challenge in explaining that even though we’re corporate droids, we still believe in the Right Thing ™ and actually try to effect change to the extent that we can, but that there are many internal corporate challenges when someone as large as HP are signing your paystubs. I remember being on the other side of that fence while in University and wondering why the corporations just didn’t get it.

This is not to say madduck is naive; I don’t believe that to be the case at all (although I do find him to be a somewhat representative example of the true believer mentality, aka, smart, rational, and holding the unfortunate assumption that the world is populated with other rational agents who can be reasoned with, (which in my opinion, is sadly and shamefully is not the case)). Until one works for a large $MEGACORP, one doesn’t truly appreciate the complete dichotomy between developers, whose goal in life is to enable others, and lawyers/managers, whose goal in life is to avoid risk. If you think about it, those goals are almost perfectly at cross-purposes, and more often than not, those with the money (aka, lawyers/managers) win the battle.

But that’s mostly self-evident anyhow and not saying anything interesting. What might be interesting for those on the other side of the fence is the realization that the lawyers and management at $MEGACORP a) are humans too b) can actually be quite clueful when it comes to open source and c) want to effect change for the greater good as much as anyone. But change doesn’t come quick in gigantic companies, and it may be a useful analogy to remind ourselves that the patient, steady dripping of water can eventually create the Grand Canyon.

Well, enough of the sycophantry for now. Time for bed and day 2 of LCA2008.

I was kinda disappointed with this talk. The first speaker basically just said, “big companies! give debian money!”, and as the audience tried to understand why, he couldn’t really give good reasons. The question “what, exactly, are your goals as regard to corporations — what do you want?” was posed several times in different forms, and the answer was basically, “we want more resources”. Ok, right. On the plus side, he did plug HP several times, although he seemed to think that HP only ships debian on embedded systems. Not true — we ship ia64 debian for telcos and proliant debian too.

The second speaker was slightly better. He explained what it would take for a corporation to support debian, and basically, it came down to: “they would have to reproduce the effort that Red Hat and Novell are already doing”. Duh. Enterprise linux is simply a different ballgame than consumer linux, and after the explanation on why, I came away believing that no sane corporation would ever want to truly make debian a first class supported distro, at least not in the general case.

The reason why debian is great is exactly the reason why it will never be a good fit for corporate America. Because no one is really in charge, any investment a company might put into debian is going to have a very low ROI due to lack of control. This is not an aspersion; it’s simply the way things work.

The speaker asked for magic solutions, and YT spoke up and suggested that the problem was not going to be solved by debian developers begging corporations for support, but by lobbying ISVs to code to debian as a reference platform. My point was: no business customer buys a distro for the sake of the distro; they purchase it on the basis of the applications. A CxO does not care that his sysadmin loves debian because it’s so easy to administrate if Oracle and friends won’t run on it. The speaker did not seem to agree with me, but I had some good air cover from the rest of the peanut gallery (thanks ggg for immediately grokking my point!).

In any case, my heart will always be with debian, but I walked away from this talk thinking that the future of corporate debian isn’t so great (at least not in the near term).

Presented by Dave Jones, another good talk. Some notes:

• Fedora is going to start providing a vanilla upstream kernel in their yum repos to make reproducing bugs easier. This solves the problem of, “oh, you saw a problem in fedora? does it reproduce upstream?” Any distro that maintains custom patches should implement this solution.
• Fedora 9 is tentatively aimed for May release, which is approximately the 2.6.25 timeframe. Woo!
• One of the most anticipated features for F9 will be the stabilization of the on-disk format for ext4. This is a big deal for early adopters, since it would really suck if they started using ext4 and all of a sudden the format changes. Oops!
• Starting to change thinking about drivers that are not yet accepted upstream. Fedora is going to start shipping them because “a crappy driver is better than no driver”. Doesn’t mean it’s a free-for-all — there are still baseline requirements, like there needs to be an active maintainer, etc. Still, this is a good sign for the users.
• Xen continues to be a headache. (Does anyone like xen?) Parts of Xen have finally been accepted upstream (after 3 years), but what’s currently shipping in F8 today is actually more featureful than what’s upstream, as the Fedora guys maintain their own bunch of backports and patches. This is the case for almost every distro, I think. There might be a nice way out of it — running a xen domu on top of kvm. Clever and clean, in my opinion.
• gonna start supporting kernel mode setting. This provides a way for the kernel to change / modify the graphics display (yes, X will need to change to support this). Why would you want this? In case you get a kernel panic while in X — instead of just seeing your screen lock up without reason, the kernel can now dump panic info to your graphical console. Booya.
• better kgdb / kdump integration. akpm is already carrying kgdb patches in -mm, davej wants to see these merged. Anything that makes debugging easier is good. kdump is pretty cool and shipping in F8 today (modulo whatever setup that needs to happen). The scripts could use some improvement, and my impression was that there’s going to be something better for F9.
• RHEL6 may or may not be based on F9. Seems like internally, they’re treating it as if it were to be so, which is a pretty good hedge. At worst, all it means is that F9 will be really good. Still, there’s a chance it’ll be based on F10, but in any case, pretending that F9 is the new base of RHEL6 would seem a prudent course of action.
• Big problem is gathering enough debug information. One of the problems is that Fedora adds the “quiet” argument to the kernel cmdline, so users can’t tell where their kernel might hang. Educating them on how to modify /etc/grub/menu.lst and turn that off, etc. etc. is hard. Yours truly suggested providing a “debug mode” (similar to safe mode) which turns quiet off and boots a debug kernel as a grub menu option, but davej said that the Fedora grub guys are pretty resistant to adding new options (don’t want to confuse the users). They’ve got a point, but there must be a way to hide that sort of magic normally until a developer asks you to press the magic key combination. Someone else suggested just making safe mode do exactly that, which is a good idea too. Real time open source, brosephs!
• Other bug related things — installing Arjan’s oops daemon to help get more info, setting up a netconsole, and perhaps something like bug-triage day. Good ideas, all.

Yowza, that was a lot.

Edit: hrm, better update this entry so people who did not come here via a link from Dave Jones’ LCA entryget the right ext4 information. Dave writes:

(The bit about ext4 is somewhat optimistic. There’s a ton of stuff left to do for ext4 before it loses its ext4dev moniker. I possibly misled people into believing we’re shipping this in F9 regardless of it state due to the pending changes for .25 The reality is that until the ‘dev’ goes away, I’m paranoid about more last minute on-disk format changes.)