Beyond GIGO

The Garbage In, Garbage Out principle is hard to apply to massive amounts of data. Some of the data will always be eventually flawed. And so the results will be too.

Ergo, algorithms that expect to work on perfect data will probably be less useful than ever.

Sigh.

Linux

How the heck can it be so easy?

[Kudos to the Desktop Linux architects and programmers!]

Why Debian is my favorite distro #17

I was struggling to get the PID of an SSH invocation that backgrounds itself (-f). It's being setup as a TCP forwarder in a script and I need to tear that down when the script finishes. Unfortunately, I couldn't find a clean way -- I ended up using a hackish "ps", but isn't there a better way? Can't ssh print its pid, like for example wget -b does?

Like that search shows, I'm not alone. This issue was reported as Debian bug #446932 in 2007. OpenSSH has it as the bug #1473 about six months later. Note that I tried the exec solution, but it did not work because ssh forks -- I hadn't read that bug report when I tried it -- but the "control socket" method seems workable.

oldstable (sarge) files going away ...

And just when I was creating my first ever Debian mirror (which has been crawling at 6GB/day). Oh well, at least this will test whether files get deleted.

Frustrated by the slow speed, I wrote a script ("kanadi") that improves upon one of my older scripts, by merging in files on DVDs into a Debian repository. I merged in amd64 and i386 of etch (the soon-to-be oldstable), but have been resisting doing so for Lenny.

Some thoughts on Windows HPC

Also known as Windows Compute Cluster Server 2003, or Windows Server 2003 R2 + Compute Cluster Pack (which we used).

I'm hugely grateful to the following communities:

• The Linux Kernel Team: for a kernel that detects all the exotic hardware that we pushed at it and works with all of them out of the box. Without requiring BIOS upgrades. Without hanging forever, with no indication of what went wrong.
• The Linux Kernel driver authors: for giving us 100% hardware performance out-of-the-box without any tuning whatsoever. Seems silly to praise this, but yes, thank you! And without "Quality Labs" certification too!
  
• The Debian GNU/Linux Project: for building one of the finest customizable distributions out there. And one that requires no license keys. And that downloads updates using standard mechanisms that allow me to accelerate update delivery.
  
• The FAI authors: For a system that installs an entire OS with applications and configures it in 4 minutes flat across all nodes (dependent only on Internet speed and network and disk transfer rates). And without borking on "incorrect" license keys despite us doing everything correctly.
• The OpenSSH authors: for a secure, remote shell that integrates very well with the system, allowing us to save loads of time, and doesn't have silly restrictions on the number of concurrent users.
  

Windows 2003: "The most productive platform ever". Yeah right.

And before I forget: to the people who implemented "Last Known Good Configuration", thank you!

Out of touch with reality

The administrator who expects systems to work out-of-the-box is either working on a single function device, ordering other people about, is seriously deluded, or maybe most of the above.

How to save any Flash Video to disk from Firefox in Linux

I've used youtube-dl to download videos from YouTube, but it can't handle, say, about.com videos. Or any of the other random video sites popping up all around the net. Turns out there is a very simple way to save any FLV file to disk in Firefox (as a friend showed me):

1. Type about:cache in the Location bar.
2. Click "Disk Cache Device", loading this page might take some time.
   
3. Search for the FLV file (Ctrl+F). You may have to look around a bit.
   
4. Once found, click the link, which will open up a "Cache entry information" page.
   
5. On that page, the "file on disk" parameter shows you where Firefox has stored that FLV file in its cache on disk. Now just cp that file to wherever you want it:
   
   cp /path/from/file/on/disk /path/to/where/I/want/FLV/file.flv
   

Incredibly simple and easy! (Tested on Ubuntu Feisty)

Programming Truths

Repeat after me:

Programming is difficult

Again, for effect:

Programming is difficult

Here something else to ponder about:

Just because the computer seems to do what you wanted it to do, does not mean you've written the program correctly.

A analogy to a pack of cards that collapses because one card wasn't properly placed at the bottom is helpful to understand the above.

And above all:

If your code isn't working, it's not perfect.

Never come up to me and tell me "My code's perfect, but it's not working". Please re-parse that sentence to understand how silly it is. I've lost count of the number of people who tell me that. The computer makes mistakes, but more often than not, your code's not right. Let alone perfect.

And yes, of course:

Perfect code is a thing of beauty.

Never forget that.

Supporting Linux

Many of my friends have recently switched to Ubuntu, making me the tech-support guy again. Not that I like to play that role, but if there's something new to learn, then I'm game!

Unfortunately, the support has largely been anti-climatic. When the wireless wasn't working on a friend's laptop, he lugged it to my lab, only to find out that the "wireless enable" switch was off. Turning it on got it working immediately.

Another friend recently installed Ubuntu, and he has a TV tuner. It worked out of the box, except that there was no sound. We finally tracked it down to a mute AUX mixer control. Raising the volume there got that working again.

The second could be attributed to bad defaults -- the AUX control isn't shown by default, and is muted too, but frankly there are no sane defaults in this case.

Thankfully both of these friends do read manuals and explore, so helping them is not quite the pain it usually is.

There have been more "complex" problems that I've handled, installing the Flash and Java plugins on a 64-bit machine. Neither are officially supported (Java's 64-bit JRE ships without the browser plugin!), so problems are to be expected. For Flash, nspluginwrapper worked like a charm, but for Java, the only recourse was a custom 32-bit install of Firefox.

Tech support is boring, especially when you've done it again and again. And these days, if the problem is something that Google can solve, I refuse to solve it. Sometimes rudely.

Amazon EC2 is out of beta!

So Amazon's Elastic Compute Cloud is finally out of beta, meaning anybody can sign up for it. Now, you can hire a computer on the net for 10cents an hour. Want 10000 machines computing for you? It'll cost you only $1000 or INR 50,000.

You don't have to worry about where these machines will live, how they're setup, who pays the electricity bills -- nothing.

Does this mean that I now win the NetFlixPrize? Maybe...

The IA64 Experience

The machine in the Supercomputing Bay is old. Older than most people in the lab. Yet it has impressive specs. Four Itanium processors. 8GB RAM. Two 80GB SCSI HDDs. And Red Hat 7.2.

This Itanium "server" was condemned to that slow death that is inevitable when you install a distribution that doesn't support in-place upgrades. It grows so old, software-wise, that nobody wants to use the perfectly good hardware anymore. So, last week, when a friend wanted the latest kernel on this machine, I decided to install Debian Etch onto it.

The first thing, of course, is that Itanium machines don't have a BIOS bootloader. Like the Intel Macs, they have EFI. Think of it as a inbuilt Grub, but with lots of additional features. For example, it can boot over the network using PXE directly. There is a command prompt with commands faintly reminscient of DOS commands. It even features a full screen text editor that you can use to edit files on a FAT partition. And yes, it takes a glorious 100MB partition for itself. No more 512-byte boot sectors for EFI.

Unfortunately, this EFI bootloader couldn't boot from the ATAPI slave CD-ROM, without a master.

So it was upto network installation to save the day. Debian installed onto the IA64 over the Internet nicely (yay for Mbps internet speeds!), and booted up just fine. Like all Linux distributions, it uses elilo to boot on the IA64 -- Grub is not supported. However, elilo seems much nicer than its cousin which has been supplanted by Grub on
the x86.

Here are somethings I learnt:

• The Pause key is useful for paging through output at the EFI console.
  
• If the "EFI Shell [Built-in]" option does not drop you at a shell, but starts up elilo instead, the file startup.nsh might exist. You can press CTRL+C quickly after selecting EFI Shell to get to a prompt. This will cancel elilo and allow you to delete/edit the startup.nsh file.
  

Simplicity and the Beast

Go read the WSJ article.

This blog post is about the sentence that says:

"If you're a wonk and you want lots of controls and features, Microsoft is right for you," Mr. Kay says. "If you want a simple experience and you're not tech-savvy, then you'll probably do better with Apple."

Why is this wrong?

It's simply the wrong way to think about the UI for an application.

There are two issues here: Features and configuration. Features govern what go into the program. The configuration controls which features are turned on, and how they behave. Programmers will usually put in all the features they can imagine, and then produce a hideous UI that controls which feature is enabled and how.

But that's the lazy way out. As a programmer, you'd reduce a lot of your load if you figured out which were the most common features used by your users and in which configurations. Then by choosing such a configuration as the default, you're going to make a lot of your users happy because they don't have to fiddle with your program's settings. Note that it's not *all* users, it's a majority of users. For the fringes, make sure you have configuration options, but I'm willing to bet that the UI that results will be a lot more manageable.

One story I've heard tells of a programmer who released a program with some configuration options accidently removed. Nobody complained. In fact, nobody even noticed. That was sign of overdesign. He promptly ripped out the features in the next release. Less code == less maintenance. The original Unix programmers were sparse in their outlook, adding features only when asked to, not mis-anticipating user's demands.

GNOME 2.0 went this way, and I'm extremely happy they decided to take it that way. It's no less powerful, but I spend time working instead of tweaking my GNOME configuration. Ubuntu pared down the menus, bringing down the number of image editors to one from maybe a dozen, one web browser, etc. The rest are still available, but most people use Firefox, so why install Elinks and clutter up the menu?

The only program that I use that is overflowing with features is Amarok. Thankfully, it can be used in a simple mode to play my music and I use the exotic options only rarely. But it's a remarkable program able to tackle everything I've thrown at it (iPods, album art, synchronization, queues, auto-mounting, replay gain, etc.) but I fear as these features will become more "standard", other programs will include them in the GNOME fashion -- it just works -- and I'll move away.

In short, let the users do their work, make your program get out of their way.

And it isn't the number of buttons that count.

Suspend works again!

Decided to spend a peaceful Sunday pursuing Feisty's inability to suspend my laptop.

After a dozen reboots and no solution in sight, I decided to install the latest 2.6.22-8 kernel from Gutsy. I'd tested suspend with 2.6.22-6, but that hadn't worked. But with 2.6.22-8, it worked! Kinda. The nv driver doesn't bring up the backlight, so I had to go back to the nvidia driver. But now it works!

Regressions: bcm43xx doesn't really work with my card, and ndiswrapper is not included in 2.6.22-8. Dunno why. The changelog has this cryptic comment "No longer provide ndiswrapper or ivtv modules (l-u-m does)". Unfortunately, I've no idea what l-u-m is!
Attempting to recompile ndiswrapper results in some dependency problems which I'm in no mood to fix (requires linux-headers, but linux-headers requires libc6 2.6, etc.).

Ah well. It's still a couple of months away.

Update: l-u-m is linux-ubuntu-modules, which I found out by following the dependency chain from linux-generic.

Intel BIOSes require the Bootable flag on partitions

Not much on my tech blog these days, but here's a quick solution to what seems to be a common gotcha. We recently got new machines for our lab, and they all run a 946 chipset. The installs went smoothly with grub getting installed, but after rebooting, the BIOS refused to boot from the hard disk, pretending as if it wasn't there at all!

It turns out the BIOS on these machines refuse to boot from the hard disk if none of the partitions has the bootable flag set. This is a bit crazy, and can make you go hunting for unconnected hard disks and stuff like that.

Here's the quickest solution, for the above mentioned problem:

1. Boot from CD, I used the Ubuntu CD.
2. Choose "Boot from Hard Disk" or equivalent.
   
3. In the resulting grub menu, press 'c' to get to the prompt, and execute the following commands, where hd(X, Y) is your boot/root partition:
   

   
   root (hdX,Y)
   makeactive
   

   This will set the bootable flag on (hdX, Y).
   
4. Press 'Esc' to get back to the menu, and boot normally.
   

Does Windows have that?

Sometimes I wonder, on seeing some wonderful new feature in Linux, whether Windows possesses the same feature. The feature this time, was dynamic ticks, that headline grabbing, gut-wrenching (atleast for 2.6.21) patch that allows Linux to save a lot more power than your Windows laptops. Or does it? I mean, if Windows has that feature, then Linux has just "caught up", and I can't claim advantage.

Thanks to Kernel Comparison: Linux vs Windows -- whose motto is "Everything you wanted to know about your kernel, (but were afraid to ask)" -- I now know that Windows does not support dynamic ticks.

Useful reference that.

An Open Source Google Base

I think Google Base is far too good an idea to see only an implementation from Google, accessible only through their APIs.

Is there an open source clone?

Or wanna build one? *grin*

At the very least, it'd require the following components:

1. A frontend to convert and store the XML data into the backend. See the "* Feeds" reference documentation for the formats.
   
2. A query engine to understand and execute the URL query syntax. At high-speeds. This is the hard part. See also: Query Lang Specification.
   
3. A mechanism to store data in such a way to enable high-speed lookups as well as support all those different types of data. This is the very hard part, in my humble opinion.

Think of it, at a minimum, as a way of putting an URL query mechanism to an existing database like PostgreSQL. We could probably take it from there.

What's the use? Well, the web could do with a database service that can do everything Google Base can, but without any Google-specific idiosyncracies. Plus, it'd be compatible with Google Base!

Unlike most of my projects, this one is not so well thought out, as you can probably see from the fact that I don't even have a name for this project yet. Suggestions?

Why FOSS is Better, Yet another Case Study

Recently developers at Intel released a tool called PowerTOP. For users familiar with top on Unix systems, this tool is similar, except it displays information on programs that are "waking" up the CPU often. What does this mean? To cut a long story short, it means you can now identify which programs are interrupting a CPU's sleep, and determine if that interruption could have been handled in a better way, letting the CPU sleep longer. That way, battery power could be saved.

Why is this tool a good example of Free Software at work? Not only did the developers create this tool, they also identified many free/open source software programs where things could be done better. Because the source was available, they were even able to make patches -- or modifications -- available so that battery life was extended by "an hour or more!" This is amazing. Imagine doing this on your insert-favorite-proprietary-operating-system here. It'd take years for the OS vendor to get it, then the programmers to get it, and then for them to get together and do it.

Why is this on my blog? I use liferea, a feed reader that manages my RSS feeds, and the latest release has this cryptic changelog: "This release decreases CPU power consumption for laptop users". My life is already better, thanks to PowerTOP.

Oh yeah, bonszai says, "Thank you, Intel!"

Summer Surprise

This is a PicoPeta Simputer...you are assimilated!


blob version 2.0.5-pre2 for PicoPeta Simputer

Copyright (C) 1999 2000 2001 Jan-Derk Bakker and Erik Mouw

Simputer platform Copyright (C) 2002 Vivek K S, PicoPeta Simputers Pvt. Ltd.

blob comes with ABSOLUTELY NO WARRANTY; read the GNU GPL for details.

This is free software, and you are welcome to redistribute it

under certain conditions; read the GNU GPL for details.

Memory map:

  0x01000000 @ 0xc0000000 (16 MB)

  0x01000000 @ 0xc8000000 (16 MB)

Loading blob from flash . done

Loading kernel from flash ... done

Autoboot in progress, press any key to stop ..

Starting kernel ...


Uncompressing Linux............................................ done, booting the kernel.


Simputer login: root
[root@Simputer /dev]$cat /proc/cpuinfo /proc/meminfo
Processor       : Intel StrongARM-1110 rev 8 (v4l)
BogoMIPS        : 127.38

Features        : swp half 26bit fastmult

Hardware        : Picopeta-Simputer
Revision        : 0000
Serial          : 0000000000000000
        total:    used:    free:  shared: buffers:  cached:
Mem:  31494144 19243008 12251136        0  1429504  9195520
Swap:        0        0        0
MemTotal:        30756 kB
MemFree:         11964 kB
MemShared:           0 kB
Buffers:          1396 kB
Cached:           8980 kB
SwapCached:          0 kB

Active:           3452 kB
Inactive:        11716 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:        30756 kB
LowFree:         11964 kB
SwapTotal:           0 kB
SwapFree:            0 kB

[root@Simputer /dev]$

Saturday, May 12.

The Feisty Fawn Upgrade

I finally did it last Thursday, upgrading from Edgy to Feisty. I thought it would be fun to relax by installing Feisty after a rather tough academic session. Oh boy, was I wrong! Regressions, a root file system corruption, and the eventual re-installation kinda disappointed me. But I'm running Feisty now! Here's how it went off.

After burning the DVD, I noticed the cdromupgrade script on the DVD, and decided to run it, instead of the normal aptitude dist-upgrade. A couple of clicks and one hour later, my system was upgraded to Feisty Fawn. Such simplicity -- I thought -- it can't be this easy! Yet it was. In a way.

Unfortunately the reboot failed. It would come to the boot screen and simply hang. Turns out, I had installed a custom kernel in Edgy, which was also set to be the default. This was simple to solve. Just select the Feisty kernel in the grub boot menu, and later reconfigure the previous kernel, dpkg-reconfigure linux-image-old, and it booted fine. I guess upstart's not really working well with the old kernel's initrd. Boot time is now 45s, down from 55s for Edgy.

The nv driver, as I mentioned in an earlier post, works perfectly. The coolest feature about X.org 7.2, is its ability to run without a xorg.conf file. Just remove the xorg.conf file, and see it run perfectly! It did, for me, atleast. I did have problems with the scrolling on the touchpad, but that was easy to fix, just add options psmouse proto=exps to /etc/modprobe.d/options.

Unfortunately, there were two regressions. The most evident, and a show-stopper for my laptop was support for the headphone jack. Unlike Edgy, where it wasn't even recognized, and I had to install the latest ALSA, in Feisty, both speakers and headphones work, but have a single volume control. Unfortunately the speakers aren't muted when I plug in my headphones. Incredibly annoying, as my labmates will testify, as I blasted Jonny Quest's theme song again and again. Feisty ships with ALSA 1.0.14rc1, and I had to compile and install 1.0.14rc3 from Debian, which was, thanks to make-kpkg, a breeze. Now, plugging in my headphones automatically mutes my speaker, and vice-versa. It's not perfect yet, I've to trigger the detection by muting/unmuting, but this is how it should be. The speakers also seem to sound much louder. FYI, the sound card/codec is a NVIDIA/Conexant CX20549 (Venice).

The second regression was Suspend/Hibernate. I use Suspend rarely, and it used to work beautifully in Edgy. Unfortunately it's broken in Feisty. I first thought it was a problem with nv, but the nvidia driver has the same problems. Also it turned out that my forced shutdowns ended up in me corrupting the root filesystem. Ouch.

Recover or re-install? Well, I always wanted to re-install Feisty (just for kicks), and here was the opportunity. So I wiped the root partition, and re-installed Feisty. Took about another hour, and I'm still re-installing apps -- but thanks to aptitude that's not much of a pain. All of my settings and data were not affected -- thanks to the good fortune of having a separate /home partition. What is a painful is that main in Feisty does not fit on the DVD, and you'll need the web repositories too. Irritating.

Oh yes, beryl 0.2.1 doesn't work either. The window decorator (emerald) doesn't work.

After all of this, what's Feisty worth to you? If you're using Edgy on a laptop and it's working, I recommend you continue. However, Feisty's GNOME feels much faster and lighter (and is too!), and having the latest and greatest applications always feels good. As I said, fun, but not relaxing. It works, but keep your fingers crossed!

What's really different about this install is the amount of software I had to add to make it all work. For Feisty, only ALSA. For Edgy, I had to do ALSA, ndiswrapper, network-manager, and some other stuff I've forgotten. So, this is progress. Two steps forward and one step backward, but progess nevertheless. Now to wait for the Gutsy Gibbon.

Update: May 7, 2007 Monty is my hero! Here are the options to add to the screen section in xorg.conf to get beryl working:

Options "AddARGBGLXVisuals" "True"
Options "DisableGLXRootClipping" "True"

Flash 9.0 on 64-bit Ubuntu with Firefox 2.0

I'm watching the Mac vs. PC ads on Youtube, in the 64-bit Firefox 2.0 browser. This shouldn't be working, Adobe does not have an official 64-bit Flash player for any operating system, let alone Linux. You ask how? Here's the trick:

1. Get nspluginwrapper sources from the Debian website.
2. Compile and install.
3. Get the Flash 9.0 player from Adobe.
4. Untar, and copy libflashplayer.so and flashplayer.xpt to ~/.mozilla/plugins/
5. Run nspluginwrapper -i ~/.mozilla/plugins/libflashplayer.so
6. Enjoy!

I've done this before, with an older version of nspluginwrapper, but could never get sound working. Maybe the fact that I used RPMs (converted by alien) messed things up. Even now, however, nspluginwrapper segfaults. Thankfully it works for nspluginwrapper -i, which is all I that's required.

As an aside, swfdec is a free software Flash viewer that plays Flash videos, and so is gnash.

Furthermore, youtube-dl allows you to download the flash videos to your hard disk, where mplayer or ffplay (better) can play them.