Beyond GIGO

The Garbage In, Garbage Out principle is hard to apply to massive amounts of data. Some of the data will always be eventually flawed. And so the results will be too.

Ergo, algorithms that expect to work on perfect data will probably be less useful than ever.

Sigh.

Linux

How the heck can it be so easy?

[Kudos to the Desktop Linux architects and programmers!]

Why Debian is my favorite distro #17

I was struggling to get the PID of an SSH invocation that backgrounds itself (-f). It's being setup as a TCP forwarder in a script and I need to tear that down when the script finishes. Unfortunately, I couldn't find a clean way -- I ended up using a hackish "ps", but isn't there a better way? Can't ssh print its pid, like for example wget -b does?

Like that search shows, I'm not alone. This issue was reported as Debian bug #446932 in 2007. OpenSSH has it as the bug #1473 about six months later. Note that I tried the exec solution, but it did not work because ssh forks -- I hadn't read that bug report when I tried it -- but the "control socket" method seems workable.

oldstable (sarge) files going away ...

And just when I was creating my first ever Debian mirror (which has been crawling at 6GB/day). Oh well, at least this will test whether files get deleted.

Frustrated by the slow speed, I wrote a script ("kanadi") that improves upon one of my older scripts, by merging in files on DVDs into a Debian repository. I merged in amd64 and i386 of etch (the soon-to-be oldstable), but have been resisting doing so for Lenny.

Some thoughts on Windows HPC

Also known as Windows Compute Cluster Server 2003, or Windows Server 2003 R2 + Compute Cluster Pack (which we used).

I'm hugely grateful to the following communities:

• The Linux Kernel Team: for a kernel that detects all the exotic hardware that we pushed at it and works with all of them out of the box. Without requiring BIOS upgrades. Without hanging forever, with no indication of what went wrong.
• The Linux Kernel driver authors: for giving us 100% hardware performance out-of-the-box without any tuning whatsoever. Seems silly to praise this, but yes, thank you! And without "Quality Labs" certification too!
  
• The Debian GNU/Linux Project: for building one of the finest customizable distributions out there. And one that requires no license keys. And that downloads updates using standard mechanisms that allow me to accelerate update delivery.
  
• The FAI authors: For a system that installs an entire OS with applications and configures it in 4 minutes flat across all nodes (dependent only on Internet speed and network and disk transfer rates). And without borking on "incorrect" license keys despite us doing everything correctly.
• The OpenSSH authors: for a secure, remote shell that integrates very well with the system, allowing us to save loads of time, and doesn't have silly restrictions on the number of concurrent users.
  

Windows 2003: "The most productive platform ever". Yeah right.

And before I forget: to the people who implemented "Last Known Good Configuration", thank you!

Out of touch with reality

The administrator who expects systems to work out-of-the-box is either working on a single function device, ordering other people about, is seriously deluded, or maybe most of the above.

How to save any Flash Video to disk from Firefox in Linux

I've used youtube-dl to download videos from YouTube, but it can't handle, say, about.com videos. Or any of the other random video sites popping up all around the net. Turns out there is a very simple way to save any FLV file to disk in Firefox (as a friend showed me):

1. Type about:cache in the Location bar.
2. Click "Disk Cache Device", loading this page might take some time.
   
3. Search for the FLV file (Ctrl+F). You may have to look around a bit.
   
4. Once found, click the link, which will open up a "Cache entry information" page.
   
5. On that page, the "file on disk" parameter shows you where Firefox has stored that FLV file in its cache on disk. Now just cp that file to wherever you want it:
   
   cp /path/from/file/on/disk /path/to/where/I/want/FLV/file.flv
   

Incredibly simple and easy! (Tested on Ubuntu Feisty)