MLA Wire

pg_sample: extract a sample dataset from a larger PostgreSQL database

2010-07-03T15:01:00.000-07:00

pg_sample is a PostgreSQL utility for making smaller versions of large databases.

download pg_sample 0.01

When you have a relatively large database (tables with, say, millions or billions of rows), it can be difficult to generate smaller datasets to work with, especially if foreign keys are heavily used.

That's where this script comes in. It will create smaller instances of each table along with any additional rows needed to satisfy foreign key constraints (circular dependencies are supported).

The script's operation closely resembles that of pg_dump. For example, assuming we have a large database named largedb, a smaller version could be produced with:


  createdb smalldb
  pg_sample largedb | psql smalldb

The smalldb would then contain a subset of largedb's data.

Here are the command-line options (many of which mirror pg_dump):


    -a
    --data-only
        Output only the data, not the schema (data definitions).

    -E *encoding*
    --encoding=*encoding*
        Use the specified character set encoding. If not specified, uses the
        environment variable PGCLIENTENCODING, if defined; otherwise, uses
        the encoding of the database.

    -f *file*
    --file=*file*
        Send output to the specified file. If omitted, standard output is
        used.

    --force
        Drop the sample schema if it exists.

    --keep
        Don't delete the sample schema when the script finishes.

    --limit=*number*
        The maximum number of rows to initially copy from each table
        (defaults to 100). Note that sample tables may end up with
        significantly more rows in order to satisfy foreign key constraints.

    --random
        Randomize the rows initially selected from each table. May
        significantly increase the running time of the script.

    --schema=*name*
        The schema name to use for the sample database (defaults to
        _pg_sample).

    --trace
        Turn on Perl DBI tracing. See the DBI module documentation for
        details.

    --verbose
        Output status information to standard error.

    The following options control the database connection parameters.

    -h *host*
    --host=*host*
        The host name to connect to. Defaults to the PGHOST environment
        variable if not specified.

    -p *port*
    --port=*port*
        The database port to connect to. Defaults to the PGPORT environment
        variable, if set; otherwise, the default port is used.

    -U *username*
    --username=*username*
        User name to connect as.

    -W *password*
    -password=*password*
        Password to connect with.

ip2host 1.11 Release

2010-05-09T15:51:00.001-07:00

A new release of ip2host is now available. It's a small maintenance release incorporating changes from the Debian package maintainer. The source repository has also been moved to Github.

http://github.com/mla/ip2host


DESCRIPTION

    Resolves IPs to hostnames in web server logs. This is a faster, drop-in
    replacement for the logresolve utility distributed with the Apache web
    server.

CHANGELOG

ip2host 1.11

  * Silence warnings thanks to Andrew McNaughton and Gunnar Wolf
    http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=563129

Installing Adobe AIR on 64-bit Ubuntu 9.10 Linux

2010-05-08T18:26:00.000-07:00

Adobe AIR isn't currently available for 64-bit Linux, but they provide instructions on running the 32-bit version.

I've converted the Ubuntu instructions into this shell script:
http://sites.google.com/site/mlawire/installing-adobe-air-1-5-on-64-bit-ubuntu-linux/install-adobe-air-ubuntu-64bit.sh

From a terminal:


wget http://sites.google.com/site/mlawire/installing-adobe-air-1-5-on-64-bit-ubuntu-linux/install-adobe-air-ubuntu-64bit.sh
chmod a+rx install-adobe-air-ubuntu-64bit.sh
sudo ./install-adobe-air-ubuntu-64bit.sh

This worked for me on Ubuntu 9.10. Please let me know if you try it on other Ubuntu releases or otherwise have trouble with it.

Keywords: Installing Adobe AIR 64-bit Linux Ubuntu

Doxygen Example

2009-10-12T22:37:00.001-07:00

Doxygen is a documentation system for use with many languages, including C++, C, Java, and Python.

First, install Doxygen. Using Ubuntu, I installed the package with:

$ apt-get install doxygen

Next. enter the source directory of a project you're working on and run:

$ doxygen -g

That will create a config file named Doxyfile in the current directory. You can customize it, but we'll accept the defaults for now.

Now begin writing your documentation. Here's an example of documenting a simple C++ program:


#include <iostream>

using namespace std;

/**
 * @brief Example class to demonstrate basic Doxygen usage
 * @author MLA
 *
 * This is a simple class to demonstrate how Doxygen is used.
 * It implements the Euclidean algorithm to compute the greatest
 * common divisor of two numbers.
 */

class Euclid {
  public:
    
    /**
     * Compute the greatest common divisor of two integers.
     *
     * @param a first integer
     * @param b second integer
     * @return greatest common divisor of a and b
     */
    static const int gcd(const int a, const int b) {
      if (0 == b) return a;
      return gcd(b, a % b);
    }
};

Finally, run doxygen, which will process the files and generate documentation in the html subdirectory:

$ doxygen

For more details, see the Doxygen homepage.

C++ Constant for PI

2009-08-17T23:20:00.000-07:00

Where's the definition for pi? Doesn't C++ provide a constant? What about C?

The C++ standard doesn't provide a value for pi but it's simple to define yourself:


#include <iostream>
#include <cmath> // M_PI is not standard

using namespace std;

class MathConst {
  public:
    static const long double PI;
};

const long double MathConst::PI = acos((long double) -1);

int main() {
  cout.precision(100);
  cout << "PI ~ " << MathConst::PI << endl;
}

Output:


$ g++ MathConst.cpp -o pi
$ ./pi 
PI ~ 3.14159265358979323851280895940618620443274267017841339111328125

That is accurate to 18 decimal places.

Here's a definition in C:


#include <math.h>
#include <stdio.h>

double pi() {
  const double pi = acos((double) - 1);
  return pi;
}

int main() {
  printf("%.20f\n", pi());
  return 0;
}

Postgresql: Indexes on Foreign Keys

2009-08-05T20:50:00.001-07:00

This query identifies foreign keys that are potentially missing indexes (Postgresql does not create indexes on foreign keys automatically).


/*
  Look for foreign key constraints that are missing indexes on the
  referencing table.

  Orders results by the size of the referencing table, largest first,
  on the assumption that, all else being equal, they are the most likely
  to benefit from the addition of indexes.

  This is only meant as a starting point, and isn't perfect.
  It's possible, for example, that it will report a missing index
  when in fact one is available. e.g., it won't realize that an index on
  (f1, f2) could be used with a fk on (f1). However, it will recognize
  that an index on (f1, f2) can be used with a fk on (f2, f1).

  Usage: psql -q dbname -f pg-find-missing-fk-indexes.sql
*/

CREATE FUNCTION pg_temp.sortarray(int2[]) returns int2[] as '
  SELECT ARRAY(
      SELECT $1[i]
        FROM generate_series(array_lower($1, 1), array_upper($1, 1)) i
    ORDER BY 1
  )
' language sql;

  SELECT conrelid::regclass
         ,conname
         ,reltuples::bigint
    FROM pg_constraint
         JOIN pg_class ON (conrelid = pg_class.oid)
   WHERE contype = 'f'
         AND NOT EXISTS (
           SELECT 1
             FROM pg_index
            WHERE indrelid = conrelid
                  AND pg_temp.sortarray(conkey) = pg_temp.sortarray(indkey)
         )
ORDER BY reltuples DESC
;

Linux: Download YouTube Videos

2009-08-01T19:32:00.000-07:00

Install the youtube-dl script. With Ubuntu:


sudo apt-get install youtube-dl

Next, find the YouTube video you want to download and pass it to the script:


youtube-dl -b -t "http://www.youtube.com/watch?v=aEXFUbSbg1I"

Command-line options:


Usage: youtube-dl [options] video_url

Options:
  -h, --help            print this help text and exit
  -v, --version         print program version and exit
  -u USERNAME, --username=USERNAME
                        account username
  -p PASSWORD, --password=PASSWORD
                        account password
  -o FILE, --output=FILE
                        output video file name
  -q, --quiet           activates quiet mode
  -s, --simulate        do not download video
  -t, --title           use title in file name
  -l, --literal         use literal title in file name
  -n, --netrc           use .netrc authentication data
  -g, --get-url         print final video URL only
  -2, --title-too       used with -g, print title too
  -f FORMAT, --format=FORMAT
                        append &fmt=FORMAT to the URL
  -b, --best-quality    alias for -f 18

Eight Ways to Convert Tabs to Spaces

2009-07-26T00:51:00.000-07:00

A collection of various ways to convert tabs to spaces in Unix/Linux with standard programs and utilities.


#1 expand/unexpand utilities
expand < input.txt > output.txt

#2 Sed
sed 's/\t/ /g' < input.txt > output.txt

#3 Awk
awk '{ gsub(/\t/, "  "); print }' < input.txt > output.txt

#4 Perl
perl -pe 's/\t/  /g' < input.txt > output.txt

#5 Ruby
ruby -pe '$_.gsub!(/\t/, "  ")' < input.txt > output.txt

#6a Vim editor
:set expandtab
:%retab! 

#6b Vim editor
:%s/\t/  /g

#7 Emacs editor
Set 'indent-tabs-mode' to nil to have tabs automatically converted to spaces.
To convert a region, "M-x untabify" will change tabs to spaces and "M-x tabify" will do the inverse.

#8 cut
cut -f1- --output-delimiter='  ' < input.txt > output.txt

Web: HTML Encoder

2009-07-25T22:24:00.000-07:00

This page will convert HTML special characters into HTML entities. The encoded text can then be inserted into another HTML document or blog posting and the reserved characters (such as < and &) will show up normally.

Enter your text in the form field below. The encoded result will be displayed in the lower field as you type.

This page requires Javascript. You currently have Javascript disabled.
Instructions for enabling Javascript on your browser

Enter HTML to Encode:

HTML Encoded Result: (Click to Select All)

The HTML encoding is performed with this Javascript function:

/* encode html entities */
var char2entity = {  '"' : '&quot;',  '<' : '&lt;',  '>' : '&gt;',  '&' : '&amp;',  "'" : '&#39;' }; /* IE can't handle &apos; */
function encode_entities(str) {
  var rv = '';
  for (var i = 0; i < str.length; i++) {
    var ch = str.charAt(i);
    rv += char2entity[ch] || ch;  
  }  
  return rv;
}

See also: handy shell function for html encoding.

Keywords: html encoding, html encoder

Web: Multiple Firefox Profiles with Linux

2009-07-25T18:21:00.000-07:00

To run multiple instances of Firefox simultaneously, or run multiple copies with different settings/plugins, start Firefox like this:

firefox -P -no-remote

That will bring up a dialog box that lets you select a profile or create a new one.

Other Firefox command-line options:


Usage: firefox [ options ... ] [URL]
       where options include:

X11 options
  --display=DISPLAY    X display to use
  --sync               Make X calls synchronous
  --no-xshm            Don't use X shared memory extension
  --xim-preedit=STYLE
  --xim-status=STYLE
  --g-fatal-warnings   Make all warnings fatal

Mozilla options
  -height <value>      Set height of startup window to <value>.
  -h or -help          Print this message.
  -width <value>       Set width of startup window to <value>.
  -v or -version       Print Firefox version.
  -P <profile>         Start with <profile>.
  -ProfileManager      Start with ProfileManager.
  -no-remote           Open new instance, not a new window in running instance.
  -UILocale <locale>   Start with <locale> resources as UI Locale.
  -safe-mode           Disables extensions and themes for this session.
  -jsconsole           Open the Error console.
  -g or --debug        Start within /usr/bin/gdb (Must be first)

Patch: Perl Module HTTP::Proxy 0.23 Hop-By-Hop Test Failure

2009-07-22T03:14:00.000-07:00

HTTP::Proxy 0.23 is failing test t/50hopbyhop:


t/50hopbyhop........1/28
# Failed test 'Hop-by-hop Foo'
# at t/50hopbyhop.t line 38.
# got: undef
# expected: 'foofoo'

# Failed test 'Hop-by-hop Bar'
# at t/50hopbyhop.t line 39.
# got: undef
# expected: 'barbar'

# Failed test 'Connection header removed'
# at t/50hopbyhop.t line 45.
# got: 'foofoo'
# expected: undef

# Failed test 'Connection header removed'
# at t/50hopbyhop.t line 46.
# got: 'barbar'
# expected: undef

I've submitted a patch that's been accepted by the author; it should be in the next release.

Update: the patch is included as of version 0.24.

Blogger: Exclude Your Visits from Google Analytics

2009-07-19T00:14:00.000-07:00

Google Analytics has a way of excluding certain visits to a site based on the presence of a special cookie. Many resources suggest that you create a "hidden" Web page to accomplish this, but that's difficult to do with Blogger. There is a workaround, however.

First, make sure you have Google Analytics installed and running on your blog.

Next, from each Web browser you use, pull up your blog. Once the page has fully loaded, enter the following javascript in the browser's location bar and press return/enter:

javascript:pageTracker._setVar('notrack');alert('cookie set');

That will set a "notrack" cookie on your browser.

Finally, sign into your Google Analytics account and create a custom filter to exclude all visitors that have the "notrack" cookie set. The filter values should be:


Filter Type: Custom filter > Exclude
Filter Field: User Defined
Filter Pattern: notrack
Case Sensitive: No

Google Analytics should now begin ignoring your visits.

This approach isn't specific to Blogger; it should work on any site, including other blogging platforms such as Wordpress and Movable Type.

Also, to make it simpler to set the cookie, you may want to create a bookmark with the above javascript. Once added, simply go to your blog and select the bookmark.

Keywords: blogger ignore visits, blogger exclude, blogger hide

Perl: Using LWP with a SOCKS Proxy

2009-07-18T18:54:00.000-07:00

For this example, we'll use LWP to connect through a SOCKS proxy to the Tor anonymizing network. Under Ubuntu/Debian, installation of Tor is as simple as:

sudo apt-get install tor

Next, install the LWP::Protocol::socks Perl module to add support for the "socks" scheme. Installing from CPAN:

sudo cpan LWP::Protocol::socks

You should now be able to use LWP with Tor:


#!/usr/bin/perl
use strict;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new(
  agent => q{Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; YPC 3.2.0; .NET 
CLR 1.1.4322)},
);
$ua->proxy([qw/ http https /] => 'socks://localhost:9050'); # Tor proxy
$ua->cookie_jar({});
my $rsp = $ua->get('http://www.yahoo.com/');
print $rsp->content;

Web: Resizing the Browser Window

2009-07-17T20:01:00.000-07:00

Use this javascript snippet to test web pages at different browser sizes:

javascript:resizeTo(1024, 768)

Enter that into your browser's location bar and it should resize your window to 1024x768 (the most common resolution, currently). The first value is the width in pixels. The second is the height.

Here are resize links to the five most popular resolutions. Bookmark these or drag them to your toolbar for quick access.

Resize 1024x768

Resize 1280x800

Resize 1280x1024

Resize 1440x900

Resize 800x600

There are also browser plugins that can adjust the window size, such as Firefox's Web Developer Toolbar.

Bash: Iterating Over Command-Line Arguments

2009-07-17T00:16:00.000-07:00

Shell script parameters are available through the positional variables $1, $2, $3, etc. They are also placed in the "special" variables $* and $@. The following script illustrates how those values are expanded.


#!/bin/bash

print_arguments () {
  idx=1
  for arg; do
    echo "  param $idx: $arg"
    let idx++
  done
  echo
}

echo "\$* - arguments with spaces are split"
print_arguments $*

echo "\$@ - same"
print_arguments $@

echo "\"\$*\" - expands to single value"
print_arguments "$*"

echo "\"\$@\" - preserves arguments exactly as passed"
print_arguments "$@"

Example output:


$ ./test-param-expansion orange 9 "Bringing Up Baby"
$* - arguments with spaces are split
  param 1: orange
  param 2: 9
  param 3: Bringing
  param 4: Up
  param 5: Baby

$@ - same
  param 1: orange
  param 2: 9
  param 3: Bringing
  param 4: Up
  param 5: Baby

"$*" - expands to single value
  param 1: orange 9 Bringing Up Baby

"$@" - preserves arguments exactly as passed
  param 1: orange
  param 2: 9
  param 3: Bringing Up Baby

So to preserve the command-line arguments exactly as supplied, use the special variable "$@" when iterating. Mnemonic: quote at the right place.

Example for loops:


#!/bin/sh

# loop over all args
for arg in "$@"; do
  echo $arg
done

# short-cut; same as above
for arg; do
  echo $arg
done

Linux: Encrypted Filesystem on a Regular Disk File

2009-07-15T02:05:00.000-07:00

Some notes on creating a LUKS-based encrypted filesystem on a regular disk file with Ubuntu 9.04 Jaunty.

Create the Encrypted Filesystem:


# create a 10M file
$ dd if=/dev/urandom of=testfs bs=1M count=10

# associate it with the loop device
$ losetup /dev/loop0 testfs

# encrypt it (will ask for password to use)
$ cryptsetup luksFormat /dev/loop0

# open the encrypted loop device
$ cryptsetup luksOpen /dev/loop0 testfs

# format it with ext2 (or whatever you prefer)
$ mkfs.ext2 /dev/mapper/testfs

# mount it
$ mount /dev/mapper/testfs /mnt/test

# confirm mount
$ df -h /mnt/test
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/testfs    9.2M   88K  8.7M   1% /mnt/test

Unmount the Filesystem:


# unmount it
$ umount /mnt/test

# close encryption
$ cryptsetup luksClose /dev/mapper/testfs

# release loop device
$ losetup -d /dev/loop0

Mount an Encrypted Filesystem:


# associate file with the loop device
$ losetup /dev/loop0 testfs

# open the encrypted loop device
$ cryptsetup luksOpen /dev/loop0 testfs

# mount it
$ mount /dev/mapper/testfs /mnt/test

Shell: Aliases for Escaping/Unescaping Query Parameters

2009-07-15T01:34:00.000-07:00

Here are a couple more Unix aliases, this time for quickly escaping and unescaping URL query parameters. These are defined in my ~/.bash_aliases file.


alias urlencode="perl -MURI::Escape -ple '\$_=uri_escape(\$_); s/%20/+/g'"
alias urldecode="perl -MURI::Escape -ple '\$_=uri_unescape(\$_); s/\+/ /'"

To escape/encode all unsafe characters in a query parameter:


$ echo "blue socks?" | urlencode
blue+socks%3F

And to unescape/decode a parameter:


$ echo blue+socks%3F | urldecode
blue socks?

These aliases depend on Perl and the URI::Escape module which are standard on most modern Unixes.

Shell: Linux Password Generator

2009-07-13T15:24:00.000-07:00

A simple bash function to generate secure passwords with standard Unix programs:


function mkpw() { head /dev/urandom | uuencode -m - | sed -n 2p | cut -c1-${1:-8}; }

Add that to your ~/.bashrc file. The password length can be specified as an argument (defaults to 8).
Example usage:


$ mkpw
B5PcCWC3
$ mkpw 10
V8T7ZrTQuN

On Ubuntu, the uuencode program is in the sharutils package. Alternatively, you could replace uuencode with md5sum or sha1sum, which would restrict the passwords to hex digits (and which would therefore make them appropriate for WEP keys and the like):


function mkpw_md5() { head /dev/urandom | md5sum | cut -c1-${1:-8}; }
function mkpw_sha1() { head /dev/urandom | sha1sum | cut -c1-${1:-8}; }

Blogger Syntax Highlighting

2009-07-11T18:01:00.000-07:00

How to add syntax highlighting to Blogger with Alex Gorbatchev's open-source SyntaxHighlighter.

Log into your blogspot account, select the "Layout" tab, and then click "Edit HTML."

Click "Download Full Template" to make a backup of your current template.

Make a copy of your template, open it in an editor, and find the </head> closing tag. Before that tag, add the following:

<link href='http://alexgorbatchev.com/pub/sh/current/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href='http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css' rel='stylesheet' type='text/css'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js' type='text/javascript'/>
<!-- add brushes here -->
<script type='text/javascript'>
  SyntaxHighlighter.config.bloggerMode = true;
  SyntaxHighlighter.all();
</script>

After the comment that says "add brushes here," add the languages you plan to use. For example, I'm using brushes for Javascript, Bash, SQL, XML/HTML, and C++:

<!-- add brushes here -->
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJScript.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushBash.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushSql.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCpp.js' type='text/javascript'/>

See the full list of supported syntaxes.

Save your modified template, and upload it to Blogger (again, under Layout > Edit HTML).

You should now be able to add syntax highlighting to your blog. Wrap your code in a <pre> tag and specify the brush to use in the class attribute. For example, to highlight a block of SQL:

<pre class="brush:sql">
SELECT *
FROM users
WHERE user_id = 1212;
</pre>

Here's how that looks when rendered:
```
  SELECT *
    FROM users
   WHERE user_id = 1212;
```

HTML encoded

Syntax Highlighting with JS and CSS

2009-07-11T17:01:00.000-07:00

The code syntax highlighting in this blog is being done with SyntaxHighlighter by Alex Gorbatchev. See the instructions on configuring syntax highlighting with Blogger.

Shell: Aliases for Encoding/Decoding HTML Strings

2009-07-11T15:06:00.000-07:00

Here are a couple of simple Unix aliases for quickly encoding and decoding HTML strings. I have these defined in my ~/.bash_aliases file.


alias htmlencode="perl -MHTML::Entities -pe 'encode_entities(\$_)'"
alias htmldecode="perl -MHTML::Entities -pe 'decode_entities(\$_)'"

To encode/escape all unsafe characters in a string with their HTML entities:


$ echo "This is <b>bold</b>" | htmlencode
This is &lt;b&gt;bold&lt;/b&gt;

And to decode/unescape HTML entities:


$ echo "This is &lt;b&gt;bold&lt;/b&gt;" | htmldecode
This is bold

These aliases depend on Perl and the HTML::Entities module which are standard on most modern Unixes.

See also: Web-based HTML encoder.

C++ Whitespace Trimming Functions

2009-07-11T14:28:00.000-07:00

Some simple functions for removing leading or trailing whitespace from strings in C++. It uses isspace() and your current locale (by default) to determine what is considered whitespace (e.g., space, tab, newline, etc.) and modifies the strings in-place.

The header file, btrim.hpp:


#ifndef BTRIM_HPP
#define BTRIM_HPP

#include <string>
#include <locale>

/** 
 * Strip leading whitespace from a string.
 *
 * @param[in] str Reference to the string to be trimmed. Modified in-place.
 * @param[in] loc Locale to use (optional).
 * @return        void
 */
void ltrim(std::string& str, const std::locale& loc = std::locale());

/**
 * Strip trailing whitespace from a string.
 *
 * @param[in] str Reference to the string to be trimmed. Modified in-place.
 * @param[in] loc Locale to use (optional).
 * @return        void
 */
void rtrim(std::string& str, const std::locale& loc = std::locale());

/**
 * Strip both leading and trailing whitespace from a string.
 *
 * @param[in] str Reference to the string to be trimmed. Modified in-place.
 * @param[in] loc Locale to use (optional).
 * @return        void
 */
void btrim(std::string& str, const std::locale& loc = std::locale());

#endif // BTRIM_HPP

The implementation file, btrim.cpp:


#include "btrim.hpp"

using namespace std;

void ltrim(string& str, const locale& loc) {
  string::size_type pos = 0;
  while (pos < str.size() && isspace(str[pos], loc)) pos++;
  str.erase(0, pos);
}

void rtrim(string& str, const locale& loc) {
  string::size_type pos = str.size();
  while (pos > 0 && isspace(str[pos - 1], loc)) pos--;
  str.erase(pos);
}

void btrim(string& str, const locale& loc) {
  ltrim(str, loc);
  rtrim(str, loc);
}

Example usage:


// Example use of trim whitespace functions
#include <string>
#include <iostream>
#include "btrim.hpp"

using namespace std;

int main () {
  string s = "   zort   ";
  btrim(s); /* trim spaces, both leading and trailing */
  cout << "Trimmed string: |" << s << "|" << endl;
}

SSH Tunnel with FreeNX

2009-07-04T14:00:00.000-07:00

To use FreeNX to access a machine that's behind a firewall, you can tunnel an SSH connection:

ssh -L 2222:desktop:22 user@firewall

Configure FreeNX to connect to port 2222 on localhost, and log in as normal.

Access Linux From Windows with FreeNX

2009-06-28T22:25:00.000-07:00

FreeNX is a cross-platform remote desktop application. It's similar to VNC but is faster and leverages SSH for security. Installation on Ubuntu is simple (I'm using Jaunty).

Edit /etc/apt/sources.list and add the FreeNX repository:


# Ubuntu FreeNX Repository for Jaunty
# sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 2a8e3034d018a4ce
deb http://ppa.launchpad.net/freenx-team/ppa/ubuntu jaunty main
deb-src http://ppa.launchpad.net/freenx-team/ppa/ubuntu jaunty main

Update your packages list: sudo apt-get update
Install the FreeNX server: sudo apt-get install freenx-server
Install the appropriate FreeNX client software for Windows or Mac, and try logging into your Linux machine remotely.

Dell 2408WFP Calibration Settings

2008-12-28T14:55:00.000-08:00

Recommended settings for the Dell 2408WFP monitor:

Color Model: AdobeRGB (not sRGB)
Brightness: 7-8
Contrast: 50%
R = 96, G = 92, B = 100

Values taken from these calibration results. The author notes "this is by no means a universal recipe as models vary," but I don't have the equipment to run my own tests and these settings look excellent to me.

If you want to calibrate the monitor yourself, see calibrating a Dell 2408WFP with Spyder Elite 3.

Muhlenkamp (MUHLX) Conference Call

2007-11-14T02:16:00.000-08:00

Listen to the Muhlenkamp October 18, 2007 Advisers Conference Call (in mp3 format).

iconv: file too large

2007-08-20T13:29:00.000-07:00

The iconv utility is used to convert file encodings. I'm using it to convert a Postgresql database from LATIN1 to UTF8.

However, the standard iconv program slurps the entire file into memory, which doesn't work for large data sets (such as database exports). You'll see errors like:

iconv: unable to allocate buffer for input: Cannot allocate memory
 iconv: cannot open input file `database.txt': File too large

This script is just a wrapper that processes the input file in manageable chunks and writes it to standard output: iconv-chunks

The Q-Ratio

2007-04-29T20:50:00.000-07:00

The q-ratio is a method of estimating the fair value of the stock market, and is argued for in the book Valuing Wall Street, by Andrew Smithers and Stephen Wright.

They indicate that the data need for Q from 1952 onward is in the Z1 Flow of Funds report from the Federal Reserve. I'm trying to independently recreate the graphs from that source.

Favorite Ubuntu Linux Tips

2007-04-26T21:10:00.000-07:00

Inline/embed PDFs in Firefox

To view PDFs, Firefox normally runs Acroread as an external helper application. Use this to display them directly in the browser window instead.

Lifehacker - Top Ten Ubuntu Apps and Tweaks

Moving Back to Legg Mason

2007-04-22T00:11:00.000-07:00

I've been a longtime shareholder of several Legg Mason funds. In 2005, Legg Mason sold their brokerage operation to Smith Barney (Citigroup) and my account was transferred.

I've been very disappointed with Smith Barney. Their account fees are much higher ($50/year for taxable accounts, $40/year for IRAs). You can access your account through their website but in order to actually do anything (e.g., sell or buy funds) you need an "FMA Account," which costs extra. They're even taking a cut of reinvested dividends and capital gains now. In short, Smith Barney nickels and dimes you for everything and provides no value.

But...I thought I didn't have any options. Unlike most funds, you can't buy Legg Mason funds through most brokers, so I couldn't just transfer the accounts to a different firm. I could liquidate, but I really didn't want to do that.

Turns out there is another option though: you can transfer your account right back to Legg Mason! Maybe I missed it, but they didn't seem to advertise that option very prominently.

In order to do the transfer you need to complete a new account application (either the "Retail" or "IRA" form as appropriate), and then the "Direct Transfer" form:

Legg Mason Transfer Forms »

Legg Mason has no fees except for the typical IRA custodial fee ($10/year), and it looks like you can actually buy and sell through the Legg Mason website.

Table Formatting in Blogger

2007-01-13T16:01:00.001-08:00

I'm pretty new to Blogger but one of the first snags I hit was with formatting tables due to the way Blogger automatically inserts line breaks. You can disable that behavior under Settings but it's a global option so it affects all posts. Another option is to write your tables without line breaks, but that's a maintenance problem.

Here's another approach that works (tested with Firefox 1.5 and IE 6; anyone have other browser results?). At the top of your post add this:

<style type="text/css">.nobr br { display: none }</style>

Then place your table within that class:

<div class="nobr">
<table border="1">
  <tr>
    <td>GOOG</td>
    <td>505.00</td>
    <td>2006-01-12</td>
  </tr>
  <tr>
    <td>BRK-A</td>
    <td>110,000.00</td>
    <td>2006-01-12</td>
  </tr>
</table>
</div>

That has the effect of turning off the display of the Blogger-generated line breaks within the div, causing the table to be rendered properly.

Here's how that table looks in your browser:

GOOG	505.00	2006-01-12
BRK-A	110,000.00	2006-01-12

I think it should be possible to move the CSS declaration into the template so you don't have to repeat it in each post.

Note from reader S.B.:

Unfortunately, both the "Compose" and "Preview" views of a post still render incorrectly, but when you actually publish, the table does come out right!

Postgresql Performance Tips for Data Loads

2007-01-07T01:19:00.000-08:00

Turn off fsync in postgresql.conf. This can make a huge difference.

Temporarily disable triggers.

Postgresql 8.x supports an ALTER statement:

ALTER TABLE foo DISABLE TRIGGER ALL;
ALTER TABLE foo ENABLE TRIGGER ALL;

With Postgresql 7.x you need to modify the system tables directly:

UPDATE pg_class
   SET reltriggers = 0
 WHERE relname = 'foo';

To re-enable:

UPDATE pg_class
   SET reltriggers = (
         SELECT count(*)
           FROM pg_trigger
          WHERE pg_class.oid = tgrelid
       )
 WHERE relname = 'foo';

Postgresql docs on efficient data loads

Creating a Postgresql Development Database

2007-01-06T14:48:00.000-08:00

Notes on how to restore a Pg database for a dev environment to make it simple to create additional copies or revert back to the production schema.

Drop and restore your db from a recent backup. Include a datestamp in the database name to keep track of the version you're restoring from. The specifics of this commands will vary depending on your permissions, backup method, etc.


$ dropdb mydevdb_20070106
$ createdb -U postgres -E LATIN1 mydevdb_20070106
$ pg_restore -U postgres -d mydevdb_20070106 ~/db-backup.2007-01-06

If you're restoring from production data, be sure to munge it at this point to remove all valuable/sensitive information, and delete the backup file so your dev environment isn't a target.

Now with this restored version of the database you can create any number of additional copies to actually work on. Leave the initial restore untouched so you can always revert back to it.


$ psql mydevdb_20070106
mydevdb_20070106=# CREATE DATABASE mydevdb WITH TEMPLATE mydevdb_20070106;
CREATE DATABASE
mydevdb_20070106=# CREATE DATABASE toms_devdb WITH TEMPLATE mydevdb_20070106;
CREATE DATABASE