Monday, August 20, 2007

iconv: file too large

The iconv utility is used to convert file encodings. I'm using it to convert a Postgresql database from LATIN1 to UTF8.

However, the standard iconv program slurps the entire file into memory, which doesn't work for large data sets (such as database exports). You'll see errors like:

iconv: unable to allocate buffer for input: Cannot allocate memory
iconv: cannot open input file `database.txt': File too large


This script is just a wrapper that processes the input file in manageable chunks and writes it to standard output: iconv-chunks

Sunday, April 29, 2007

The Q-Ratio


The q-ratio is a method of estimating the fair value of the stock market, and is argued for in the book Valuing Wall Street, by Andrew Smithers and Stephen Wright.

They indicate that the data need for Q from 1952 onward is in the Z1 Flow of Funds report from the Federal Reserve. I'm trying to independently recreate the graphs from that source.

Thursday, April 26, 2007

Favorite Ubuntu Linux Tips

Sunday, April 22, 2007

Moving Back to Legg Mason

I've been a longtime shareholder of several Legg Mason funds. In 2005, Legg Mason sold their brokerage operation to Smith Barney (Citigroup) and my account was transferred.

I've been very disappointed with Smith Barney. Their account fees are much higher ($50/year for taxable accounts, $40/year for IRAs). You can access your account through their website but in order to actually do anything (e.g., sell or buy funds) you need an "FMA Account," which costs extra. They're even taking a cut of reinvested dividends and capital gains now. In short, Smith Barney nickels and dimes you for everything and provides no value.

But...I thought I didn't have any options. Unlike most funds, you can't buy Legg Mason funds through most brokers, so I couldn't just transfer the accounts to a different firm. I could liquidate, but I really didn't want to do that.

Turns out there is another option though: you can transfer your account right back to Legg Mason! Maybe I missed it, but they didn't seem to advertise that option very prominently.

In order to do the transfer you need to complete a new account application (either the "Retail" or "IRA" form as appropriate), and then the "Direct Transfer" form:

Legg Mason Transfer Forms »

Legg Mason has no fees except for the typical IRA custodial fee ($10/year), and it looks like you can actually buy and sell through the Legg Mason website.

Saturday, January 13, 2007

Table Formatting in Blogger

I'm pretty new to Blogger but one of the first snags I hit was with formatting tables due to the way Blogger automatically inserts line breaks. You can disable that behavior under Settings but it's a global option so it affects all posts. Another option is to write your tables without line breaks, but that's a maintenance problem.

Here's another approach that works (tested with Firefox 1.5 and IE 6; anyone have other browser results?). At the top of your post add this:

<style type="text/css">.nobr br { display: none }</style>

Then place your table within that class:
<div class="nobr">
<table border="1">
<tr>
<td>GOOG</td>
<td>505.00</td>
<td>2006-01-12</td>
</tr>
<tr>
<td>BRK-A</td>
<td>110,000.00</td>
<td>2006-01-12</td>
</tr>
</table>
</div>
That has the effect of turning off the display of the Blogger-generated line breaks within the div, causing the table to be rendered properly.

Here's how that table looks in your browser:













GOOG 505.00 2006-01-12
BRK-A 110,000.00 2006-01-12


I think it should be possible to move the CSS declaration into the template so you don't have to repeat it in each post.

Note from reader S.B.:
Unfortunately, both the "Compose" and "Preview" views of a post still render incorrectly, but when you actually publish, the table does come out right!

Sunday, January 7, 2007

Postgresql Performance Tips for Data Loads


  • Turn off fsync in postgresql.conf. This can make a huge difference.

  • Temporarily disable triggers.

    Postgresql 8.x supports an ALTER statement:

    ALTER TABLE foo DISABLE TRIGGER ALL;
    ALTER TABLE foo ENABLE TRIGGER ALL;

    With Postgresql 7.x you need to modify the system tables directly:
    UPDATE pg_class
    SET reltriggers = 0
    WHERE relname = 'foo';
    To re-enable:
    UPDATE pg_class
    SET reltriggers = (
    SELECT count(*)
    FROM pg_trigger
    WHERE pg_class.oid = tgrelid
    )
    WHERE relname = 'foo';
  • Postgresql docs on efficient data loads

Saturday, January 6, 2007

Creating a Postgresql Development Database

Notes on how to restore a Pg database for a dev environment to make it simple to create additional copies or revert back to the production schema.

Drop and restore your db from a recent backup. Include a datestamp in the database name to keep track of the version you're restoring from. The specifics of this commands will vary depending on your permissions, backup method, etc.

$ dropdb mydevdb_20070106
$ createdb -U postgres -E LATIN1 mydevdb_20070106
$ pg_restore -U postgres -d mydevdb_20070106 ~/db-backup.2007-01-06

If you're restoring from production data, be sure to munge it at this point to remove all valuable/sensitive information, and delete the backup file so your dev environment isn't a target.

Now with this restored version of the database you can create any number of additional copies to actually work on. Leave the initial restore untouched so you can always revert back to it.

$ psql mydevdb_20070106
mydevdb_20070106=# CREATE DATABASE mydevdb WITH TEMPLATE mydevdb_20070106;
CREATE DATABASE
mydevdb_20070106=# CREATE DATABASE toms_devdb WITH TEMPLATE mydevdb_20070106;
CREATE DATABASE