jfw.id.au

I’m migrating a database from a postgres 7.4 server to a 8.1 server and hit a small issue with the import of the dumpfile. I’m in the middle of testing the migration at the moment and the complaint coming back from the import was about invalid UTF-8 characters in the input file. A quick google and the first link was to a page on the PostgreSQL site with the following snippet:

Some users are having problems loading UTF-8 data into 8.1.X. This is because previous versions allowed invalid UTF-8 byte sequences to be entered into the database, and this release properly accepts only valid UTF-8 sequences. One way to correct a dumpfile is to run the command iconv -c -f UTF-8 -t UTF-8 -o cleanfile.sql dumpfile.sql. The -c option removes invalid character sequences. A diff of the two files will show the sequences that are invalid. iconv reads the entire input file into memory so it might be necessary to use split to break up the dump into multiple smaller files for processing.

It works :) I had no doubt, but it is good to know that it has solved this problem.

Now my only problem is to work out why there are invalid characters in the database in the first place.  But that can wait for another day.  I’m off to bed!