Fixing broken text encodings with sqlite-transform and ftfy

I was working with a database table that included values that were clearly in the wrong character encoding - values like this:

Rue Léopold I

I used my sqlite-transform tool with the ftfy Python library to fix that by running the following:

sqlite-transform lambda chiens.db espaces-pour-chiens-et-espaces-interdits-aux-chiens namefr \
  --code 'ftfy.fix_encoding(value)' \
  --import ftfy

That's the database file, the table and the column, then --code and --import to specify the transformation.

Since I had installed sqlite-transform using pipx install sqlite-transform I needed to first install the ftfy library into the correct virtual environment. The recipe for doing that is:

pipx inject sqlite-transform ftfy

Related

• linux Using iconv to convert the text encoding of a file - 2022-06-14
• sqlite Compiling the SQLite spellfix.c module on macOS - 2020-09-19
• python Running PyPy on macOS using Homebrew - 2022-09-14
• sqlite Using LD_PRELOAD to run any version of SQLite with Python - 2020-06-17
• bash Escaping a SQL query to use with curl and Datasette - 2020-12-08
• zsh Passing command arguments using heredoc syntax - 2022-07-07
• python Explicit file encodings using click.File - 2020-10-16
• sqlite Loading SQLite extensions in Python on macOS - 2023-01-07
• sqlite Importing CSV data into SQLite with .import - 2021-07-13
• bash Using awk to add a prefix - 2022-04-08

Created 2021-01-18T12:44:30-08:00, updated 2021-01-18T17:27:54-08:00 · History · Edit