Using iconv to change character encodings

Introduction

iconv is used for character set conversion facility. With this command, you can turn a string represented by a local character set into the one represented by another character set, which may be the Unicode character set. Supported character sets depend on the iconv implementation of your system. Note that the iconv function on some systems may not work as you expect. In such case, it'd be a good idea to install the » GNU libiconv library. It will most likely end up with more consistent results.

Detail

The basic command is:

iconv -f old-encoding -t new-encoding file.txt > newfile.txt

You can get a list of supported encodings with (that's a lower-case L, not a one):

iconv -l

Example

iconv -f ISO-8859-1 -t UTF8 sample.txt > sample.txt

It is converting from ISO_8859-1 to UTF8.

Resources

• The official iconv man page.
• For Linux users who don't have it in their distribution, the iconv source is available from GNU.

Handle UTF8 file with BOM

From Wikipedia, the byte order mark (BOM) is a Unicode character used to signal the endianness (byte order) of a text file or stream. Its code point is U+FEFF. BOM use is optional, and, if used, should appear at the start of the text stream. Beyond its specific use as a byte-order indicator, the BOM character may also indicate which of the several Unicode representations the text is encoded in.

The common BOMs are :

Encoding	Representation (hexadecimal)	Representation (decimal)
UTF-8	EF BB BF	239 187 191
UTF-16 (BE)	FE FF	254 255
UTF-16 (LE)	FF FE	255 254
UTF-32 (BE)	00 00 FE FF	0 0 254 255
UTF-32 (LE)	FF FE 00 00	255 254 0 0

UTF8 file are a special case because it is not recommended to add a BOM to them because it can break other tools like Java. In fact, Java assumes the UTF8 don't have a BOM so if the BOM is present it won't be discarded and it will be seen as data.

To create an UTF8 file with a BOM, open the Windows create a simple text file and save it as utf8.txt with the encoding UTF-8.

Now if you examine the file content as binary, you see the BOM at the beginning.

If we read it with Java.

package com.java;

import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.InputStreamReader;

public class BomExample {

/**
* @author Imroze.Mohammad
*/
public static void main(String args[]) {
try {
FileInputStream fis = new FileInputStream("E:\\demo\\UTF8withBOM.txt");
BufferedReader r = new BufferedReader(new InputStreamReader(fis,
"UTF8"));
for (String s = ""; (s = r.readLine()) != null;) {
System.out.println(s);
}
r.close();
System.exit(0);
}

catch (Exception e) {
e.printStackTrace();
System.exit(1);
}
}
}

The output contains a strange character at the beginning because the BOM is not discarded :

?helloworld

The next example converts an UTF8 file to ANSI. We check the first line for the presence of the BOM and if present, we simply discard it.

package com.java;

import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.InputStreamReader;

public class BomExample {

/**
* @author Imroze.Mohammad
*/
 public static void main(String args[]) {
   try {
       FileInputStream fis = new FileInputStream("E:\\jeeworkspace\\talend_web_app\\demo\\UTF8withBOM.txt");
       BufferedReader r = new BufferedReader(new InputStreamReader(fis,
               "UTF8"));
       boolean firstLine=true;
       for (String s = ""; (s = r.readLine()) != null;) {
        if(firstLine){
        s=removeBOMChar(s);
        }
           System.out.println(s);
       }
       r.close();
       System.exit(0);
   }

   catch (Exception e) {
       e.printStackTrace();
       System.exit(1);
   }
 }
 
 
 private static String removeBOMChar(String s){
 
 if(s.startsWith("\uFEFF"))
 s=s.substring(1);
return s;
 
 }
 
}