Internally, Perl currently uses either whatever the native eight-bit character set of the platform (for example Latin-1) is, defaulting to UTF-8, to encode Unicode strings. Specifically, if all code points in the string are 0xFF or less, Perl uses the native eight-bit character set. Otherwise, it uses UTF-8.

A user of Perl does not normally need to know nor care how Perl happens to encode its internal strings, but it becomes relevant when outputting Unicode strings to a stream without a PerlIO layer -- one with the ``default'' encoding. In such a case, the raw bytes used internally (the native character set or UTF-8, as appropriate for each string) will be used, and a ``Wide character'' warning will be issued if those strings contain a character beyond 0x00FF.

For example,

produces a fairly useless mixture of native bytes and UTF-8, as well as a warning:

To output UTF-8, use the :utf8 output layer. Prepending

to this sample program ensures that the output is completely UTF-8, and removes the program's warning.

If your locale environment variables (LANGUAGE, LC_ALL, LC_CTYPE, LANG) contain the strings 'UTF-8' or 'UTF8', regardless of case, then the default encoding of your STDIN, STDOUT, and STDERR and of any subsequent file open, is UTF-8. Note that this means that Perl expects other software to work, too: if Perl has been led to believe that STDIN should be UTF-8, but then STDIN coming in from another command is not UTF-8, Perl will complain about the malformed UTF-8.

All features that combine Unicode and I/O also require using the new PerlIO feature. Almost all Perl 5.8 platforms do use PerlIO, though: you can see whether yours is by running ``perl -V'' and looking for useperlio=define.

Unicode and EBCDIC

In any case, the Unicode support on EBCDIC platforms is better than in the 5.6 series, which didn't work much at all for EBCDIC platform. On EBCDIC platforms, the internal Unicode encoding form is UTF-EBCDIC instead of UTF-8. The difference is that as UTF-8 is ``ASCII-safe'' in that ASCII characters encode to UTF-8 as-is, while UTF-EBCDIC is ``EBCDIC-safe''.

Creating Unicode

Similarly, it can be used in regular expression literals

At run-time you can use chr() :

See ``Further Resources'' for how to find all these numeric codes.

Naturally, ord() will do the reverse: it turns a character into a code point.

Note that \x.. (no {} and only two hexadecimal digits), \x{...}, and chr(...) for arguments less than 0x100 (decimal 256) generate an eight-bit character for backward compatibility with older Perls. For arguments of 0x100 or more, Unicode characters are always produced. If you want to force the production of Unicode characters regardless of the numeric value, use pack(``U'', ...) instead of \x.., \x{...}, or chr() .

You can also use the charnames pragma to invoke characters by name in double-quoted strings:

And, as mentioned above, you can also pack() numbers into Unicode characters:

Note that both \x{...} and \N{...} are compile-time string constants: you cannot use variables in them. if you want similar run-time functionality, use chr() and charnames::vianame().

Also note that if all the code points for pack ``U'' are below 0x100, bytes will be generated, just like if you were using chr() .

If you want to force the result to Unicode characters, use the special ``U0'' prefix. It consumes no arguments but forces the result to be in Unicode characters, instead of bytes.

Handling Unicode

Note that Perl considers combining character sequences to be characters, so for example

will print 2, not 1. The only exception is that regular expressions have \X for matching a combining character sequence.

Life is not quite so transparent, however, when working with legacy encodings, I/O, and certain special cases:

Legacy Encodings

in which case literals (string or regular expressions), chr() , and ord() in your whole script are assumed to produce Unicode characters from ISO 8859-2 code points. Note that the matching for encoding names is forgiving: instead of latin2 you could have said Latin 2, or iso8859-2, or other variations. With just

the environment variable PERL_ENCODING will be consulted. If that variable isn't set, the encoding pragma will fail.

The Encode module knows about many encodings and has interfaces for doing conversions between those encodings:

Unicode I/O

produces raw bytes that Perl happens to use to internally encode the Unicode string. Perl's internal encoding depends on the system as well as what characters happen to be in the string at the time. If any of the characters are at code points 0x100 or above, you will get a warning. To ensure that the output is explicitly rendered in the encoding you desire--and to avoid the warning--open the stream with the desired encoding. Some examples:

and on already open streams, use binmode() :

The matching of encoding names is loose: case does not matter, and many encodings have several aliases. Note that the :utf8 layer must always be specified exactly like that; it is not subject to the loose matching of encoding names.

See PerlIO for the :utf8 layer, PerlIO::encoding and Encode::PerlIO for the :encoding() layer, and Encode::Supported for many encodings supported by the Encode module.

Reading in a file that you know happens to be encoded in one of the Unicode or legacy encodings does not magically turn the data into Unicode in Perl's eyes. To do that, specify the appropriate layer when opening files

The I/O layers can also be specified more flexibly with the open pragma. See the open manpage , or look at the following example.

With the open pragma you can use the :locale layer

or you can also use the ':encoding(...)' layer

These methods install a transparent filter on the I/O stream that converts data from the specified encoding when it is read in from the stream. The result is always Unicode.

The the open manpage pragma affects all the open() calls after the pragma by setting default layers. If you want to affect only certain streams, use explicit layers directly in the open() call.

You can switch encodings on an already opened stream by using binmode() ; see binmode .

The :locale does not currently (as of Perl 5.8.0) work with open() and binmode() , only with the open pragma. The :utf8 and :encoding(...) methods do work with all of open() , binmode() , and the open pragma.

Similarly, you may use these I/O layers on output streams to automatically convert Unicode to the specified encoding when it is written to the stream. For example, the following snippet copies the contents of the file ``text.jis'' (encoded as ISO-2022-JP, aka JIS) to the file ``text.utf8'', encoded as UTF-8:

The naming of encodings, both by the open() and by the open pragma, is similar to the encoding pragma in that it allows for flexible names: koi8-r and KOI8R will both be understood.

Common encodings recognized by ISO, MIME, IANA, and various other standardisation organisations are recognised; for a more detailed list see Encode::Supported.

read() reads characters and returns the number of characters. seek() and tell() operate on byte counts, as do sysread() and sysseek() .

Notice that because of the default behaviour of not doing any conversion upon input if there is no default layer, it is easy to mistakenly write code that keeps on expanding a file by repeatedly encoding the data:

If you run this code twice, the contents of the file will be twice UTF-8 encoded. A use open ':utf8' would have avoided the bug, or explicitly opening also the file for input as UTF-8.

NOTE: the :utf8 and :encoding features work only if your Perl has been built with the new PerlIO feature.

Displaying Unicode As Text

For example,

returns:

Special Cases

• Bit Complement Operator ~ And vec()

  The bit complement operator ~ may produce surprising results if used on strings containing characters with ordinal values above 255. In such a case, the results are consistent with the internal encoding of the characters, but not with much else. So don't do that. Similarly for vec() : you will be operating on the internally-encoded bit patterns of the Unicode characters, not on the code point values, which is very probably not what you want.

• Peeking At Perl's Internal Encoding

  Normal users of Perl should never care how Perl encodes any particular Unicode string (because the normal ways to get at the contents of a string with Unicode--via input and output--should always be via explicitly-defined I/O layers). But if you must, there are two ways of looking behind the scenes.

  One way of peeking inside the internal encoding of Unicode characters is to use unpack(``C*'', ... to get the bytes or unpack(``H*'', ...) to display the bytes:

  # this prints c4 80 for the UTF-8 bytes 0xc4 0x80 print join(" ", unpack("H*", pack("U", 0x100))), "\n";

  Yet another way would be to use the Devel::Peek module:

  perl -MDevel::Peek -e 'Dump(chr(0x100))'

  That shows the UTF8 flag in FLAGS and both the UTF-8 bytes and Unicode characters in PV. See also later in this document the discussion about the is_utf8 function of the Encode module.

Advanced Topics

• String Equivalence

  The question of string equivalence turns somewhat complicated in Unicode: what do you mean by ``equal''?

  (Is LATIN CAPITAL LETTER A WITH ACUTE equal to LATIN CAPITAL LETTER A?)

  The short answer is that by default Perl compares equivalence (eq, ne) based only on code points of the characters. In the above case, the answer is no (because 0x00C1 != 0x0041). But sometimes, any CAPITAL LETTER As should be considered equal, or even As of any case.

  The long answer is that you need to consider character normalization and casing issues: see Unicode::Normalize, Unicode Technical Reports #15 and #21, Unicode Normalization Forms and Ihttp://www.unicode.org/unicode/reports/tr15/ and http://www.unicode.org/unicode/reports/tr21/

  As of Perl 5.8.0, the ``Full'' case-folding of I

• String Collation

  People like to see their strings nicely sorted--or as Unicode parlance goes, collated. But again, what do you mean by collate?

  (Does LATIN CAPITAL LETTER A WITH ACUTE come before or after LATIN CAPITAL LETTER A WITH GRAVE?)

  The short answer is that by default, Perl compares strings (lt, le, cmp, ge, gt) based only on the code points of the characters. In the above case, the answer is ``after'', since 0x00C1 > 0x00C0.

  The long answer is that ``it depends'', and a good answer cannot be given without knowing (at the very least) the language context. See Unicode::Collate, and Unicode Collation Algorithm http://www.unicode.org/unicode/reports/tr10/

Miscellaneous

• Character Ranges and Classes

  Character ranges in regular expression character classes (/[a-z]/) and in the tr/// (also known as y/// ) operator are not magically Unicode-aware. What this means that [A-Za-z] will not magically start to mean ``all alphabetic letters''; not that it does mean that even for 8-bit characters, you should be using /[[:alpha:]]/ in that case.

  For specifying character classes like that in regular expressions, you can use the various Unicode properties--\pL, or perhaps \p{Alphabetic}, in this particular case. You can use Unicode code points as the end points of character ranges, but there is no magic associated with specifying a certain range. For further information--there are dozens of Unicode character classes--see the perlunicode manpage .

• String-To-Number Conversions

  Unicode does define several other decimal--and numeric--characters besides the familiar 0 to 9, such as the Arabic and Indic digits. Perl does not support string-to-number conversion for digits other than ASCII 0 to 9 (and ASCII a to f for hexadecimal).

Questions With Answers

• Will My Old Scripts Break?

  Very probably not. Unless you are generating Unicode characters somehow, old behaviour should be preserved. About the only behaviour that has changed and which could start generating Unicode is the old behaviour of chr() where supplying an argument more than 255 produced a character modulo 255. chr(300) , for example, was equal to chr(45) or ``-'' (in ASCII), now it is LATIN CAPITAL LETTER I WITH BREVE.

• How Do I Make My Scripts Work With Unicode?

  Very little work should be needed since nothing changes until you generate Unicode data. The most important thing is getting input as Unicode; for that, see the earlier I/O discussion.

• How Do I Know Whether My String Is In Unicode?

  You shouldn't care. No, you really shouldn't. No, really. If you have to care--beyond the cases described above--it means that we didn't get the transparency of Unicode quite right.

  Okay, if you insist:

  use Encode 'is_utf8'; print is_utf8($string) ? 1 : 0, "\n";

  But note that this doesn't mean that any of the characters in the string are necessary UTF-8 encoded, or that any of the characters have code points greater than 0xFF (255) or even 0x80 (128), or that the string has any characters at all. All the is_utf8() does is to return the value of the internal ``utf8ness'' flag attached to the $string. If the flag is off, the bytes in the scalar are interpreted as a single byte encoding. If the flag is on, the bytes in the scalar are interpreted as the (multi-byte, variable-length) UTF-8 encoded code points of the characters. Bytes added to an UTF-8 encoded string are automatically upgraded to UTF-8. If mixed non-UTF8 and UTF-8 scalars are merged (double-quoted interpolation, explicit concatenation, and printf/sprintf parameter substitution), the result will be UTF-8 encoded as if copies of the byte strings were upgraded to UTF-8: for example,

  $a = "ab\x80c"; $b = "\x{100}"; print "$a = $b\n";

  the output string will be UTF-8-encoded ab\x80c\x{100}\n, but note that $a will stay byte-encoded.

  Sometimes you might really need to know the byte length of a string instead of the character length. For that use either the Encode::encode_utf8() function or the bytes pragma and its only defined function length() :

  my $unicode = chr(0x100); print length($unicode), "\n"; # will print 1 require Encode; print length(Encode::encode_utf8($unicode)), "\n"; # will print 2 use bytes; print length($unicode), "\n"; # will also print 2 # (the 0xC4 0x80 of the UTF-8)

• How Do I Detect Data That's Not Valid In a Particular Encoding?

  Use the Encode package to try converting it. For example,

  use Encode 'encode_utf8'; if (encode_utf8($string_of_bytes_that_I_think_is_utf8)) { # valid } else { # invalid }

  For UTF-8 only, you can use:

  use warnings; @chars = unpack("U0U*", $string_of_bytes_that_I_think_is_utf8);

  If invalid, a Malformed UTF-8 character (byte 0x##) in unpack warning is produced. The ``U0'' means "expect strictly UTF-8 encoded Unicode``. Without that the unpack(''U*", ...) would accept also data like chr(0xFF ), similarly to the pack as we saw earlier.

• How Do I Convert Binary Data Into a Particular Encoding, Or Vice Versa?

  This probably isn't as useful as you might think. Normally, you shouldn't need to.

  In one sense, what you are asking doesn't make much sense: encodings are for characters, and binary data are not ``characters'', so converting ``data'' into some encoding isn't meaningful unless you know in what character set and encoding the binary data is in, in which case it's not just binary data, now is it?

  If you have a raw sequence of bytes that you know should be interpreted via a particular encoding, you can use Encode:

  use Encode 'from_to'; from_to($data, "iso-8859-1", "utf-8"); # from latin-1 to utf-8

  The call to from_to() changes the bytes in $data, but nothing material about the nature of the string has changed as far as Perl is concerned. Both before and after the call, the string $data contains just a bunch of 8-bit bytes. As far as Perl is concerned, the encoding of the string remains as ``system-native 8-bit bytes''.

  You might relate this to a fictional 'Translate' module:

  use Translate; my $phrase = "Yes"; Translate::from_to($phrase, 'english', 'deutsch'); ## phrase now contains "Ja"

  The contents of the string changes, but not the nature of the string. Perl doesn't know any more after the call than before that the contents of the string indicates the affirmative.

  Back to converting data. If you have (or want) data in your system's native 8-bit encoding (e.g. Latin-1, EBCDIC, etc.), you can use pack/unpack to convert to/from Unicode.

  $native_string = pack("C*", unpack("U*", $Unicode_string)); $Unicode_string = pack("U*", unpack("C*", $native_string));

  If you have a sequence of bytes you know is valid UTF-8, but Perl doesn't know it yet, you can make Perl a believer, too:

  use Encode 'decode_utf8'; $Unicode = decode_utf8($bytes);

  You can convert well-formed UTF-8 to a sequence of bytes, but if you just want to convert random binary data into UTF-8, you can't. Any random collection of bytes isn't well-formed UTF-8. You can use unpack(``C*'', $string) for the former, and you can create well-formed Unicode data by pack(``U*'', 0xff, ...) .

• How Do I Display Unicode? How Do I Input Unicode?

  See http://www.alanwood.net/unicode/ and http://www.cl.cam.ac.uk/~mgk25/unicode.html

• How Does Unicode Work With Traditional Locales?

  In Perl, not very well. Avoid using locales through the locale pragma. Use only one or the other.

Hexadecimal Notation

The 0x prefix means a hexadecimal number, the digits are 0-9 and a-f (or A-F, case doesn't matter). Each hexadecimal digit represents four bits, or half a byte. print 0x..., ``\n'' will show a hexadecimal number in decimal, and printf ``%x\n'', $decimal will show a decimal number in hexadecimal. If you have just the ``hex digits'' of a hexadecimal number, you can use the hex() function.

Further Resources

• Unicode Consortium
  http://www.unicode.org/

• Unicode FAQ
  http://www.unicode.org/unicode/faq/

• Unicode Glossary
  http://www.unicode.org/glossary/

• Unicode Useful Resources
  http://www.unicode.org/unicode/onlinedat/resources.html

• Unicode and Multilingual Support in HTML, Fonts, Web Browsers and Other Applications
  http://www.alanwood.net/unicode/

• UTF-8 and Unicode FAQ for Unix/Linux
  http://www.cl.cam.ac.uk/~mgk25/unicode.html

• Legacy Character Sets
  http://www.czyborra.com/ http://www.eki.ee/letter/

• The Unicode support files live within the Perl installation in the directory
  $Config{installprivlib}/unicore

  in Perl 5.8.0 or newer, and

  $Config{installprivlib}/unicode

  in the Perl 5.6 series. (The renaming to lib/unicore was done to avoid naming conflicts with lib/Unicode in case-insensitive filesystems.) The main Unicode data file is UnicodeData.txt (or Unicode.301 in Perl 5.6.1.) You can find the $Config{installprivlib} by

  perl "-V:installprivlib"

  You can explore various information from the Unicode data files using the Unicode::UCD module.

UNICODE IN OLDER PERLS

The following are fast conversions from ISO 8859-1 (Latin-1) bytes to UTF-8 bytes and back, the code works even with older Perl 5 versions.

    # ISO 8859-1 to UTF-8
    s/([\x80-\xFF])/chr(0xC0|ord(
$1
)>>6).chr(0x80|ord(
$1
)&0x3F)/eg;
    # UTF-8 to ISO 8859-1
    s/([\xC2\xC3])([\x80-\xBF])/chr(ord(
$1
)<<6&0xC0|ord(
$2
)&0x3F)/eg;

SEE ALSO

ACKNOWLEDGMENTS

AUTHOR, COPYRIGHT, AND LICENSE

This document may be distributed under the same terms as Perl itself.