708 lines
76 KiB
HTML
708 lines
76 KiB
HTML
|
||
<!DOCTYPE html>
|
||
|
||
<html xmlns="http://www.w3.org/1999/xhtml" lang="zh_CN">
|
||
<head>
|
||
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
|
||
|
||
<title>Unicode 指南 — Python 3.8.20 文档</title><meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
|
||
<link rel="stylesheet" href="../_static/pydoctheme.css" type="text/css" />
|
||
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
|
||
|
||
<script id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
|
||
<script src="../_static/jquery.js"></script>
|
||
<script src="../_static/underscore.js"></script>
|
||
<script src="../_static/doctools.js"></script>
|
||
<script src="../_static/language_data.js"></script>
|
||
<script src="../_static/translations.js"></script>
|
||
|
||
<script src="../_static/sidebar.js"></script>
|
||
|
||
<link rel="search" type="application/opensearchdescription+xml"
|
||
title="在 Python 3.8.20 文档 中搜索"
|
||
href="../_static/opensearch.xml"/>
|
||
<link rel="author" title="关于这些文档" href="../about.html" />
|
||
<link rel="index" title="索引" href="../genindex.html" />
|
||
<link rel="search" title="搜索" href="../search.html" />
|
||
<link rel="copyright" title="版权所有" href="../copyright.html" />
|
||
<link rel="next" title="如何利用 urllib 包获取网络资源" href="urllib2.html" />
|
||
<link rel="prev" title="排序指南" href="sorting.html" />
|
||
<link rel="canonical" href="https://docs.python.org/3/howto/unicode.html" />
|
||
|
||
|
||
|
||
|
||
|
||
<style>
|
||
@media only screen {
|
||
table.full-width-table {
|
||
width: 100%;
|
||
}
|
||
}
|
||
</style>
|
||
<link rel="shortcut icon" type="image/png" href="../_static/py.svg" />
|
||
<script type="text/javascript" src="../_static/copybutton.js"></script>
|
||
<script type="text/javascript" src="../_static/menu.js"></script>
|
||
|
||
</head>
|
||
<body>
|
||
<div class="mobile-nav">
|
||
<input type="checkbox" id="menuToggler" class="toggler__input" aria-controls="navigation"
|
||
aria-pressed="false" aria-expanded="false" role="button" aria-label="Menu" />
|
||
<label for="menuToggler" class="toggler__label">
|
||
<span></span>
|
||
</label>
|
||
<nav class="nav-content" role="navigation">
|
||
<a href="https://www.python.org/" class="nav-logo">
|
||
<img src="../_static/py.svg" alt="Logo"/>
|
||
</a>
|
||
<div class="version_switcher_placeholder"></div>
|
||
<form role="search" class="search" action="../search.html" method="get">
|
||
<svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" class="search-icon">
|
||
<path fill-rule="nonzero"
|
||
d="M15.5 14h-.79l-.28-.27a6.5 6.5 0 001.48-5.34c-.47-2.78-2.79-5-5.59-5.34a6.505 6.505 0 00-7.27 7.27c.34 2.8 2.56 5.12 5.34 5.59a6.5 6.5 0 005.34-1.48l.27.28v.79l4.25 4.25c.41.41 1.08.41 1.49 0 .41-.41.41-1.08 0-1.49L15.5 14zm-6 0C7.01 14 5 11.99 5 9.5S7.01 5 9.5 5 14 7.01 14 9.5 11.99 14 9.5 14z" fill="#444"></path>
|
||
</svg>
|
||
<input type="text" name="q" aria-label="快速搜索"/>
|
||
<input type="submit" value="转向"/>
|
||
</form>
|
||
</nav>
|
||
<div class="menu-wrapper">
|
||
<nav class="menu" role="navigation" aria-label="main navigation">
|
||
<div class="language_switcher_placeholder"></div>
|
||
<h3><a href="../contents.html">目录</a></h3>
|
||
<ul>
|
||
<li><a class="reference internal" href="#">Unicode 指南</a><ul>
|
||
<li><a class="reference internal" href="#introduction-to-unicode">Unicode 概述</a><ul>
|
||
<li><a class="reference internal" href="#definitions">定义</a></li>
|
||
<li><a class="reference internal" href="#encodings">编码</a></li>
|
||
<li><a class="reference internal" href="#references">参考文献</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#python-s-unicode-support">Python对Unicode的支持</a><ul>
|
||
<li><a class="reference internal" href="#the-string-type">字符串类型</a></li>
|
||
<li><a class="reference internal" href="#converting-to-bytes">转换为字节</a></li>
|
||
<li><a class="reference internal" href="#unicode-literals-in-python-source-code">Python 源代码中的 Unicode 文字</a></li>
|
||
<li><a class="reference internal" href="#unicode-properties">Unicode属性</a></li>
|
||
<li><a class="reference internal" href="#comparing-strings">字符串比较</a></li>
|
||
<li><a class="reference internal" href="#unicode-regular-expressions">Unicode 正则表达式</a></li>
|
||
<li><a class="reference internal" href="#id2">参考文献</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#reading-and-writing-unicode-data">Unicode 数据的读写</a><ul>
|
||
<li><a class="reference internal" href="#unicode-filenames">Unicode 文件名</a></li>
|
||
<li><a class="reference internal" href="#tips-for-writing-unicode-aware-programs">识别 Unicode 的编程技巧</a><ul>
|
||
<li><a class="reference internal" href="#converting-between-file-encodings">在文件编码格式之间进行转换</a></li>
|
||
<li><a class="reference internal" href="#files-in-an-unknown-encoding">编码格式未知的文件</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#id3">参考文献</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#acknowledgements">致谢</a></li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
|
||
<h4>上一个主题</h4>
|
||
<p class="topless"><a href="sorting.html"
|
||
title="上一章">排序指南</a></p>
|
||
<h4>下一个主题</h4>
|
||
<p class="topless"><a href="urllib2.html"
|
||
title="下一章">如何利用 urllib 包获取网络资源</a></p>
|
||
<div role="note" aria-label="source link">
|
||
<h3>本页</h3>
|
||
<ul class="this-page-menu">
|
||
<li><a href="../bugs.html">报告 Bug</a></li>
|
||
<li>
|
||
<a href="https://github.com/python/cpython/blob/3.8/Doc/howto/unicode.rst"
|
||
rel="nofollow">显示源代码
|
||
</a>
|
||
</li>
|
||
</ul>
|
||
</div>
|
||
</nav>
|
||
</div>
|
||
</div>
|
||
|
||
<div id="outdated-warning" style="padding: .5em; text-align: center; background-color: #FFBABA; color: #6A0E0E;">
|
||
这个文档所针对的是一个已不再受支持的 Python 旧版本。
|
||
你应当升级版本，并阅读
|
||
<a href="/3/howto/unicode.html"> Python 当前稳定版本的文档</a>.
|
||
</div>
|
||
|
||
<div class="related" role="navigation" aria-label="related navigation">
|
||
<h3>导航</h3>
|
||
<ul>
|
||
<li class="right" style="margin-right: 10px">
|
||
<a href="../genindex.html" title="总目录"
|
||
accesskey="I">索引</a></li>
|
||
<li class="right" >
|
||
<a href="../py-modindex.html" title="Python 模块索引"
|
||
>模块</a> |</li>
|
||
<li class="right" >
|
||
<a href="urllib2.html" title="如何利用 urllib 包获取网络资源"
|
||
accesskey="N">下一页</a> |</li>
|
||
<li class="right" >
|
||
<a href="sorting.html" title="排序指南"
|
||
accesskey="P">上一页</a> |</li>
|
||
|
||
<li><img src="../_static/py.svg" alt="python logo" style="vertical-align: middle; margin-top: -1px"/></li>
|
||
<li><a href="https://www.python.org/">Python</a> »</li>
|
||
<li class="switchers">
|
||
<div class="language_switcher_placeholder"></div>
|
||
<div class="version_switcher_placeholder"></div>
|
||
</li>
|
||
<li>
|
||
|
||
</li>
|
||
<li id="cpython-language-and-version">
|
||
<a href="../index.html">3.8.20 Documentation</a> »
|
||
</li>
|
||
|
||
<li class="nav-item nav-item-1"><a href="index.html" accesskey="U">Python 指南</a> »</li>
|
||
<li class="right">
|
||
|
||
|
||
<div class="inline-search" role="search">
|
||
<form class="inline-search" action="../search.html" method="get">
|
||
<input placeholder="快速搜索" aria-label="快速搜索" type="text" name="q" />
|
||
<input type="submit" value="转向" />
|
||
<input type="hidden" name="check_keywords" value="yes" />
|
||
<input type="hidden" name="area" value="default" />
|
||
</form>
|
||
</div>
|
||
|
|
||
</li>
|
||
|
||
</ul>
|
||
</div>
|
||
|
||
<div class="document">
|
||
<div class="documentwrapper">
|
||
<div class="bodywrapper">
|
||
<div class="body" role="main">
|
||
|
||
<section id="unicode-howto">
|
||
<span id="id1"></span><h1>Unicode 指南<a class="headerlink" href="#unicode-howto" title="永久链接至标题">¶</a></h1>
|
||
<dl class="field-list simple">
|
||
<dt class="field-odd">发布版本</dt>
|
||
<dd class="field-odd"><p>1.12</p>
|
||
</dd>
|
||
</dl>
|
||
<p>本文介绍了 Python 对表示文本数据的 Unicode 规范的支持，并对各种 Unicode 常见使用问题做了解释。</p>
|
||
<section id="introduction-to-unicode">
|
||
<h2>Unicode 概述<a class="headerlink" href="#introduction-to-unicode" title="永久链接至标题">¶</a></h2>
|
||
<section id="definitions">
|
||
<h3>定义<a class="headerlink" href="#definitions" title="永久链接至标题">¶</a></h3>
|
||
<p>如今的程序需要能够处理各种各样的字符。应用程序通常做了国际化处理，用户可以选择不同的语言显示信息和输出数据。同一个程序可能需要以英语、法语、日语、希伯来语或俄语输出错误信息。网页内容可能由这些语言书写，并且可能包含不同的表情符号。Python 的字符串类型采用 Unicode 标准来表示字符，使得 Python 程序能够正常处理所有这些不同的字符。</p>
|
||
<p>Unicode 规范（<a class="reference external" href="https://www.unicode.org/">https://www.unicode.org/</a>）旨在罗列人类语言所用到的所有字符，并赋予每个字符唯一的编码。该规范一直在进行修订和更新，不断加入新的语种和符号。</p>
|
||
<p>一个 <strong>字符</strong> 是文本的最小组件。‘A’、‘B’、‘C’ 等都是不同的字符。‘È’ 和 ‘Í’ 也一样。字符会随着语言或者上下文的变化而变化。比如，‘Ⅰ’ 是一个表示 “罗马数字 1” 的字符，它与大写字母 ‘I’ 不同。他们往往看起来相同，但这是两个有着不同含义的字符。</p>
|
||
<p>Unicode 标准描述了**码位**如何表示字符。一个码位的值是在 0 到 0x10FFFF （大约 110 万个值，目前有其中 11 万个被指派）。在这一标准中并且在这一文档中，一个码位写作 <code class="docutils literal notranslate"><span class="pre">U+265E</span></code> 来表示拥有值 <code class="docutils literal notranslate"><span class="pre">0x265e</span></code> 的字符（十进制下为 9,822）。</p>
|
||
<p>Unicode 标准中包含了许多表格，列出了很多字符及其对应的码位。</p>
|
||
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>0061 'a'; LATIN SMALL LETTER A
|
||
0062 'b'; LATIN SMALL LETTER B
|
||
0063 'c'; LATIN SMALL LETTER C
|
||
...
|
||
007B '{'; LEFT CURLY BRACKET
|
||
...
|
||
2167 'Ⅷ'; ROMAN NUMERAL EIGHT
|
||
2168 'Ⅸ'; ROMAN NUMERAL NINE
|
||
...
|
||
265E '♞'; BLACK CHESS KNIGHT
|
||
265F '♟'; BLACK CHESS PAWN
|
||
...
|
||
1F600 '😀'; GRINNING FACE
|
||
1F609 '😉'; WINKING FACE
|
||
...
|
||
</pre></div>
|
||
</div>
|
||
<p>严格地说，上述定义暗示了以下说法是没有意义的：“这是字符 <code class="docutils literal notranslate"><span class="pre">U+265E</span></code>”。<code class="docutils literal notranslate"><span class="pre">U+265E</span></code> 只是一个码位，代表某个特定的字符；这里它代表了字符 “国际象棋黑骑士” '♞'。在非正式的上下文中，有时会忽略码位和字符的区别。</p>
|
||
<p>一个字符在屏幕或纸上被表示为一组图形元素，被称为 <strong>字形（glyph）</strong> 。比如，大写字母 A 的字形，是两笔斜线和一笔横线，而具体的细节取决于所使用的字体。大部分 Python 代码不必担心字形，找到正确的显示字形通常是交给 GUI 工具包或终端的字体渲染程序来完成。</p>
|
||
</section>
|
||
<section id="encodings">
|
||
<h3>编码<a class="headerlink" href="#encodings" title="永久链接至标题">¶</a></h3>
|
||
<p>上一段可以归结为：一个 Unicode 字符串是一系列码位（从 0 到 <code class="docutils literal notranslate"><span class="pre">0x10FFFF</span></code> 或者说十进制的 1,114,111 的数字）组成的序列。这一序列在内存中需被表示为一组 <strong>码元（code unit）</strong> ， <strong>码元</strong> 会映射成包含八个二进制位的字节。将 Unicode 字符串翻译成字节序列的规则称为 <strong>字符编码</strong> ，或者 <strong>编码</strong> 。</p>
|
||
<p>大家首先会想到的编码可能是用 32 位的整数作为代码位，然后采用 CPU 对 32 位整数的表示法。字符串 “Python” 用这种表示法可能会如下所示：</p>
|
||
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span> P y t h o n
|
||
0x50 00 00 00 79 00 00 00 74 00 00 00 68 00 00 00 6f 00 00 00 6e 00 00 00
|
||
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
|
||
</pre></div>
|
||
</div>
|
||
<p>这种表示法非常直白，但也存在 一些问题。</p>
|
||
<ol class="arabic simple">
|
||
<li><p>不具可移植性；不同的处理器的字节序不同。</p></li>
|
||
<li><p>非常浪费空间。 在大多数文本中，大部分码位都小于 127 或 255，因此字节 <code class="docutils literal notranslate"><span class="pre">0x00</span></code> 占用了大量空间。相较于 ASCII 表示法所需的 6 个字节，以上字符串需要占用 24 个字节。RAM 用量的增加没那么要紧（台式计算机有成 GB 的 RAM，而字符串通常不会有那么大），但要把磁盘和网络带宽的用量增加 4 倍是无法忍受的。</p></li>
|
||
<li><p>与现有的 C 函数（如 <code class="docutils literal notranslate"><span class="pre">strlen()</span></code> ）不兼容，因此需要采用一套新的宽字符串函数。</p></li>
|
||
</ol>
|
||
<p>因此这种编码用得不多，人们转而选择其他更高效、更方便的编码，比如 UTF-8。</p>
|
||
<p>UTF-8 是最常用的编码之一，Python 往往默认会采用它。UTF 代表“Unicode Transformation Format”，'8' 表示编码采用 8 位数。（UTF-16 和 UTF-32 编码也是存在的，但其使用频率不如 UTF-8。）UTF-8 的规则如下：</p>
|
||
<ol class="arabic simple">
|
||
<li><p>如果码位 < 128，则直接用对应的字节值表示。</p></li>
|
||
<li><p>如果码位 >= 128，则转换为 2、3、4 个字节的序列，每个字节值都位于 128 和 255 之间。</p></li>
|
||
</ol>
|
||
<p>UTF-8 有几个很方便的特性：</p>
|
||
<ol class="arabic simple">
|
||
<li><p>可以处理任何 Unicode 码位。</p></li>
|
||
<li><p>Unicode 字符串被转换为一个字节序列，仅在表示空（null ）字符（U+0000）时才会包含零值的字节。这意味着 <code class="docutils literal notranslate"><span class="pre">strcpy()</span></code> 之类的C 函数可以处理 UTF-8 字符串，而且用那些不能处理字符串结束符之外的零值字节的协议也能发送。</p></li>
|
||
<li><p>ASCII 字符串也是也是也是合法的 UTF-8 文本。</p></li>
|
||
<li><p>UTF-8 相当紧凑；大多数常用字符均可用一两个字节表示。</p></li>
|
||
<li><p>如果字节数据被损坏或丢失，则可以找出下一个 UTF-8 码点的开始位置并重新开始同步。随机的 8 位数据也不太可能像是有效的 UTF-8 编码。</p></li>
|
||
<li><p>UTF-8 是一种面向字节的编码。编码规定了每个字符由一个或多个字节的序列表示。这避免了整数和双字节编码（如 UTF-16 和 UTF-32）可能出现的字节顺序问题，那时的字节序列会因执行编码的硬件而异。</p></li>
|
||
</ol>
|
||
</section>
|
||
<section id="references">
|
||
<h3>参考文献<a class="headerlink" href="#references" title="永久链接至标题">¶</a></h3>
|
||
<p><a class="reference external" href="http://www.unicode.org">Unicode Consortium 站点</a> 有 Unicode 规范的字符图表、词汇表和 PDF 版本。 为一些困难的阅读做好准备。 <a class="reference external" href="http://www.unicode.org/history/">Unicode 起源和发展的年表</a> 也可在该站点上找到。</p>
|
||
<p>在 Computerphile 的 Youtube 频道上，Tom Scott 简要地 <a class="reference external" href="https://www.youtube.com/watch?v=MijmeoH9LT4">讨论了 Unicode 和 UTF-8</a> （9 分 36 秒）的历史。</p>
|
||
<p>为了帮助理解该标准，Jukka Korpela 编写了阅读 Unicode 字符表的`介绍性指南 <<a class="reference external" href="http://jkorpela.fi/unicode/guide.html">http://jkorpela.fi/unicode/guide.html</a>>`_ 。</p>
|
||
<p>Joel Spolsky 撰写了另一篇不错的介绍性文章 <<a class="reference external" href="https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character">https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character</a>- set-no-excuses/>`_ 。如果本文没让您弄清楚，那应在继续之前先试着读读这篇文章。</p>
|
||
<p>Wikipedia 条目通常也有帮助；请参阅“<a class="reference external" href="https://en.wikipedia.org/wiki/Character_encoding">字符编码</a>”和 <a class="reference external" href="https://en.wikipedia.org/wiki/UTF-8">UTF-8</a> 的条目，例如：</p>
|
||
</section>
|
||
</section>
|
||
<section id="python-s-unicode-support">
|
||
<h2>Python对Unicode的支持<a class="headerlink" href="#python-s-unicode-support" title="永久链接至标题">¶</a></h2>
|
||
<p>现在您已经了解了 Unicode 的基础知识，可以看下 Python 的 Unicode 特性。</p>
|
||
<section id="the-string-type">
|
||
<h3>字符串类型<a class="headerlink" href="#the-string-type" title="永久链接至标题">¶</a></h3>
|
||
<p>从 Python 3.0 开始， <a class="reference internal" href="../library/stdtypes.html#str" title="str"><code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code></a> 类型包含了 Unicode 字符，这意味着用 <code class="docutils literal notranslate"><span class="pre">"unicode</span> <span class="pre">rocks!"</span></code>、<code class="docutils literal notranslate"><span class="pre">'unicode</span> <span class="pre">rocks!'</span></code> 或三重引号字符串语法创建的任何字符串都会存储为 Unicode。</p>
|
||
<p>Python 源代码的默认编码是 UTF-8，因此可以直接在字符串中包含 Unicode 字符：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="k">try</span><span class="p">:</span>
|
||
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'/tmp/input.txt'</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
|
||
<span class="o">...</span>
|
||
<span class="k">except</span> <span class="ne">OSError</span><span class="p">:</span>
|
||
<span class="c1"># 'File not found' error message.</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Fichier non trouvé"</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>旁注：Python 3 还支持在标识符中使用 Unicode 字符：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="n">répertoire</span> <span class="o">=</span> <span class="s2">"/tmp/records.log"</span>
|
||
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">répertoire</span><span class="p">,</span> <span class="s2">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
|
||
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s2">"test</span><span class="se">\n</span><span class="s2">"</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>如果无法在编辑器中输入某个字符，或出于某种原因想只保留 ASCII 编码的源代码，则还可以在字符串中使用转义序列。（根据系统的不同，可能会看到真的大写 Delta 字体而不是 u 转义符。）：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="s2">"</span><span class="se">\N{GREEK CAPITAL LETTER DELTA}</span><span class="s2">"</span> <span class="c1"># Using the character name</span>
|
||
<span class="go">'\u0394'</span>
|
||
<span class="gp">>>> </span><span class="s2">"</span><span class="se">\u0394</span><span class="s2">"</span> <span class="c1"># Using a 16-bit hex value</span>
|
||
<span class="go">'\u0394'</span>
|
||
<span class="gp">>>> </span><span class="s2">"</span><span class="se">\U00000394</span><span class="s2">"</span> <span class="c1"># Using a 32-bit hex value</span>
|
||
<span class="go">'\u0394'</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>此外，可以用 <a class="reference internal" href="../library/stdtypes.html#bytes" title="bytes"><code class="xref py py-class docutils literal notranslate"><span class="pre">bytes</span></code></a> 的 <a class="reference internal" href="../library/stdtypes.html#bytes.decode" title="bytes.decode"><code class="xref py py-func docutils literal notranslate"><span class="pre">decode()</span></code></a> 方法创建一个字符串。 该方法可以接受 <em>encoding</em> 参数，比如可以为 <code class="docutils literal notranslate"><span class="pre">UTF-8</span></code> ，以及可选的 <em>errors</em> 参数。</p>
|
||
<p>若无法根据编码规则对输入字符串进行编码，<em>errors</em> 参数指定了响应策略。 该参数的合法值可以是 <code class="docutils literal notranslate"><span class="pre">'strict'</span></code> (触发 <a class="reference internal" href="../library/exceptions.html#UnicodeDecodeError" title="UnicodeDecodeError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">UnicodeDecodeError</span></code></a> 异常)、<code class="docutils literal notranslate"><span class="pre">'replace'</span></code> (用 <code class="docutils literal notranslate"><span class="pre">U+FFFD</span></code>、<code class="docutils literal notranslate"><span class="pre">REPLACEMENT</span> <span class="pre">CHARACTER</span></code>)、<code class="docutils literal notranslate"><span class="pre">'ignore'</span></code> (只是将字符从 Unicode 结果中去掉)，或 <code class="docutils literal notranslate"><span class="pre">'backslashreplace'</span></code> (插入一个 <code class="docutils literal notranslate"><span class="pre">\xNN</span></code> 转义序列)。 以下示例演示了这些不同的参数:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="sa">b</span><span class="s1">'</span><span class="se">\x80</span><span class="s1">abc'</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">,</span> <span class="s2">"strict"</span><span class="p">)</span>
|
||
<span class="gt">Traceback (most recent call last):</span>
|
||
<span class="w"> </span><span class="o">...</span>
|
||
<span class="gr">UnicodeDecodeError</span>: <span class="n">'utf-8' codec can't decode byte 0x80 in position 0:</span>
|
||
<span class="x"> invalid start byte</span>
|
||
<span class="gp">>>> </span><span class="sa">b</span><span class="s1">'</span><span class="se">\x80</span><span class="s1">abc'</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">,</span> <span class="s2">"replace"</span><span class="p">)</span>
|
||
<span class="go">'\ufffdabc'</span>
|
||
<span class="gp">>>> </span><span class="sa">b</span><span class="s1">'</span><span class="se">\x80</span><span class="s1">abc'</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">,</span> <span class="s2">"backslashreplace"</span><span class="p">)</span>
|
||
<span class="go">'\\x80abc'</span>
|
||
<span class="gp">>>> </span><span class="sa">b</span><span class="s1">'</span><span class="se">\x80</span><span class="s1">abc'</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">,</span> <span class="s2">"ignore"</span><span class="p">)</span>
|
||
<span class="go">'abc'</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>编码格式以包含编码格式名称的字符串来指明。 Python 有大约 100 种不同的编码格式；清单详见 Python 库参考文档 <a class="reference internal" href="../library/codecs.html#standard-encodings"><span class="std std-ref">标准编码</span></a>。 一些编码格式有多个名称，比如 <code class="docutils literal notranslate"><span class="pre">'latin-1'</span></code>、<code class="docutils literal notranslate"><span class="pre">'iso_8859_1'</span></code> 和 <code class="docutils literal notranslate"><span class="pre">'8859</span></code> 都是指同一种编码。</p>
|
||
<p>利用内置函数 <a class="reference internal" href="../library/functions.html#chr" title="chr"><code class="xref py py-func docutils literal notranslate"><span class="pre">chr()</span></code></a> 还可以创建单字符的 Unicode 字符串，该函数可接受整数参数，并返回包含对应码位的长度为 1 的 Unicode 字符串。内置函数 <a class="reference internal" href="../library/functions.html#ord" title="ord"><code class="xref py py-func docutils literal notranslate"><span class="pre">ord()</span></code></a> 是其逆操作，参数为单个字符的 Unicode 字符串，并返回码位值：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">chr</span><span class="p">(</span><span class="mi">57344</span><span class="p">)</span>
|
||
<span class="go">'\ue000'</span>
|
||
<span class="gp">>>> </span><span class="nb">ord</span><span class="p">(</span><span class="s1">'</span><span class="se">\ue000</span><span class="s1">'</span><span class="p">)</span>
|
||
<span class="go">57344</span>
|
||
</pre></div>
|
||
</div>
|
||
</section>
|
||
<section id="converting-to-bytes">
|
||
<h3>转换为字节<a class="headerlink" href="#converting-to-bytes" title="永久链接至标题">¶</a></h3>
|
||
<p><a class="reference internal" href="../library/stdtypes.html#bytes.decode" title="bytes.decode"><code class="xref py py-meth docutils literal notranslate"><span class="pre">bytes.decode()</span></code></a> 的逆方法是 <a class="reference internal" href="../library/stdtypes.html#str.encode" title="str.encode"><code class="xref py py-meth docutils literal notranslate"><span class="pre">str.encode()</span></code></a> ，它会返回 Unicode 字符串的 <a class="reference internal" href="../library/stdtypes.html#bytes" title="bytes"><code class="xref py py-class docutils literal notranslate"><span class="pre">bytes</span></code></a> 形式，已按要求的 <em>encoding</em> 进行了编码。</p>
|
||
<p>参数 <em>errors</em> 的意义与 <a class="reference internal" href="../library/stdtypes.html#bytes.decode" title="bytes.decode"><code class="xref py py-meth docutils literal notranslate"><span class="pre">decode()</span></code></a> 方法相同，但支持更多可能的handler。除了 <code class="docutils literal notranslate"><span class="pre">'strict'</span></code> 、 <code class="docutils literal notranslate"><span class="pre">'ignore'</span></code> 和 <code class="docutils literal notranslate"><span class="pre">'replace'</span></code> （这时会插入问号替换掉无法编码的字符），还有 <code class="docutils literal notranslate"><span class="pre">'xmlcharrefreplace'</span></code> （插入一个 XML 字符引用）、 <code class="docutils literal notranslate"><span class="pre">backslashreplace</span></code> （插入一个 <code class="docutils literal notranslate"><span class="pre">\uNNNN</span></code> 转义序列）和 <code class="docutils literal notranslate"><span class="pre">namereplace</span></code> （插入一个 <code class="docutils literal notranslate"><span class="pre">\N{...}</span></code> 转义序列 ）。</p>
|
||
<p>以下例子演示了各种不同的结果：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">u</span> <span class="o">=</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">40960</span><span class="p">)</span> <span class="o">+</span> <span class="s1">'abcd'</span> <span class="o">+</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">1972</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">u</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
|
||
<span class="go">b'\xea\x80\x80abcd\xde\xb4'</span>
|
||
<span class="gp">>>> </span><span class="n">u</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">)</span>
|
||
<span class="gt">Traceback (most recent call last):</span>
|
||
<span class="w"> </span><span class="o">...</span>
|
||
<span class="gr">UnicodeEncodeError</span>: <span class="n">'ascii' codec can't encode character '\ua000' in</span>
|
||
<span class="x"> position 0: ordinal not in range(128)</span>
|
||
<span class="gp">>>> </span><span class="n">u</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">,</span> <span class="s1">'ignore'</span><span class="p">)</span>
|
||
<span class="go">b'abcd'</span>
|
||
<span class="gp">>>> </span><span class="n">u</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">,</span> <span class="s1">'replace'</span><span class="p">)</span>
|
||
<span class="go">b'?abcd?'</span>
|
||
<span class="gp">>>> </span><span class="n">u</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">,</span> <span class="s1">'xmlcharrefreplace'</span><span class="p">)</span>
|
||
<span class="go">b'&#40960;abcd&#1972;'</span>
|
||
<span class="gp">>>> </span><span class="n">u</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">,</span> <span class="s1">'backslashreplace'</span><span class="p">)</span>
|
||
<span class="go">b'\\ua000abcd\\u07b4'</span>
|
||
<span class="gp">>>> </span><span class="n">u</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">,</span> <span class="s1">'namereplace'</span><span class="p">)</span>
|
||
<span class="go">b'\\N{YI SYLLABLE IT}abcd\\u07b4'</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>用于注册和访问可用编码格式的底层函数，位于 <a class="reference internal" href="../library/codecs.html#module-codecs" title="codecs: Encode and decode data and streams."><code class="xref py py-mod docutils literal notranslate"><span class="pre">codecs</span></code></a> 模块中。 若要实现新的编码格式，则还需要了解 <a class="reference internal" href="../library/codecs.html#module-codecs" title="codecs: Encode and decode data and streams."><code class="xref py py-mod docutils literal notranslate"><span class="pre">codecs</span></code></a> 模块。 不过该模块返回的编码和解码函数通常更为底层一些，不大好用，编写新的编码格式是一项专业的任务，因此本文不会涉及该模块。</p>
|
||
</section>
|
||
<section id="unicode-literals-in-python-source-code">
|
||
<h3>Python 源代码中的 Unicode 文字<a class="headerlink" href="#unicode-literals-in-python-source-code" title="永久链接至标题">¶</a></h3>
|
||
<p>在 Python 源代码中，可以用 <code class="docutils literal notranslate"><span class="pre">\u</span></code> 转义序列书写特定的 Unicode 码位，该序列后跟 4 个代表码位的十六进制数字。<code class="docutils literal notranslate"><span class="pre">\U</span></code> 转义序列用法类似，但要用8 个十六进制数字，而不是 4 个：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">s</span> <span class="o">=</span> <span class="s2">"a</span><span class="se">\xac\u1234\u20ac\U00008000</span><span class="s2">"</span>
|
||
<span class="gp">... </span><span class="c1"># ^^^^ two-digit hex escape</span>
|
||
<span class="gp">... </span><span class="c1"># ^^^^^^ four-digit Unicode escape</span>
|
||
<span class="gp">... </span><span class="c1"># ^^^^^^^^^^ eight-digit Unicode escape</span>
|
||
<span class="gp">>>> </span><span class="p">[</span><span class="nb">ord</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">s</span><span class="p">]</span>
|
||
<span class="go">[97, 172, 4660, 8364, 32768]</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>对大于 127 的码位使用转义序列，数量不多时没什么问题，但如果要用到很多重音字符，这会变得很烦人，类似于程序中的信息是用法语或其他使用重音的语言写的。也可以用内置函数 <a class="reference internal" href="../library/functions.html#chr" title="chr"><code class="xref py py-func docutils literal notranslate"><span class="pre">chr()</span></code></a> 拼装字符串，但会更加乏味。</p>
|
||
<p>理想情况下，都希望能用母语的编码书写文本。还能用喜好的编辑器编辑 Python 源代码，编辑器要能自然地显示重音符，并在运行时使用正确的字符。</p>
|
||
<p>默认情况下，Python 支持以 UTF-8 格式编写源代码，但如果声明要用的编码，则几乎可以使用任何编码。只要在源文件的第一行或第二行包含一个特殊注释即可：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env python</span>
|
||
<span class="c1"># -*- coding: latin-1 -*-</span>
|
||
|
||
<span class="n">u</span> <span class="o">=</span> <span class="s1">'abcdé'</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="nb">ord</span><span class="p">(</span><span class="n">u</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]))</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>上述语法的灵感来自于 Emacs 用于指定文件局部变量的符号。Emacs 支持许多不同的变量，但 Python 仅支持“编码”。 <code class="docutils literal notranslate"><span class="pre">-*-</span></code> 符号向 Emacs 标明该注释是特殊的；这对 Python 没有什么意义，只是一种约定。Python 会在注释中查找 <code class="docutils literal notranslate"><span class="pre">coding:</span> <span class="pre">name</span></code> 或 <code class="docutils literal notranslate"><span class="pre">coding=name</span></code> 。</p>
|
||
<p>如果没有这种注释，则默认编码将会是前面提到的 UTF-8。更多信息请参阅 <span class="target" id="index-2"></span><a class="pep reference external" href="https://www.python.org/dev/peps/pep-0263"><strong>PEP 263</strong></a> 。</p>
|
||
</section>
|
||
<section id="unicode-properties">
|
||
<h3>Unicode属性<a class="headerlink" href="#unicode-properties" title="永久链接至标题">¶</a></h3>
|
||
<p>Unicode 规范包含了一个码位信息数据库。对于定义的每一个码位，都包含了字符的名称、类别、数值（对于表示数字概念的字符，如罗马数字、分数如三分之一和五分之四等）。还有有关显示的属性，比如如何在双向文本中使用码位。</p>
|
||
<p>以下程序显示了几个字符的信息，并打印一个字符的数值：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">unicodedata</span>
|
||
|
||
<span class="n">u</span> <span class="o">=</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">233</span><span class="p">)</span> <span class="o">+</span> <span class="nb">chr</span><span class="p">(</span><span class="mh">0x0bf2</span><span class="p">)</span> <span class="o">+</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">3972</span><span class="p">)</span> <span class="o">+</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">6000</span><span class="p">)</span> <span class="o">+</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">13231</span><span class="p">)</span>
|
||
|
||
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">c</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">u</span><span class="p">):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="s1">'</span><span class="si">%04x</span><span class="s1">'</span> <span class="o">%</span> <span class="nb">ord</span><span class="p">(</span><span class="n">c</span><span class="p">),</span> <span class="n">unicodedata</span><span class="o">.</span><span class="n">category</span><span class="p">(</span><span class="n">c</span><span class="p">),</span> <span class="n">end</span><span class="o">=</span><span class="s2">" "</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">unicodedata</span><span class="o">.</span><span class="n">name</span><span class="p">(</span><span class="n">c</span><span class="p">))</span>
|
||
|
||
<span class="c1"># Get numeric value of second character</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">unicodedata</span><span class="o">.</span><span class="n">numeric</span><span class="p">(</span><span class="n">u</span><span class="p">[</span><span class="mi">1</span><span class="p">]))</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>当运行时，这将打印出：</p>
|
||
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>0 00e9 Ll LATIN SMALL LETTER E WITH ACUTE
|
||
1 0bf2 No TAMIL NUMBER ONE THOUSAND
|
||
2 0f84 Mn TIBETAN MARK HALANTA
|
||
3 1770 Lo TAGBANWA LETTER SA
|
||
4 33af So SQUARE RAD OVER S SQUARED
|
||
1000.0
|
||
</pre></div>
|
||
</div>
|
||
<p>类别代码是描述字符性质的缩写。 这些被分组为“字母”、“数字”、“标点符号”或“符号”等类别，而这些类别又分为子类别。 从上面的输出中获取代码，<code class="docutils literal notranslate"><span class="pre">'Ll'</span></code> 表示“字母，小写”，<code class="docutils literal notranslate"><span class="pre">'No'</span></code> 表示“数字，其他”，<code class="docutils literal notranslate"><span class="pre">'Mn'</span></code> 是“标记，非间距”，<code class="docutils literal notranslate"><span class="pre">'So'</span></code> 是“符号，其他”。 有关类别代码列表，请参阅 Unicode 字符数据库文档 <<a class="reference external" href="http://www.unicode.org/reports/tr44/#General_Category_Values">http://www.unicode.org/reports/tr44/#General_Category_Values</a>>`_ 的“通用类别值”部分。</p>
|
||
</section>
|
||
<section id="comparing-strings">
|
||
<h3>字符串比较<a class="headerlink" href="#comparing-strings" title="永久链接至标题">¶</a></h3>
|
||
<p>Unicode 让字符串的比较变得复杂了一些，因为同一组字符可能由不同的码位序列组成。例如，像“ê”这样的字母可以表示为单码位 U+00EA，或是 U+0065 U+0302，即“e”的码位后跟“COMBINING CIRCUMFLEX ACCENT”的码位。虽然在打印时会产生同样的输出，但一个是长度为 1 的字符串，另一个是长度为 2 的字符串。</p>
|
||
<p>一种不区分大小写比较的工具是字符串方法 <a class="reference internal" href="../library/stdtypes.html#str.casefold" title="str.casefold"><code class="xref py py-meth docutils literal notranslate"><span class="pre">casefold()</span></code></a> ，将按照 Unicode 标准描述的算法将字符串转换为不区分大小写的形式。该算法对诸如德语字母“ß”（代码点 U+00DF）之类的字符进行了特殊处理，变为一对小写字母“ss”。</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">street</span> <span class="o">=</span> <span class="s1">'Gürzenichstraße'</span>
|
||
<span class="gp">>>> </span><span class="n">street</span><span class="o">.</span><span class="n">casefold</span><span class="p">()</span>
|
||
<span class="go">'gürzenichstrasse'</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>第二个工具是 <a class="reference internal" href="../library/unicodedata.html#module-unicodedata" title="unicodedata: Access the Unicode Database."><code class="xref py py-mod docutils literal notranslate"><span class="pre">unicodedata</span></code></a> 模块的 <a class="reference internal" href="../library/unicodedata.html#unicodedata.normalize" title="unicodedata.normalize"><code class="xref py py-func docutils literal notranslate"><span class="pre">normalize()</span></code></a> 函数，将字符串转换为几种规范化形式之一，其中后跟组合字符的字母将被替换为单个字符。 <code class="xref py py-func docutils literal notranslate"><span class="pre">normalize()</span></code> 可用于执行字符串比较，即便两个字符串采用不同的字符组合，也不会错误地报告两者不相等：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">unicodedata</span>
|
||
|
||
<span class="k">def</span> <span class="nf">compare_strs</span><span class="p">(</span><span class="n">s1</span><span class="p">,</span> <span class="n">s2</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="nf">NFD</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="n">unicodedata</span><span class="o">.</span><span class="n">normalize</span><span class="p">(</span><span class="s1">'NFD'</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span>
|
||
|
||
<span class="k">return</span> <span class="n">NFD</span><span class="p">(</span><span class="n">s1</span><span class="p">)</span> <span class="o">==</span> <span class="n">NFD</span><span class="p">(</span><span class="n">s2</span><span class="p">)</span>
|
||
|
||
<span class="n">single_char</span> <span class="o">=</span> <span class="s1">'ê'</span>
|
||
<span class="n">multiple_chars</span> <span class="o">=</span> <span class="s1">'</span><span class="se">\N{LATIN SMALL LETTER E}\N{COMBINING CIRCUMFLEX ACCENT}</span><span class="s1">'</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'length of first string='</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">single_char</span><span class="p">))</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'length of second string='</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">multiple_chars</span><span class="p">))</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">compare_strs</span><span class="p">(</span><span class="n">single_char</span><span class="p">,</span> <span class="n">multiple_chars</span><span class="p">))</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>当运行时，这将输出：</p>
|
||
<div class="highlight-shell-session notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>python3<span class="w"> </span>compare-strs.py
|
||
<span class="go">length of first string= 1</span>
|
||
<span class="go">length of second string= 2</span>
|
||
<span class="go">True</span>
|
||
</pre></div>
|
||
</div>
|
||
<p><a class="reference internal" href="../library/unicodedata.html#unicodedata.normalize" title="unicodedata.normalize"><code class="xref py py-func docutils literal notranslate"><span class="pre">normalize()</span></code></a> 函数的第一个参数是个字符串，给出所需的规范化形式，可以是“NFC”、“NFKC”、“NFD”和“NFKD”之一。</p>
|
||
<p>Unicode 标准还设定了如何进行不区分大小写的比较：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">unicodedata</span>
|
||
|
||
<span class="k">def</span> <span class="nf">compare_caseless</span><span class="p">(</span><span class="n">s1</span><span class="p">,</span> <span class="n">s2</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="nf">NFD</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="n">unicodedata</span><span class="o">.</span><span class="n">normalize</span><span class="p">(</span><span class="s1">'NFD'</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span>
|
||
|
||
<span class="k">return</span> <span class="n">NFD</span><span class="p">(</span><span class="n">NFD</span><span class="p">(</span><span class="n">s1</span><span class="p">)</span><span class="o">.</span><span class="n">casefold</span><span class="p">())</span> <span class="o">==</span> <span class="n">NFD</span><span class="p">(</span><span class="n">NFD</span><span class="p">(</span><span class="n">s2</span><span class="p">)</span><span class="o">.</span><span class="n">casefold</span><span class="p">())</span>
|
||
|
||
<span class="c1"># Example usage</span>
|
||
<span class="n">single_char</span> <span class="o">=</span> <span class="s1">'ê'</span>
|
||
<span class="n">multiple_chars</span> <span class="o">=</span> <span class="s1">'</span><span class="se">\N{LATIN CAPITAL LETTER E}\N{COMBINING CIRCUMFLEX ACCENT}</span><span class="s1">'</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">compare_caseless</span><span class="p">(</span><span class="n">single_char</span><span class="p">,</span> <span class="n">multiple_chars</span><span class="p">))</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>这将打印 <code class="docutils literal notranslate"><span class="pre">True</span></code> 。（为什么 <code class="xref py py-func docutils literal notranslate"><span class="pre">NFD()</span></code> 会被调用两次？因为有几个字符让 <code class="xref py py-meth docutils literal notranslate"><span class="pre">casefold()</span></code> 返回非规范化的字符串，所以结果需要再次进行规范化。参见 Unicode 标准的 3.13 节 的一个讨论和示例。）</p>
|
||
</section>
|
||
<section id="unicode-regular-expressions">
|
||
<h3>Unicode 正则表达式<a class="headerlink" href="#unicode-regular-expressions" title="永久链接至标题">¶</a></h3>
|
||
<p><a class="reference internal" href="../library/re.html#module-re" title="re: Regular expression operations."><code class="xref py py-mod docutils literal notranslate"><span class="pre">re</span></code></a> 模块支持的正则表达式可以用字节串或字符串的形式提供。有一些特殊字符序列，比如 <code class="docutils literal notranslate"><span class="pre">\d</span></code> 和 <code class="docutils literal notranslate"><span class="pre">\w</span></code> 具有不同的含义，具体取决于匹配模式是以字节串还是字符串形式提供的。例如，<code class="docutils literal notranslate"><span class="pre">\d</span></code> 将匹配字节串中的字符 <code class="docutils literal notranslate"><span class="pre">[0-9]</span></code> ，但对于字符串将会匹配 <code class="docutils literal notranslate"><span class="pre">'Nd'</span></code> 类别中的任何字符。</p>
|
||
<p>上述示例中的字符串包含了泰语和阿拉伯数字书写的数字 57：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">re</span>
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s1">'\d+'</span><span class="p">)</span>
|
||
|
||
<span class="n">s</span> <span class="o">=</span> <span class="s2">"Over </span><span class="se">\u0e55\u0e57</span><span class="s2"> 57 flavours"</span>
|
||
<span class="n">m</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="nb">repr</span><span class="p">(</span><span class="n">m</span><span class="o">.</span><span class="n">group</span><span class="p">()))</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>执行时，<code class="docutils literal notranslate"><span class="pre">\d+</span></code> 将匹配上泰语数字并打印出来。如果向 <a class="reference internal" href="../library/re.html#re.compile" title="re.compile"><code class="xref py py-func docutils literal notranslate"><span class="pre">compile()</span></code></a> 提供的是 <a class="reference internal" href="../library/re.html#re.ASCII" title="re.ASCII"><code class="xref py py-const docutils literal notranslate"><span class="pre">re.ASCII</span></code></a> 标志，<code class="docutils literal notranslate"><span class="pre">\d+</span></code> 则会匹配子串 "57"。</p>
|
||
<p>类似地，<code class="docutils literal notranslate"><span class="pre">\w</span></code> 将匹配多种 Unicode 字符，但对于字节串则只会匹配 <code class="docutils literal notranslate"><span class="pre">[a-zA-Z0-9_]</span></code> ，如果指定 <a class="reference internal" href="../library/re.html#re.ASCII" title="re.ASCII"><code class="xref py py-const docutils literal notranslate"><span class="pre">re.ASCII</span></code></a> ， <code class="docutils literal notranslate"><span class="pre">\s</span></code> 将匹配 Unicode 空白符或 <code class="docutils literal notranslate"><span class="pre">[</span> <span class="pre">\t\n\r\f\v]</span></code> 。</p>
|
||
</section>
|
||
<section id="id2">
|
||
<h3>参考文献<a class="headerlink" href="#id2" title="永久链接至标题">¶</a></h3>
|
||
<p>关于 Python 的 Unicode 支持，其他还有一些很好的讨论：</p>
|
||
<ul class="simple">
|
||
<li><p><a class="reference external" href="http://python-notes.curiousefficiency.org/en/latest/python3/text_file_processing.html">用 Python 3 处理文本文件</a> ，作者 Nick Coghlan。</p></li>
|
||
<li><p><a class="reference external" href="https://nedbatchelder.com/text/unipain.html">实用的 Unicode</a>，Ned Batchelder 在 PyCon 2012 上的演示。</p></li>
|
||
</ul>
|
||
<p><a class="reference internal" href="../library/stdtypes.html#str" title="str"><code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code></a> 类型在 Python 库参考文档 <a class="reference internal" href="../library/stdtypes.html#textseq"><span class="std std-ref">文本序列类型 --- str</span></a> 中有介绍。</p>
|
||
<p><a class="reference internal" href="../library/unicodedata.html#module-unicodedata" title="unicodedata: Access the Unicode Database."><code class="xref py py-mod docutils literal notranslate"><span class="pre">unicodedata</span></code></a> 模块的文档</p>
|
||
<p><a class="reference internal" href="../library/codecs.html#module-codecs" title="codecs: Encode and decode data and streams."><code class="xref py py-mod docutils literal notranslate"><span class="pre">codecs</span></code></a> 模块的文档</p>
|
||
<p>Marc-André Lemburg 在 EuroPython 2002 上做了一个题为“Python 和 Unicode”（PDF 幻灯片）<<a class="reference external" href="https://downloads.egenix.com/python/Unicode-EPC2002-Talk.pdf">https://downloads.egenix.com/python/Unicode-EPC2002-Talk.pdf</a>>`_ 的演示文稿。该幻灯片很好地概括了 Python 2 的 Unicode 功能设计（其中 Unicode 字符串类型称为 <code class="docutils literal notranslate"><span class="pre">unicode</span></code>，文字以 <code class="docutils literal notranslate"><span class="pre">u</span></code> 开头）。</p>
|
||
</section>
|
||
</section>
|
||
<section id="reading-and-writing-unicode-data">
|
||
<h2>Unicode 数据的读写<a class="headerlink" href="#reading-and-writing-unicode-data" title="永久链接至标题">¶</a></h2>
|
||
<p>既然处理 Unicode 数据的代码写好了，下一个问题就是输入/输出了。如何将 Unicode 字符串读入程序，如何将 Unicode 转换为适于存储或传输的形式呢？</p>
|
||
<p>根据输入源和输出目标的不同，或许什么都不用干；请检查一下应用程序用到的库是否原生支持 Unicode。例如，XML 解析器往往会返回 Unicode 数据。许多关系数据库的字段也支持 Unicode 值，并且 SQL 查询也能返回 Unicode 值。</p>
|
||
<p>在写入磁盘或通过套接字发送之前，Unicode 数据通常要转换为特定的编码。可以自己完成所有工作：打开一个文件，从中读取一个 8 位字节对象，然后用 <code class="docutils literal notranslate"><span class="pre">bytes.decode(encoding)</span></code> 对字节串进行转换。但是，不推荐采用这种全人工的方案。</p>
|
||
<p>编码的多字节特性就是一个难题； 一个 Unicode 字符可以用几个字节表示。 如果要以任意大小的块（例如 1024 或 4096 字节）读取文件，那么在块的末尾可能只读到某个 Unicode 字符的部分字节，这就需要编写错误处理代码。 有一种解决方案是将整个文件读入内存，然后进行解码，但这样就没法处理很大的文件了；若要读取 2 GB 的文件，就需要 2 GB 的 RAM。（其实需要的内存会更多些，因为至少有一段时间需要在内存中同时存放已编码字符串及其 Unicode 版本。）</p>
|
||
<p>解决方案是利用底层解码接口去捕获编码序列不完整的情况。这部分代码已经是现成的：内置函数 <a class="reference internal" href="../library/functions.html#open" title="open"><code class="xref py py-func docutils literal notranslate"><span class="pre">open()</span></code></a> 可以返回一个文件类的对象，该对象认为文件的内容采用指定的编码，<a class="reference internal" href="../library/io.html#io.TextIOBase.read" title="io.TextIOBase.read"><code class="xref py py-meth docutils literal notranslate"><span class="pre">read()</span></code></a> 和 <a class="reference internal" href="../library/io.html#io.TextIOBase.write" title="io.TextIOBase.write"><code class="xref py py-meth docutils literal notranslate"><span class="pre">write()</span></code></a> 等方法接受 Unicode 参数。只要用 <a class="reference internal" href="../library/functions.html#open" title="open"><code class="xref py py-func docutils literal notranslate"><span class="pre">open()</span></code></a> 的 <em>encoding</em> 和 <em>errors</em> 参数即可，参数释义同 <a class="reference internal" href="../library/stdtypes.html#str.encode" title="str.encode"><code class="xref py py-meth docutils literal notranslate"><span class="pre">str.encode()</span></code></a> 和 <a class="reference internal" href="../library/stdtypes.html#bytes.decode" title="bytes.decode"><code class="xref py py-meth docutils literal notranslate"><span class="pre">bytes.decode()</span></code></a> 。</p>
|
||
<p>因此从文件读取 Unicode 就比较简单了：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'unicode.txt'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
|
||
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">f</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="nb">repr</span><span class="p">(</span><span class="n">line</span><span class="p">))</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>也可以在更新模式下打开文件，以便同时读取和写入：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'test'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">'w+'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
|
||
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'</span><span class="se">\u4500</span><span class="s1"> blah blah blah</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
|
||
<span class="n">f</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="nb">repr</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">readline</span><span class="p">()[:</span><span class="mi">1</span><span class="p">]))</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Unicode 字符 <code class="docutils literal notranslate"><span class="pre">U+FEFF</span></code> 用作字节顺序标记（BOM），通常作为文件的第一个字符写入，以帮助自动检测文件的字节顺序。某些编码（例如 UTF-16）期望在文件开头出现 BOM；当采用这种编码时，BOM 将自动作为第一个字符写入，并在读取文件时会静默删除。这些编码有多种变体，例如用于 little-endian 和 big-endian 编码的 “utf-16-le” 和 “utf-16-be”，会指定一种特定的字节顺序并且不会忽略 BOM。</p>
|
||
<p>在某些地区，习惯在 UTF-8 编码文件的开头用上“BOM”；此名称具有误导性，因为 UTF-8 与字节顺序无关。此标记只是声明该文件以 UTF-8 编码。要读取此类文件，请使用“utf-8-sig”编解码器自动忽略此标记。</p>
|
||
<section id="unicode-filenames">
|
||
<h3>Unicode 文件名<a class="headerlink" href="#unicode-filenames" title="永久链接至标题">¶</a></h3>
|
||
<p>当今大多数操作系统的文件名都支持包含任意 Unicode 字符。通常这是通过将 Unicode 字符串转换为某系统的编码来实现的。如今的 Python 趋向于采用 UTF-8 编码：MacOS 上的 Python 已经在多个版本中采用了 UTF-8，而 Python 3.6 在 Windows 上也改用了 UTF-8。在 Unix 系统中，只有设置了 <code class="docutils literal notranslate"><span class="pre">LANG</span></code> 或 <code class="docutils literal notranslate"><span class="pre">LC_CTYPE</span></code> 环境变量，才会采用某个文件系统编码；否则默认编码还是 UTF-8。</p>
|
||
<p><a class="reference internal" href="../library/sys.html#sys.getfilesystemencoding" title="sys.getfilesystemencoding"><code class="xref py py-func docutils literal notranslate"><span class="pre">sys.getfilesystemencoding()</span></code></a> 函数将返回要在当前系统采用的编码，若想手动进行编码时即可用到，但无需多虑。在打开文件进行读写时，通常只需提供 Unicode 字符串作为文件名，会自动转换为合适的编码格式：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="n">filename</span> <span class="o">=</span> <span class="s1">'filename</span><span class="se">\u4500</span><span class="s1">abc'</span>
|
||
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
|
||
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'blah</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
<p><a class="reference internal" href="../library/os.html#module-os" title="os: Miscellaneous operating system interfaces."><code class="xref py py-mod docutils literal notranslate"><span class="pre">os</span></code></a> 模块中的函数也能接受 Unicode 文件名，如 <a class="reference internal" href="../library/os.html#os.stat" title="os.stat"><code class="xref py py-func docutils literal notranslate"><span class="pre">os.stat()</span></code></a> 。</p>
|
||
<p><a class="reference internal" href="../library/os.html#os.listdir" title="os.listdir"><code class="xref py py-func docutils literal notranslate"><span class="pre">os.listdir()</span></code></a> 函数将返回文件名，这会引发一个难题：到底应返回文件名的 Unicode 版本，还是应返回已编码的字节串呢？这两种操作 <a class="reference internal" href="../library/os.html#os.listdir" title="os.listdir"><code class="xref py py-func docutils literal notranslate"><span class="pre">os.listdir()</span></code></a> 均可执行，具体取决于给出的目录路径是字节串形式还是 Unicode 字符串。若传入一个 Unicode 字符串作为路径，文件名将用当前文件系统的编码进行解码并返回一个 Unicode 字符串列表，而若传入一个字节串路径则会返回字节串格式的文件名。以下假定默认文件系统编码为 UTF-8，运行以下程序：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="n">fn</span> <span class="o">=</span> <span class="s1">'filename</span><span class="se">\u4500</span><span class="s1">abc'</span>
|
||
<span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">fn</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span>
|
||
<span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
|
||
|
||
<span class="kn">import</span> <span class="nn">os</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="sa">b</span><span class="s1">'.'</span><span class="p">))</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="s1">'.'</span><span class="p">))</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>将产生以下输出：</p>
|
||
<div class="highlight-shell-session notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>python<span class="w"> </span>listdir-test.py
|
||
<span class="go">[b'filename\xe4\x94\x80abc', ...]</span>
|
||
<span class="go">['filename\u4500abc', ...]</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>第一个列表包含 UTF-8 编码的文件名，第二个列表则包含 Unicode 版本的。</p>
|
||
<p>请注意，大多时候应该坚持用这些 API 处理 Unicode。字节串 API 应该仅用于可能存在不可解码文件名的系统；现在几乎仅剩 Unix 系统了。</p>
|
||
</section>
|
||
<section id="tips-for-writing-unicode-aware-programs">
|
||
<h3>识别 Unicode 的编程技巧<a class="headerlink" href="#tips-for-writing-unicode-aware-programs" title="永久链接至标题">¶</a></h3>
|
||
<p>本节提供了一些关于编写 Unicode 处理软件的建议。</p>
|
||
<p>最重要的技巧如下：</p>
|
||
<blockquote>
|
||
<div><p>程序应只在内部处理 Unicode 字符串，尽快对输入数据进行解码，并只在最后对输出进行编码。</p>
|
||
</div></blockquote>
|
||
<p>如果尝试编写的处理函数对 Unicode 和字节串形式的字符串都能接受，就会发现组合使用两种不同类型的字符串时，容易产生差错。没办法做到自动编码或解码：如果执行 <code class="docutils literal notranslate"><span class="pre">str</span> <span class="pre">+</span> <span class="pre">bytes</span></code>，则会触发 <a class="reference internal" href="../library/exceptions.html#TypeError" title="TypeError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">TypeError</span></code></a>。</p>
|
||
<p>当要使用的数据来自 Web 浏览器或其他不受信来源时，常用技术是在用该字符串生成命令行之前，或要存入数据库之前，先检查字符串中是否包含非法字符。请仔细检查解码后的字符串，而不是编码格式的字节串数据；有些编码可能具备一些有趣的特性，例如与 ASCII 不是一一对应或不完全兼容。如果输入数据还指定了编码格式，则尤其如此，因为攻击者可以选择一种巧妙的方式将恶意文本隐藏在经过编码的字节流中。</p>
|
||
<section id="converting-between-file-encodings">
|
||
<h4>在文件编码格式之间进行转换<a class="headerlink" href="#converting-between-file-encodings" title="永久链接至标题">¶</a></h4>
|
||
<p><a class="reference internal" href="../library/codecs.html#codecs.StreamRecoder" title="codecs.StreamRecoder"><code class="xref py py-class docutils literal notranslate"><span class="pre">StreamRecoder</span></code></a> 类可以在两种编码之间透明地进行转换，参数为编码格式为 #1 的数据流，表现行为则是编码格式为 #2 的数据流。</p>
|
||
<p>假设输入文件 <em>f</em> 采用 Latin-1 编码格式，即可用 <a class="reference internal" href="../library/codecs.html#codecs.StreamRecoder" title="codecs.StreamRecoder"><code class="xref py py-class docutils literal notranslate"><span class="pre">StreamRecoder</span></code></a> 包装后返回 UTF-8 编码的字节串：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="n">new_f</span> <span class="o">=</span> <span class="n">codecs</span><span class="o">.</span><span class="n">StreamRecoder</span><span class="p">(</span><span class="n">f</span><span class="p">,</span>
|
||
<span class="c1"># en/decoder: used by read() to encode its results and</span>
|
||
<span class="c1"># by write() to decode its input.</span>
|
||
<span class="n">codecs</span><span class="o">.</span><span class="n">getencoder</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">),</span> <span class="n">codecs</span><span class="o">.</span><span class="n">getdecoder</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">),</span>
|
||
|
||
<span class="c1"># reader/writer: used to read and write to the stream.</span>
|
||
<span class="n">codecs</span><span class="o">.</span><span class="n">getreader</span><span class="p">(</span><span class="s1">'latin-1'</span><span class="p">),</span> <span class="n">codecs</span><span class="o">.</span><span class="n">getwriter</span><span class="p">(</span><span class="s1">'latin-1'</span><span class="p">)</span> <span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
</section>
|
||
<section id="files-in-an-unknown-encoding">
|
||
<h4>编码格式未知的文件<a class="headerlink" href="#files-in-an-unknown-encoding" title="永久链接至标题">¶</a></h4>
|
||
<p>若需对文件进行修改，但不知道文件的编码，那该怎么办呢？如果已知编码格式与 ASCII 兼容，并且只想查看或修改 ASCII 部分，则可利用 <code class="docutils literal notranslate"><span class="pre">surrogateescape</span></code> 错误处理 handler 打开文件：</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">fname</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">"ascii"</span><span class="p">,</span> <span class="n">errors</span><span class="o">=</span><span class="s2">"surrogateescape"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
|
||
<span class="n">data</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
|
||
|
||
<span class="c1"># make changes to the string 'data'</span>
|
||
|
||
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">fname</span> <span class="o">+</span> <span class="s1">'.new'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">,</span>
|
||
<span class="n">encoding</span><span class="o">=</span><span class="s2">"ascii"</span><span class="p">,</span> <span class="n">errors</span><span class="o">=</span><span class="s2">"surrogateescape"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
|
||
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
<p><code class="docutils literal notranslate"><span class="pre">surrogateescape</span></code> 错误处理 handler 会把所有非 ASCII 字节解码为 U+DC80 至 U+DCFF 这一特殊范围的码位。当 <code class="docutils literal notranslate"><span class="pre">surrogateescape</span></code> 错误处理 handler用于数据编码并回写时，这些码位将转换回原样。</p>
|
||
</section>
|
||
</section>
|
||
<section id="id3">
|
||
<h3>参考文献<a class="headerlink" href="#id3" title="永久链接至标题">¶</a></h3>
|
||
<p>David Beazley 在 PyCon 2010 上的演讲 <a class="reference external" href="http://pyvideo.org/video/289/pycon-2010--mastering-python-3-io">掌握 Python 3 输入/输出</a> 中，有一节讨论了文本和二进制数据的处理。</p>
|
||
<p><a class="reference external" href="https://downloads.egenix.com/python/LSM2005-Developing-Unicode-aware-applications-in-Python.pdf">Marc-André Lemburg 演示的PDF 幻灯片“在 Python 中编写支持 Unicode 的应用程序”</a> ，讨论了字符编码问题以及如何国际化和本地化应用程序。这些幻灯片仅涵盖 Python 2.x。</p>
|
||
<p><a class="reference external" href="http://pyvideo.org/video/1768/the-guts-of-unicode-in-python">Python Unicode 实质</a> 是 Benjamin Peterson 在 PyCon 2013 上的演讲，讨论了 Unicode 在 Python 3.3 中的内部表示。</p>
|
||
</section>
|
||
</section>
|
||
<section id="acknowledgements">
|
||
<h2>致谢<a class="headerlink" href="#acknowledgements" title="永久链接至标题">¶</a></h2>
|
||
<p>本文初稿由 Andrew Kuchling 撰写。此后，Alexander Belopolsky、Georg Brandl、Andrew Kuchling 和 Ezio Melotti 作了进一步修订。</p>
|
||
<p>感谢以下各位指出本文错误或提出建议：Éric Araujo、Nicholas Bastin、Nick Coghlan、Marius Gedminas、Kent Johnson、Ken Krugler、Marc-André Lemburg、Martin von Löwis、Terry J. Reedy、Serhiy Storchaka , Eryk Sun, Chad Whitacre, Graham Wideman。</p>
|
||
</section>
|
||
</section>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
|
||
<div class="sphinxsidebarwrapper">
|
||
<h3><a href="../contents.html">目录</a></h3>
|
||
<ul>
|
||
<li><a class="reference internal" href="#">Unicode 指南</a><ul>
|
||
<li><a class="reference internal" href="#introduction-to-unicode">Unicode 概述</a><ul>
|
||
<li><a class="reference internal" href="#definitions">定义</a></li>
|
||
<li><a class="reference internal" href="#encodings">编码</a></li>
|
||
<li><a class="reference internal" href="#references">参考文献</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#python-s-unicode-support">Python对Unicode的支持</a><ul>
|
||
<li><a class="reference internal" href="#the-string-type">字符串类型</a></li>
|
||
<li><a class="reference internal" href="#converting-to-bytes">转换为字节</a></li>
|
||
<li><a class="reference internal" href="#unicode-literals-in-python-source-code">Python 源代码中的 Unicode 文字</a></li>
|
||
<li><a class="reference internal" href="#unicode-properties">Unicode属性</a></li>
|
||
<li><a class="reference internal" href="#comparing-strings">字符串比较</a></li>
|
||
<li><a class="reference internal" href="#unicode-regular-expressions">Unicode 正则表达式</a></li>
|
||
<li><a class="reference internal" href="#id2">参考文献</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#reading-and-writing-unicode-data">Unicode 数据的读写</a><ul>
|
||
<li><a class="reference internal" href="#unicode-filenames">Unicode 文件名</a></li>
|
||
<li><a class="reference internal" href="#tips-for-writing-unicode-aware-programs">识别 Unicode 的编程技巧</a><ul>
|
||
<li><a class="reference internal" href="#converting-between-file-encodings">在文件编码格式之间进行转换</a></li>
|
||
<li><a class="reference internal" href="#files-in-an-unknown-encoding">编码格式未知的文件</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#id3">参考文献</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#acknowledgements">致谢</a></li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
|
||
<h4>上一个主题</h4>
|
||
<p class="topless"><a href="sorting.html"
|
||
title="上一章">排序指南</a></p>
|
||
<h4>下一个主题</h4>
|
||
<p class="topless"><a href="urllib2.html"
|
||
title="下一章">如何利用 urllib 包获取网络资源</a></p>
|
||
<div role="note" aria-label="source link">
|
||
<h3>本页</h3>
|
||
<ul class="this-page-menu">
|
||
<li><a href="../bugs.html">报告 Bug</a></li>
|
||
<li>
|
||
<a href="https://github.com/python/cpython/blob/3.8/Doc/howto/unicode.rst"
|
||
rel="nofollow">显示源代码
|
||
</a>
|
||
</li>
|
||
</ul>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="clearer"></div>
|
||
</div>
|
||
<div class="related" role="navigation" aria-label="related navigation">
|
||
<h3>导航</h3>
|
||
<ul>
|
||
<li class="right" style="margin-right: 10px">
|
||
<a href="../genindex.html" title="总目录"
|
||
>索引</a></li>
|
||
<li class="right" >
|
||
<a href="../py-modindex.html" title="Python 模块索引"
|
||
>模块</a> |</li>
|
||
<li class="right" >
|
||
<a href="urllib2.html" title="如何利用 urllib 包获取网络资源"
|
||
>下一页</a> |</li>
|
||
<li class="right" >
|
||
<a href="sorting.html" title="排序指南"
|
||
>上一页</a> |</li>
|
||
|
||
<li><img src="../_static/py.svg" alt="python logo" style="vertical-align: middle; margin-top: -1px"/></li>
|
||
<li><a href="https://www.python.org/">Python</a> »</li>
|
||
<li class="switchers">
|
||
<div class="language_switcher_placeholder"></div>
|
||
<div class="version_switcher_placeholder"></div>
|
||
</li>
|
||
<li>
|
||
|
||
</li>
|
||
<li id="cpython-language-and-version">
|
||
<a href="../index.html">3.8.20 Documentation</a> »
|
||
</li>
|
||
|
||
<li class="nav-item nav-item-1"><a href="index.html" >Python 指南</a> »</li>
|
||
<li class="right">
|
||
|
||
|
||
<div class="inline-search" role="search">
|
||
<form class="inline-search" action="../search.html" method="get">
|
||
<input placeholder="快速搜索" aria-label="快速搜索" type="text" name="q" />
|
||
<input type="submit" value="转向" />
|
||
<input type="hidden" name="check_keywords" value="yes" />
|
||
<input type="hidden" name="area" value="default" />
|
||
</form>
|
||
</div>
|
||
|
|
||
</li>
|
||
|
||
</ul>
|
||
</div>
|
||
<div class="footer">
|
||
© <a href="../copyright.html">版权所有</a> 2001-2024, Python Software Foundation.
|
||
<br />
|
||
This page is licensed under the Python Software Foundation License Version 2.
|
||
<br />
|
||
Examples, recipes, and other code in the documentation are additionally licensed under the Zero Clause BSD License.
|
||
<br />
|
||
|
||
<br />
|
||
|
||
The Python Software Foundation is a non-profit corporation.
|
||
<a href="https://www.python.org/psf/donations/">Please donate.</a>
|
||
<br />
|
||
<br />
|
||
|
||
最后更新于 12月 09, 2024.
|
||
<a href="https://docs.python.org/3/bugs.html">Found a bug</a>?
|
||
<br />
|
||
|
||
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 2.4.4.
|
||
</div>
|
||
|
||
</body>
|
||
</html> |