klarkxy/hustoj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2f18fac0c4e21792d75056e9634d994b0ab2d5f3
hustoj/web/python-docs/library/tokenize.html
klarkxy 723430cade 增加python文档
2025-03-09 17:10:56 +08:00

493 lines
38 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="zh_CN">
<head>
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<title>tokenize --- 对 Python 代码使用的标记解析器 &#8212; Python 3.8.20 文档</title><meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="stylesheet" href="../_static/pydoctheme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/language_data.js"></script>
<script src="../_static/translations.js"></script>
<script src="../_static/sidebar.js"></script>
<link rel="search" type="application/opensearchdescription+xml"
title="在 Python 3.8.20 文档 中搜索"
href="../_static/opensearch.xml"/>
<link rel="author" title="关于这些文档" href="../about.html" />
<link rel="index" title="索引" href="../genindex.html" />
<link rel="search" title="搜索" href="../search.html" />
<link rel="copyright" title="版权所有" href="../copyright.html" />
<link rel="next" title="tabnanny --- 模糊缩进检测" href="tabnanny.html" />
<link rel="prev" title="keyword --- 检验Python关键字" href="keyword.html" />
<link rel="canonical" href="https://docs.python.org/3/library/tokenize.html" />
<style>
@media only screen {
table.full-width-table {
width: 100%;
}
}
</style>
<link rel="shortcut icon" type="image/png" href="../_static/py.svg" />
<script type="text/javascript" src="../_static/copybutton.js"></script>
<script type="text/javascript" src="../_static/menu.js"></script>
</head>
<body>
<div class="mobile-nav">
<input type="checkbox" id="menuToggler" class="toggler__input" aria-controls="navigation"
aria-pressed="false" aria-expanded="false" role="button" aria-label="Menu" />
<label for="menuToggler" class="toggler__label">
<span></span>
</label>
<nav class="nav-content" role="navigation">
<a href="https://www.python.org/" class="nav-logo">
<img src="../_static/py.svg" alt="Logo"/>
</a>
<div class="version_switcher_placeholder"></div>
<form role="search" class="search" action="../search.html" method="get">
<svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" class="search-icon">
<path fill-rule="nonzero"
d="M15.5 14h-.79l-.28-.27a6.5 6.5 0 001.48-5.34c-.47-2.78-2.79-5-5.59-5.34a6.505 6.505 0 00-7.27 7.27c.34 2.8 2.56 5.12 5.34 5.59a6.5 6.5 0 005.34-1.48l.27.28v.79l4.25 4.25c.41.41 1.08.41 1.49 0 .41-.41.41-1.08 0-1.49L15.5 14zm-6 0C7.01 14 5 11.99 5 9.5S7.01 5 9.5 5 14 7.01 14 9.5 11.99 14 9.5 14z" fill="#444"></path>
</svg>
<input type="text" name="q" aria-label="快速搜索"/>
<input type="submit" value="转向"/>
</form>
</nav>
<div class="menu-wrapper">
<nav class="menu" role="navigation" aria-label="main navigation">
<div class="language_switcher_placeholder"></div>
<h3><a href="../contents.html">目录</a></h3>
<ul>
<li><a class="reference internal" href="#"><code class="xref py py-mod docutils literal notranslate"><span class="pre">tokenize</span></code> --- 对 Python 代码使用的标记解析器</a><ul>
<li><a class="reference internal" href="#tokenizing-input">对输入进行解析标记</a></li>
<li><a class="reference internal" href="#command-line-usage">命令行用法</a></li>
<li><a class="reference internal" href="#examples">例子</a></li>
</ul>
</li>
</ul>
<h4>上一个主题</h4>
<p class="topless"><a href="keyword.html"
title="上一章"><code class="xref py py-mod docutils literal notranslate"><span class="pre">keyword</span></code> --- 检验Python关键字</a></p>
<h4>下一个主题</h4>
<p class="topless"><a href="tabnanny.html"
title="下一章"><code class="xref py py-mod docutils literal notranslate"><span class="pre">tabnanny</span></code> --- 模糊缩进检测</a></p>
<div role="note" aria-label="source link">
<h3>本页</h3>
<ul class="this-page-menu">
<li><a href="../bugs.html">报告 Bug</a></li>
<li>
<a href="https://github.com/python/cpython/blob/3.8/Doc/library/tokenize.rst"
rel="nofollow">显示源代码
</a>
</li>
</ul>
</div>
</nav>
</div>
</div>
<div id="outdated-warning" style="padding: .5em; text-align: center; background-color: #FFBABA; color: #6A0E0E;">
这个文档所针对的是一个已不再受支持的 Python 旧版本。
你应当升级版本，并阅读
<a href="/3/library/tokenize.html"> Python 当前稳定版本的文档</a>.
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>导航</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="总目录"
accesskey="I">索引</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python 模块索引"
>模块</a> |</li>
<li class="right" >
<a href="tabnanny.html" title="tabnanny --- 模糊缩进检测"
accesskey="N">下一页</a> |</li>
<li class="right" >
<a href="keyword.html" title="keyword --- 检验Python关键字"
accesskey="P">上一页</a> |</li>
<li><img src="../_static/py.svg" alt="python logo" style="vertical-align: middle; margin-top: -1px"/></li>
<li><a href="https://www.python.org/">Python</a> &#187;</li>
<li class="switchers">
<div class="language_switcher_placeholder"></div>
<div class="version_switcher_placeholder"></div>
</li>
<li>
</li>
<li id="cpython-language-and-version">
<a href="../index.html">3.8.20 Documentation</a> &#187;
</li>
<li class="nav-item nav-item-1"><a href="index.html" >Python 标准库</a> &#187;</li>
<li class="nav-item nav-item-2"><a href="language.html" accesskey="U">Python 语言服务</a> &#187;</li>
<li class="right">
<div class="inline-search" role="search">
<form class="inline-search" action="../search.html" method="get">
<input placeholder="快速搜索" aria-label="快速搜索" type="text" name="q" />
<input type="submit" value="转向" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
|
</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="module-tokenize">
<span id="tokenize-tokenizer-for-python-source"></span><h1><a class="reference internal" href="#module-tokenize" title="tokenize: Lexical scanner for Python source code."><code class="xref py py-mod docutils literal notranslate"><span class="pre">tokenize</span></code></a> --- 对 Python 代码使用的标记解析器<a class="headerlink" href="#module-tokenize" title="永久链接至标题"></a></h1>
<p><strong>源码：</strong> <a class="reference external" href="https://github.com/python/cpython/tree/3.8/Lib/tokenize.py">Lib/tokenize.py</a></p>
<hr class="docutils" />
<p><a class="reference internal" href="#module-tokenize" title="tokenize: Lexical scanner for Python source code."><code class="xref py py-mod docutils literal notranslate"><span class="pre">tokenize</span></code></a> 模块为 Python 源代码提供了一个词法扫描器，用 Python 实现。该模块中的扫描器也将注释作为标记返回，这使得它对于实现“漂亮的输出器”非常有用，包括用于屏幕显示的着色器。</p>
<p>为了简化标记流的处理，所有的 <a class="reference internal" href="../reference/lexical_analysis.html#operators"><span class="std std-ref">运算符</span></a> 和 <a class="reference internal" href="../reference/lexical_analysis.html#delimiters"><span class="std std-ref">定界符</span></a> 以及 <a class="reference internal" href="constants.html#Ellipsis" title="Ellipsis"><code class="xref py py-data docutils literal notranslate"><span class="pre">Ellipsis</span></code></a> 返回时都会打上通用的 <a class="reference internal" href="token.html#token.OP" title="token.OP"><code class="xref py py-data docutils literal notranslate"><span class="pre">OP</span></code></a> 标记。 可以通过 <a class="reference internal" href="#tokenize.tokenize" title="tokenize.tokenize"><code class="xref py py-func docutils literal notranslate"><span class="pre">tokenize.tokenize()</span></code></a> 返回的 <a class="reference internal" href="../glossary.html#term-named-tuple"><span class="xref std std-term">named tuple</span></a> 对象的 <code class="docutils literal notranslate"><span class="pre">exact_type</span></code> 属性来获得确切的标记类型。</p>
<section id="tokenizing-input">
<h2>对输入进行解析标记<a class="headerlink" href="#tokenizing-input" title="永久链接至标题"></a></h2>
<p>主要的入口是一个 <a class="reference internal" href="../glossary.html#term-generator"><span class="xref std std-term">generator</span></a>:</p>
<dl class="function">
<dt id="tokenize.tokenize">
<code class="sig-prename descclassname">tokenize.</code><code class="sig-name descname">tokenize</code><span class="sig-paren">(</span><em class="sig-param">readline</em><span class="sig-paren">)</span><a class="headerlink" href="#tokenize.tokenize" title="永久链接至目标"></a></dt>
<dd><p>生成器 <a class="reference internal" href="#tokenize.tokenize" title="tokenize.tokenize"><code class="xref py py-func docutils literal notranslate"><span class="pre">tokenize()</span></code></a> 需要一个 <em>readline</em> 参数，这个参数必须是一个可调用对象，且能提供与文件对象的 <a class="reference internal" href="io.html#io.IOBase.readline" title="io.IOBase.readline"><code class="xref py py-meth docutils literal notranslate"><span class="pre">io.IOBase.readline()</span></code></a> 方法相同的接口。每次调用这个函数都要 返回字节类型输入的一行数据。</p>
<p>生成器产生 5 个具有这些成员的元组：令牌类型；令牌字符串；指定令牌在源中开始的行和列的 2 元组 <code class="docutils literal notranslate"><span class="pre">(srow,</span> <span class="pre">scol)</span></code> ；指定令牌在源中结束的行和列的 2 元组 <code class="docutils literal notranslate"><span class="pre">(erow,</span> <span class="pre">ecol)</span></code> ；以及发现令牌的行。所传递的行（最后一个元组项）是 <em>实际的</em> 行。 5 个元组以 <a class="reference internal" href="../glossary.html#term-named-tuple"><span class="xref std std-term">named tuple</span></a> 的形式返回，字段名是： <code class="docutils literal notranslate"><span class="pre">type</span> <span class="pre">string</span> <span class="pre">start</span> <span class="pre">end</span> <span class="pre">line</span></code> 。</p>
<p>返回的 <a class="reference internal" href="../glossary.html#term-named-tuple"><span class="xref std std-term">named tuple</span></a> 有一个额外的属性，名为 <code class="docutils literal notranslate"><span class="pre">exact_type</span></code> ，包含了 <a class="reference internal" href="token.html#token.OP" title="token.OP"><code class="xref py py-data docutils literal notranslate"><span class="pre">OP</span></code></a> 标记的确切操作符类型。 对于所有其他标记类型， <code class="docutils literal notranslate"><span class="pre">exact_type</span></code> 等于命名元组的 <code class="docutils literal notranslate"><span class="pre">type</span></code> 字段。</p>
<div class="versionchanged">
<p><span class="versionmodified changed">在 3.1 版更改: </span>增加了对 named tuple 的支持。</p>
</div>
<div class="versionchanged">
<p><span class="versionmodified changed">在 3.3 版更改: </span>添加了对 <code class="docutils literal notranslate"><span class="pre">exact_type</span></code> 的支持。</p>
</div>
<p>根据:pep:<cite>263</cite> ， <a class="reference internal" href="#tokenize.tokenize" title="tokenize.tokenize"><code class="xref py py-func docutils literal notranslate"><span class="pre">tokenize()</span></code></a> 通过寻找 UTF-8 BOM 或编码 cookie 来确定文件的源编码。</p>
</dd></dl>
<dl class="function">
<dt id="tokenize.generate_tokens">
<code class="sig-prename descclassname">tokenize.</code><code class="sig-name descname">generate_tokens</code><span class="sig-paren">(</span><em class="sig-param">readline</em><span class="sig-paren">)</span><a class="headerlink" href="#tokenize.generate_tokens" title="永久链接至目标"></a></dt>
<dd><p>对读取 unicode 字符串而不是字节的源进行标记。</p>
<p>和 <a class="reference internal" href="#tokenize.tokenize" title="tokenize.tokenize"><code class="xref py py-func docutils literal notranslate"><span class="pre">tokenize()</span></code></a> 一样， <em>readline</em> 参数是一个返回单行输入的可调用参数。然而， <a class="reference internal" href="#tokenize.generate_tokens" title="tokenize.generate_tokens"><code class="xref py py-func docutils literal notranslate"><span class="pre">generate_tokens()</span></code></a> 希望 <em>readline</em> 返回一个 str 对象而不是字节。</p>
<p>其结果是一个产生具名元组的的迭代器，与 <a class="reference internal" href="#tokenize.tokenize" title="tokenize.tokenize"><code class="xref py py-func docutils literal notranslate"><span class="pre">tokenize()</span></code></a> 完全一样。 它不会产生 <a class="reference internal" href="token.html#token.ENCODING" title="token.ENCODING"><code class="xref py py-data docutils literal notranslate"><span class="pre">ENCODING</span></code></a> 标记。</p>
</dd></dl>
<p>所有来自 <a class="reference internal" href="token.html#module-token" title="token: Constants representing terminal nodes of the parse tree."><code class="xref py py-mod docutils literal notranslate"><span class="pre">token</span></code></a> 模块的常量也可从 <a class="reference internal" href="#module-tokenize" title="tokenize: Lexical scanner for Python source code."><code class="xref py py-mod docutils literal notranslate"><span class="pre">tokenize</span></code></a> 导出。</p>
<p>提供了另一个函数来逆转标记化过程。这对于创建对脚本进行标记、修改标记流并写回修改后脚本的工具很有用。</p>
<dl class="function">
<dt id="tokenize.untokenize">
<code class="sig-prename descclassname">tokenize.</code><code class="sig-name descname">untokenize</code><span class="sig-paren">(</span><em class="sig-param">iterable</em><span class="sig-paren">)</span><a class="headerlink" href="#tokenize.untokenize" title="永久链接至目标"></a></dt>
<dd><p>将令牌转换为 Python 源代码。 <em>iterable</em> 必须返回至少有两个元素的序列，即令牌类型和令牌字符串。任何额外的序列元素都会被忽略。</p>
<p>重构的脚本以单个字符串的形式返回。 结果被保证为标记回与输入相匹配，因此转换是无损的，并保证来回操作。 该保证只适用于标记类型和标记字符串，因为标记之间的间距（列位置）可能会改变。</p>
<p>它返回字节，使用 <a class="reference internal" href="token.html#token.ENCODING" title="token.ENCODING"><code class="xref py py-data docutils literal notranslate"><span class="pre">ENCODING</span></code></a> 标记进行编码，这是由 <a class="reference internal" href="#tokenize.tokenize" title="tokenize.tokenize"><code class="xref py py-func docutils literal notranslate"><span class="pre">tokenize()</span></code></a> 输出的第一个标记序列。如果输入中没有编码令牌，它将返回一个字符串。</p>
</dd></dl>
<p><a class="reference internal" href="#tokenize.tokenize" title="tokenize.tokenize"><code class="xref py py-func docutils literal notranslate"><span class="pre">tokenize()</span></code></a> 需要检测它所标记源文件的编码。它用来做这件事的函数是可用的：</p>
<dl class="function">
<dt id="tokenize.detect_encoding">
<code class="sig-prename descclassname">tokenize.</code><code class="sig-name descname">detect_encoding</code><span class="sig-paren">(</span><em class="sig-param">readline</em><span class="sig-paren">)</span><a class="headerlink" href="#tokenize.detect_encoding" title="永久链接至目标"></a></dt>
<dd><p><a class="reference internal" href="#tokenize.detect_encoding" title="tokenize.detect_encoding"><code class="xref py py-func docutils literal notranslate"><span class="pre">detect_encoding()</span></code></a> 函数用于检测解码 Python 源文件时应使用的编码。它需要一个参数， readline ，与 <a class="reference internal" href="#tokenize.tokenize" title="tokenize.tokenize"><code class="xref py py-func docutils literal notranslate"><span class="pre">tokenize()</span></code></a> 生成器的使用方式相同。</p>
<p>它最多调用 readline 两次，并返回所使用的编码（作为一个字符串）和它所读入的任何行（不是从字节解码的）的 list 。</p>
<p>它从 UTF-8 BOM 或编码 cookie 的存在中检测编码格式，如 <span class="target" id="index-3"></span><a class="pep reference external" href="https://www.python.org/dev/peps/pep-0263"><strong>PEP 263</strong></a> 所指明的。 如果 BOM 和 cookie 都存在，但不一致，将会引发 <a class="reference internal" href="exceptions.html#SyntaxError" title="SyntaxError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">SyntaxError</span></code></a>。 请注意，如果找到 BOM ，将返回 <code class="docutils literal notranslate"><span class="pre">'utf-8-sig'</span></code> 作为编码格式。</p>
<p>如果没有指定编码，那么将返回默认的 <code class="docutils literal notranslate"><span class="pre">'utf-8'</span></code> 编码.</p>
<p>使用 <a class="reference internal" href="#tokenize.open" title="tokenize.open"><code class="xref py py-func docutils literal notranslate"><span class="pre">open()</span></code></a> 来打开 Python 源文件：它使用 <a class="reference internal" href="#tokenize.detect_encoding" title="tokenize.detect_encoding"><code class="xref py py-func docutils literal notranslate"><span class="pre">detect_encoding()</span></code></a> 来检测文件编码。</p>
</dd></dl>
<dl class="function">
<dt id="tokenize.open">
<code class="sig-prename descclassname">tokenize.</code><code class="sig-name descname">open</code><span class="sig-paren">(</span><em class="sig-param">filename</em><span class="sig-paren">)</span><a class="headerlink" href="#tokenize.open" title="永久链接至目标"></a></dt>
<dd><p>使用由 <a class="reference internal" href="#tokenize.detect_encoding" title="tokenize.detect_encoding"><code class="xref py py-func docutils literal notranslate"><span class="pre">detect_encoding()</span></code></a> 检测到的编码，以只读模式打开一个文件。</p>
<div class="versionadded">
<p><span class="versionmodified added">3.2 新版功能.</span></p>
</div>
</dd></dl>
<dl class="exception">
<dt id="tokenize.TokenError">
<em class="property">exception </em><code class="sig-prename descclassname">tokenize.</code><code class="sig-name descname">TokenError</code><a class="headerlink" href="#tokenize.TokenError" title="永久链接至目标"></a></dt>
<dd><p>当文件中任何地方没有完成 docstring 或可能被分割成几行的表达式时触发，例如:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="s2">&quot;&quot;&quot;Beginning of</span>
<span class="s2">docstring</span>
</pre></div>
</div>
<p>或者：</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="p">[</span><span class="mi">1</span><span class="p">,</span>
<span class="mi">2</span><span class="p">,</span>
<span class="mi">3</span>
</pre></div>
</div>
</dd></dl>
<p>注意，未封闭的单引号字符串不会导致错误发生。它们被标记为 <a class="reference internal" href="token.html#token.ERRORTOKEN" title="token.ERRORTOKEN"><code class="xref py py-data docutils literal notranslate"><span class="pre">ERRORTOKEN</span></code></a> ，然后是其内容的标记化。</p>
</section>
<section id="command-line-usage">
<span id="tokenize-cli"></span><h2>命令行用法<a class="headerlink" href="#command-line-usage" title="永久链接至标题"></a></h2>
<div class="versionadded">
<p><span class="versionmodified added">3.3 新版功能.</span></p>
</div>
<p><a class="reference internal" href="#module-tokenize" title="tokenize: Lexical scanner for Python source code."><code class="xref py py-mod docutils literal notranslate"><span class="pre">tokenize</span></code></a> 模块可以作为一个脚本从命令行执行。这很简单。</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>python<span class="w"> </span>-m<span class="w"> </span>tokenize<span class="w"> </span><span class="o">[</span>-e<span class="o">]</span><span class="w"> </span><span class="o">[</span>filename.py<span class="o">]</span>
</pre></div>
</div>
<p>可以接受以下选项：</p>
<dl class="cmdoption">
<dt id="cmdoption-tokenize-h">
<code class="sig-name descname">-h</code><code class="sig-prename descclassname"></code><code class="sig-prename descclassname">, </code><code class="sig-name descname">--help</code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-tokenize-h" title="永久链接至目标"></a></dt>
<dd><p>显示此帮助信息并退出</p>
</dd></dl>
<dl class="cmdoption">
<dt id="cmdoption-tokenize-e">
<code class="sig-name descname">-e</code><code class="sig-prename descclassname"></code><code class="sig-prename descclassname">, </code><code class="sig-name descname">--exact</code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-tokenize-e" title="永久链接至目标"></a></dt>
<dd><p>使用确切的类型显示令牌名称</p>
</dd></dl>
<p>如果 <code class="file docutils literal notranslate"><span class="pre">filename.py</span></code> 被指定，其内容会被标记到 stdout 。否则，标记化将在 stdin 上执行。</p>
</section>
<section id="examples">
<h2>例子<a class="headerlink" href="#examples" title="永久链接至标题"></a></h2>
<p>脚本改写器的例子，它将 float 文本转换为 Decimal 对象：。</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">tokenize</span> <span class="kn">import</span> <span class="n">tokenize</span><span class="p">,</span> <span class="n">untokenize</span><span class="p">,</span> <span class="n">NUMBER</span><span class="p">,</span> <span class="n">STRING</span><span class="p">,</span> <span class="n">NAME</span><span class="p">,</span> <span class="n">OP</span>
<span class="kn">from</span> <span class="nn">io</span> <span class="kn">import</span> <span class="n">BytesIO</span>
<span class="k">def</span> <span class="nf">decistmt</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Substitute Decimals for floats in a string of statements.</span>
<span class="sd"> &gt;&gt;&gt; from decimal import Decimal</span>
<span class="sd"> &gt;&gt;&gt; s = &#39;print(+21.3e-5*-.1234/81.7)&#39;</span>
<span class="sd"> &gt;&gt;&gt; decistmt(s)</span>
<span class="sd"> &quot;print (+Decimal (&#39;21.3e-5&#39;)*-Decimal (&#39;.1234&#39;)/Decimal (&#39;81.7&#39;))&quot;</span>
<span class="sd"> The format of the exponent is inherited from the platform C library.</span>
<span class="sd"> Known cases are &quot;e-007&quot; (Windows) and &quot;e-07&quot; (not Windows). Since</span>
<span class="sd"> we&#39;re only showing 12 digits, and the 13th isn&#39;t close to 5, the</span>
<span class="sd"> rest of the output should be platform-independent.</span>
<span class="sd"> &gt;&gt;&gt; exec(s) #doctest: +ELLIPSIS</span>
<span class="sd"> -3.21716034272e-0...7</span>
<span class="sd"> Output from calculations with Decimal should be identical across all</span>
<span class="sd"> platforms.</span>
<span class="sd"> &gt;&gt;&gt; exec(decistmt(s))</span>
<span class="sd"> -3.217160342717258261933904529E-7</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">tokenize</span><span class="p">(</span><span class="n">BytesIO</span><span class="p">(</span><span class="n">s</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">))</span><span class="o">.</span><span class="n">readline</span><span class="p">)</span> <span class="c1"># tokenize the string</span>
<span class="k">for</span> <span class="n">toknum</span><span class="p">,</span> <span class="n">tokval</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">_</span> <span class="ow">in</span> <span class="n">g</span><span class="p">:</span>
<span class="k">if</span> <span class="n">toknum</span> <span class="o">==</span> <span class="n">NUMBER</span> <span class="ow">and</span> <span class="s1">&#39;.&#39;</span> <span class="ow">in</span> <span class="n">tokval</span><span class="p">:</span> <span class="c1"># replace NUMBER tokens</span>
<span class="n">result</span><span class="o">.</span><span class="n">extend</span><span class="p">([</span>
<span class="p">(</span><span class="n">NAME</span><span class="p">,</span> <span class="s1">&#39;Decimal&#39;</span><span class="p">),</span>
<span class="p">(</span><span class="n">OP</span><span class="p">,</span> <span class="s1">&#39;(&#39;</span><span class="p">),</span>
<span class="p">(</span><span class="n">STRING</span><span class="p">,</span> <span class="nb">repr</span><span class="p">(</span><span class="n">tokval</span><span class="p">)),</span>
<span class="p">(</span><span class="n">OP</span><span class="p">,</span> <span class="s1">&#39;)&#39;</span><span class="p">)</span>
<span class="p">])</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">toknum</span><span class="p">,</span> <span class="n">tokval</span><span class="p">))</span>
<span class="k">return</span> <span class="n">untokenize</span><span class="p">(</span><span class="n">result</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">)</span>
</pre></div>
</div>
<p>从命令行进行标记化的例子。 脚本:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">say_hello</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Hello, World!&quot;</span><span class="p">)</span>
<span class="n">say_hello</span><span class="p">()</span>
</pre></div>
</div>
<p>将被标记为以下输出，其中第一列是发现标记的行 / 列坐标范围，第二列是标记的名称，最后一列是标记的值（如果有）。</p>
<div class="highlight-shell-session notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>python<span class="w"> </span>-m<span class="w"> </span>tokenize<span class="w"> </span>hello.py
<span class="go">0,0-0,0: ENCODING &#39;utf-8&#39;</span>
<span class="go">1,0-1,3: NAME &#39;def&#39;</span>
<span class="go">1,4-1,13: NAME &#39;say_hello&#39;</span>
<span class="go">1,13-1,14: OP &#39;(&#39;</span>
<span class="go">1,14-1,15: OP &#39;)&#39;</span>
<span class="go">1,15-1,16: OP &#39;:&#39;</span>
<span class="go">1,16-1,17: NEWLINE &#39;\n&#39;</span>
<span class="go">2,0-2,4: INDENT &#39; &#39;</span>
<span class="go">2,4-2,9: NAME &#39;print&#39;</span>
<span class="go">2,9-2,10: OP &#39;(&#39;</span>
<span class="go">2,10-2,25: STRING &#39;&quot;Hello, World!&quot;&#39;</span>
<span class="go">2,25-2,26: OP &#39;)&#39;</span>
<span class="go">2,26-2,27: NEWLINE &#39;\n&#39;</span>
<span class="go">3,0-3,1: NL &#39;\n&#39;</span>
<span class="go">4,0-4,0: DEDENT &#39;&#39;</span>
<span class="go">4,0-4,9: NAME &#39;say_hello&#39;</span>
<span class="go">4,9-4,10: OP &#39;(&#39;</span>
<span class="go">4,10-4,11: OP &#39;)&#39;</span>
<span class="go">4,11-4,12: NEWLINE &#39;\n&#39;</span>
<span class="go">5,0-5,0: ENDMARKER &#39;&#39;</span>
</pre></div>
</div>
<p>可以使用 <a class="reference internal" href="#cmdoption-tokenize-e"><code class="xref std std-option docutils literal notranslate"><span class="pre">-e</span></code></a> 选项来显示确切的标记类型名称。</p>
<div class="highlight-shell-session notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>python<span class="w"> </span>-m<span class="w"> </span>tokenize<span class="w"> </span>-e<span class="w"> </span>hello.py
<span class="go">0,0-0,0: ENCODING &#39;utf-8&#39;</span>
<span class="go">1,0-1,3: NAME &#39;def&#39;</span>
<span class="go">1,4-1,13: NAME &#39;say_hello&#39;</span>
<span class="go">1,13-1,14: LPAR &#39;(&#39;</span>
<span class="go">1,14-1,15: RPAR &#39;)&#39;</span>
<span class="go">1,15-1,16: COLON &#39;:&#39;</span>
<span class="go">1,16-1,17: NEWLINE &#39;\n&#39;</span>
<span class="go">2,0-2,4: INDENT &#39; &#39;</span>
<span class="go">2,4-2,9: NAME &#39;print&#39;</span>
<span class="go">2,9-2,10: LPAR &#39;(&#39;</span>
<span class="go">2,10-2,25: STRING &#39;&quot;Hello, World!&quot;&#39;</span>
<span class="go">2,25-2,26: RPAR &#39;)&#39;</span>
<span class="go">2,26-2,27: NEWLINE &#39;\n&#39;</span>
<span class="go">3,0-3,1: NL &#39;\n&#39;</span>
<span class="go">4,0-4,0: DEDENT &#39;&#39;</span>
<span class="go">4,0-4,9: NAME &#39;say_hello&#39;</span>
<span class="go">4,9-4,10: LPAR &#39;(&#39;</span>
<span class="go">4,10-4,11: RPAR &#39;)&#39;</span>
<span class="go">4,11-4,12: NEWLINE &#39;\n&#39;</span>
<span class="go">5,0-5,0: ENDMARKER &#39;&#39;</span>
</pre></div>
</div>
<p>以编程方式对文件进行标记的例子，用 <a class="reference internal" href="#tokenize.generate_tokens" title="tokenize.generate_tokens"><code class="xref py py-func docutils literal notranslate"><span class="pre">generate_tokens()</span></code></a> 读取 unicode 字符串而不是字节:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">tokenize</span>
<span class="k">with</span> <span class="n">tokenize</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="s1">&#39;hello.py&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">tokens</span> <span class="o">=</span> <span class="n">tokenize</span><span class="o">.</span><span class="n">generate_tokens</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">readline</span><span class="p">)</span>
<span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">tokens</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
</pre></div>
</div>
<p>或者通过 <a class="reference internal" href="#tokenize.tokenize" title="tokenize.tokenize"><code class="xref py py-func docutils literal notranslate"><span class="pre">tokenize()</span></code></a> 直接读取字节数据:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">tokenize</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">&#39;hello.py&#39;</span><span class="p">,</span> <span class="s1">&#39;rb&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">tokens</span> <span class="o">=</span> <span class="n">tokenize</span><span class="o">.</span><span class="n">tokenize</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">readline</span><span class="p">)</span>
<span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">tokens</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
</pre></div>
</div>
</section>
</section>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="../contents.html">目录</a></h3>
<ul>
<li><a class="reference internal" href="#"><code class="xref py py-mod docutils literal notranslate"><span class="pre">tokenize</span></code> --- 对 Python 代码使用的标记解析器</a><ul>
<li><a class="reference internal" href="#tokenizing-input">对输入进行解析标记</a></li>
<li><a class="reference internal" href="#command-line-usage">命令行用法</a></li>
<li><a class="reference internal" href="#examples">例子</a></li>
</ul>
</li>
</ul>
<h4>上一个主题</h4>
<p class="topless"><a href="keyword.html"
title="上一章"><code class="xref py py-mod docutils literal notranslate"><span class="pre">keyword</span></code> --- 检验Python关键字</a></p>
<h4>下一个主题</h4>
<p class="topless"><a href="tabnanny.html"
title="下一章"><code class="xref py py-mod docutils literal notranslate"><span class="pre">tabnanny</span></code> --- 模糊缩进检测</a></p>
<div role="note" aria-label="source link">
<h3>本页</h3>
<ul class="this-page-menu">
<li><a href="../bugs.html">报告 Bug</a></li>
<li>
<a href="https://github.com/python/cpython/blob/3.8/Doc/library/tokenize.rst"
rel="nofollow">显示源代码
</a>
</li>
</ul>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>导航</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="总目录"
>索引</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python 模块索引"
>模块</a> |</li>
<li class="right" >
<a href="tabnanny.html" title="tabnanny --- 模糊缩进检测"
>下一页</a> |</li>
<li class="right" >
<a href="keyword.html" title="keyword --- 检验Python关键字"
>上一页</a> |</li>
<li><img src="../_static/py.svg" alt="python logo" style="vertical-align: middle; margin-top: -1px"/></li>
<li><a href="https://www.python.org/">Python</a> &#187;</li>
<li class="switchers">
<div class="language_switcher_placeholder"></div>
<div class="version_switcher_placeholder"></div>
</li>
<li>
</li>
<li id="cpython-language-and-version">
<a href="../index.html">3.8.20 Documentation</a> &#187;
</li>
<li class="nav-item nav-item-1"><a href="index.html" >Python 标准库</a> &#187;</li>
<li class="nav-item nav-item-2"><a href="language.html" >Python 语言服务</a> &#187;</li>
<li class="right">
<div class="inline-search" role="search">
<form class="inline-search" action="../search.html" method="get">
<input placeholder="快速搜索" aria-label="快速搜索" type="text" name="q" />
<input type="submit" value="转向" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
|
</li>
</ul>
</div>
<div class="footer">
&copy; <a href="../copyright.html">版权所有</a> 2001-2024, Python Software Foundation.
<br />
This page is licensed under the Python Software Foundation License Version 2.
<br />
Examples, recipes, and other code in the documentation are additionally licensed under the Zero Clause BSD License.
<br />
<br />
The Python Software Foundation is a non-profit corporation.
<a href="https://www.python.org/psf/donations/">Please donate.</a>
<br />
<br />
最后更新于 12月 09, 2024.
<a href="https://docs.python.org/3/bugs.html">Found a bug</a>?
<br />
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 2.4.4.
</div>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink