Files
hustoj/web/python-docs/howto/urllib2.html
2025-03-09 17:10:56 +08:00

713 lines
73 KiB
HTML
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="zh_CN">
<head>
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<title>如何利用 urllib 包获取网络资源 &#8212; Python 3.8.20 文档</title><meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="stylesheet" href="../_static/pydoctheme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/language_data.js"></script>
<script src="../_static/translations.js"></script>
<script src="../_static/sidebar.js"></script>
<link rel="search" type="application/opensearchdescription+xml"
title="在 Python 3.8.20 文档 中搜索"
href="../_static/opensearch.xml"/>
<link rel="author" title="关于这些文档" href="../about.html" />
<link rel="index" title="索引" href="../genindex.html" />
<link rel="search" title="搜索" href="../search.html" />
<link rel="copyright" title="版权所有" href="../copyright.html" />
<link rel="next" title="argparse 教程" href="argparse.html" />
<link rel="prev" title="Unicode 指南" href="unicode.html" />
<link rel="canonical" href="https://docs.python.org/3/howto/urllib2.html" />
<style>
@media only screen {
table.full-width-table {
width: 100%;
}
}
</style>
<link rel="shortcut icon" type="image/png" href="../_static/py.svg" />
<script type="text/javascript" src="../_static/copybutton.js"></script>
<script type="text/javascript" src="../_static/menu.js"></script>
</head>
<body>
<div class="mobile-nav">
<input type="checkbox" id="menuToggler" class="toggler__input" aria-controls="navigation"
aria-pressed="false" aria-expanded="false" role="button" aria-label="Menu" />
<label for="menuToggler" class="toggler__label">
<span></span>
</label>
<nav class="nav-content" role="navigation">
<a href="https://www.python.org/" class="nav-logo">
<img src="../_static/py.svg" alt="Logo"/>
</a>
<div class="version_switcher_placeholder"></div>
<form role="search" class="search" action="../search.html" method="get">
<svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" class="search-icon">
<path fill-rule="nonzero"
d="M15.5 14h-.79l-.28-.27a6.5 6.5 0 001.48-5.34c-.47-2.78-2.79-5-5.59-5.34a6.505 6.505 0 00-7.27 7.27c.34 2.8 2.56 5.12 5.34 5.59a6.5 6.5 0 005.34-1.48l.27.28v.79l4.25 4.25c.41.41 1.08.41 1.49 0 .41-.41.41-1.08 0-1.49L15.5 14zm-6 0C7.01 14 5 11.99 5 9.5S7.01 5 9.5 5 14 7.01 14 9.5 11.99 14 9.5 14z" fill="#444"></path>
</svg>
<input type="text" name="q" aria-label="快速搜索"/>
<input type="submit" value="转向"/>
</form>
</nav>
<div class="menu-wrapper">
<nav class="menu" role="navigation" aria-label="main navigation">
<div class="language_switcher_placeholder"></div>
<h3><a href="../contents.html">目录</a></h3>
<ul>
<li><a class="reference internal" href="#">如何利用 urllib 包获取网络资源</a><ul>
<li><a class="reference internal" href="#introduction">概述</a></li>
<li><a class="reference internal" href="#fetching-urls">获取 URL 资源</a><ul>
<li><a class="reference internal" href="#data">数据</a></li>
<li><a class="reference internal" href="#headers">HTTP 头部信息</a></li>
</ul>
</li>
<li><a class="reference internal" href="#handling-exceptions">异常的处理</a><ul>
<li><a class="reference internal" href="#urlerror">URLError</a></li>
<li><a class="reference internal" href="#httperror">HTTPError</a><ul>
<li><a class="reference internal" href="#error-codes">错误代码</a></li>
</ul>
</li>
<li><a class="reference internal" href="#wrapping-it-up">总之</a><ul>
<li><a class="reference internal" href="#number-1">第一种方案</a></li>
<li><a class="reference internal" href="#number-2">第二种方案</a></li>
</ul>
</li>
</ul>
</li>
<li><a class="reference internal" href="#info-and-geturl">info 和 geturl 方法</a></li>
<li><a class="reference internal" href="#openers-and-handlers">Opener 和 Handler</a></li>
<li><a class="reference internal" href="#id5">基本认证</a></li>
<li><a class="reference internal" href="#proxies">代理</a></li>
<li><a class="reference internal" href="#sockets-and-layers">套接字与分层</a></li>
<li><a class="reference internal" href="#footnotes">备注</a></li>
</ul>
</li>
</ul>
<h4>上一个主题</h4>
<p class="topless"><a href="unicode.html"
title="上一章">Unicode 指南</a></p>
<h4>下一个主题</h4>
<p class="topless"><a href="argparse.html"
title="下一章">argparse 教程</a></p>
<div role="note" aria-label="source link">
<h3>本页</h3>
<ul class="this-page-menu">
<li><a href="../bugs.html">报告 Bug</a></li>
<li>
<a href="https://github.com/python/cpython/blob/3.8/Doc/howto/urllib2.rst"
rel="nofollow">显示源代码
</a>
</li>
</ul>
</div>
</nav>
</div>
</div>
<div id="outdated-warning" style="padding: .5em; text-align: center; background-color: #FFBABA; color: #6A0E0E;">
这个文档所针对的是一个已不再受支持的 Python 旧版本。
你应当升级版本,并阅读
<a href="/3/howto/urllib2.html"> Python 当前稳定版本的文档</a>.
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>导航</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="总目录"
accesskey="I">索引</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python 模块索引"
>模块</a> |</li>
<li class="right" >
<a href="argparse.html" title="argparse 教程"
accesskey="N">下一页</a> |</li>
<li class="right" >
<a href="unicode.html" title="Unicode 指南"
accesskey="P">上一页</a> |</li>
<li><img src="../_static/py.svg" alt="python logo" style="vertical-align: middle; margin-top: -1px"/></li>
<li><a href="https://www.python.org/">Python</a> &#187;</li>
<li class="switchers">
<div class="language_switcher_placeholder"></div>
<div class="version_switcher_placeholder"></div>
</li>
<li>
</li>
<li id="cpython-language-and-version">
<a href="../index.html">3.8.20 Documentation</a> &#187;
</li>
<li class="nav-item nav-item-1"><a href="index.html" accesskey="U">Python 指南</a> &#187;</li>
<li class="right">
<div class="inline-search" role="search">
<form class="inline-search" action="../search.html" method="get">
<input placeholder="快速搜索" aria-label="快速搜索" type="text" name="q" />
<input type="submit" value="转向" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
|
</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="howto-fetch-internet-resources-using-the-urllib-package">
<span id="urllib-howto"></span><h1>如何利用 urllib 包获取网络资源<a class="headerlink" href="#howto-fetch-internet-resources-using-the-urllib-package" title="永久链接至标题"></a></h1>
<dl class="field-list simple">
<dt class="field-odd">作者</dt>
<dd class="field-odd"><p><a class="reference external" href="http://www.voidspace.org.uk/python/index.shtml">Michael Foord</a></p>
</dd>
</dl>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>这份 HOWTO 文档的早期版本有一份法语的译文,可在 <a class="reference external" href="http://www.voidspace.org.uk/python/articles/urllib2_francais.shtml">urllib2 - Le Manuel manquant</a> 处查阅。</p>
</div>
<section id="introduction">
<h2>概述<a class="headerlink" href="#introduction" title="永久链接至标题"></a></h2>
<aside class="sidebar">
<p class="sidebar-title">Related Articles</p>
<p>关于如何用 Python 获取 web 资源,以下文章或许也很有用:</p>
<ul>
<li><p><a class="reference external" href="http://www.voidspace.org.uk/python/articles/authentication.shtml">基础的身份认证</a></p>
<blockquote>
<div><p><em>基本认证</em> 的教程,带有一些 Python 示例。</p>
</div></blockquote>
</li>
</ul>
</aside>
<p><strong>urllib.request</strong> 是用于获取 URL (统一资源定位符)的 Python 模块。它以 <em>urlopen</em> 函数的形式提供了一个非常简单的接口,能用不同的协议获取 URL。同时它还为处理各种常见情形提供了一个稍微复杂一些的接口——比如基础身份认证、cookies、代理等等。这些功能是由名为 handlers 和 opener 的对象提供的。</p>
<p>urllib.request 支持多种 &quot;URL 方案&quot; (通过 URL中 <code class="docutils literal notranslate"><span class="pre">&quot;:&quot;</span></code> 之前的字符串加以区分——如 <code class="docutils literal notranslate"><span class="pre">&quot;ftp://python.org/&quot;</span></code> 中的 <code class="docutils literal notranslate"><span class="pre">&quot;ftp&quot;</span></code>即为采用其关联网络协议FTP、HTTP 之类)的 URL 方案 。本教程重点关注最常用的 HTTP 场景。</p>
<p>对于简单场景而言, <em>urlopen</em> 用起来十分容易。但只要在打开 HTTP URL 时遇到错误或非常情况,就需要对超文本传输协议有所了解才行。最全面、最权威的 HTTP 参考是 <span class="target" id="index-8"></span><a class="rfc reference external" href="https://tools.ietf.org/html/rfc2616.html"><strong>RFC 2616</strong></a> 。那是一份技术文档,并没有追求可读性。本 文旨在说明 <em>urllib</em> 的用法,为了便于阅读也附带了足够详细的 HTTP 信息。本文并不是为了替代 <a class="reference internal" href="../library/urllib.request.html#module-urllib.request" title="urllib.request: Extensible library for opening URLs."><code class="xref py py-mod docutils literal notranslate"><span class="pre">urllib.request</span></code></a> 文档,只是其补充说明而已。</p>
</section>
<section id="fetching-urls">
<h2>获取 URL 资源<a class="headerlink" href="#fetching-urls" title="永久链接至标题"></a></h2>
<p>urllib.request 最简单的使用方式如下所示:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">urllib.request</span>
<span class="k">with</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s1">&#39;http://python.org/&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
<span class="n">html</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</pre></div>
</div>
<p>如果想通过 URL 获取资源并临时存储一下,可以采用 <a class="reference internal" href="../library/shutil.html#shutil.copyfileobj" title="shutil.copyfileobj"><code class="xref py py-func docutils literal notranslate"><span class="pre">shutil.copyfileobj()</span></code></a><a class="reference internal" href="../library/tempfile.html#tempfile.NamedTemporaryFile" title="tempfile.NamedTemporaryFile"><code class="xref py py-func docutils literal notranslate"><span class="pre">tempfile.NamedTemporaryFile()</span></code></a> 函数:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">shutil</span>
<span class="kn">import</span> <span class="nn">tempfile</span>
<span class="kn">import</span> <span class="nn">urllib.request</span>
<span class="k">with</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s1">&#39;http://python.org/&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
<span class="k">with</span> <span class="n">tempfile</span><span class="o">.</span><span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="n">delete</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> <span class="k">as</span> <span class="n">tmp_file</span><span class="p">:</span>
<span class="n">shutil</span><span class="o">.</span><span class="n">copyfileobj</span><span class="p">(</span><span class="n">response</span><span class="p">,</span> <span class="n">tmp_file</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">tmp_file</span><span class="o">.</span><span class="n">name</span><span class="p">)</span> <span class="k">as</span> <span class="n">html</span><span class="p">:</span>
<span class="k">pass</span>
</pre></div>
</div>
<p>urllib 的很多用法就是这么简单(注意 URL 不仅可以 http: 开头,还可以是 ftp: 、file: 等)。不过本教程的目的是介绍更加复杂的应用场景,重点还是关注 HTTP。</p>
<p>HTTP 以请求和响应为基础——客户端生成请求服务器发送响应。urllib.request 用 <code class="docutils literal notranslate"><span class="pre">Request</span></code> 对象来表示要生成的 HTTP 请求。最简单的形式就是创建一个 Request 对象,指定了想要获取的 URL。用这个 Request 对象作为参数调用 <code class="docutils literal notranslate"><span class="pre">urlopen</span></code> ,将会返回该 URL 的响应对象。响应对象类似于文件对象,就是说可以对其调用 <code class="docutils literal notranslate"><span class="pre">.read()</span></code> 之类的命令:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">urllib.request</span>
<span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="s1">&#39;http://www.voidspace.org.uk&#39;</span><span class="p">)</span>
<span class="k">with</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
<span class="n">the_page</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</pre></div>
</div>
<p>请注意urllib.request 用同一个 Request 接口处理所有 URL 方案。比如可生成 FTP 请求如下:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="s1">&#39;ftp://example.com/&#39;</span><span class="p">)</span>
</pre></div>
</div>
<p>就 HTTP 而言Request 对象能够做两件额外的事情:首先可以把数据传给服务器。其次,可以将 <em>有关</em> 数据或请求本身的额外信息metadata传给服务器——这些信息将会作为 HTTP “头部”数据发送。下面依次看下。</p>
<section id="data">
<h3>数据<a class="headerlink" href="#data" title="永久链接至标题"></a></h3>
<p>有时需要向某个 URL 发送数据,通常此 URL 会指向某个CGI通用网关接口脚本或其他 web 应用。对于 HTTP 而言,这通常会用所谓的 <strong>POST</strong> 请求来完成。当要把 Web 页填写的 HTML 表单提交时,浏览器通常会执行此操作。但并不是所有的 POST 都来自表单:可以用 POST 方式传输任何数据到自己的应用上。对于通常的 HTML 表单,数据需要以标准的方式编码,然后作为 <code class="docutils literal notranslate"><span class="pre">data</span></code> 参数传给 Request 对象。编码过程是用 <a class="reference internal" href="../library/urllib.parse.html#module-urllib.parse" title="urllib.parse: Parse URLs into or assemble them from components."><code class="xref py py-mod docutils literal notranslate"><span class="pre">urllib.parse</span></code></a> 库的函数完成的:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">urllib.parse</span>
<span class="kn">import</span> <span class="nn">urllib.request</span>
<span class="n">url</span> <span class="o">=</span> <span class="s1">&#39;http://www.someserver.com/cgi-bin/register.cgi&#39;</span>
<span class="n">values</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;name&#39;</span> <span class="p">:</span> <span class="s1">&#39;Michael Foord&#39;</span><span class="p">,</span>
<span class="s1">&#39;location&#39;</span> <span class="p">:</span> <span class="s1">&#39;Northampton&#39;</span><span class="p">,</span>
<span class="s1">&#39;language&#39;</span> <span class="p">:</span> <span class="s1">&#39;Python&#39;</span> <span class="p">}</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">parse</span><span class="o">.</span><span class="n">urlencode</span><span class="p">(</span><span class="n">values</span><span class="p">)</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">&#39;ascii&#39;</span><span class="p">)</span> <span class="c1"># data should be bytes</span>
<span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="p">)</span>
<span class="k">with</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
<span class="n">the_page</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</pre></div>
</div>
<p>请注意,有时还需要采用其他编码,比如由 HTML 表单上传文件——更多细节请参见 <a class="reference external" href="https://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13">HTML 规范,提交表单</a></p>
<p>如果不传递 <code class="docutils literal notranslate"><span class="pre">data</span></code> 参数urllib 将采用 <strong>GET</strong> 请求。GET 和 POST 请求有一点不同POST 请求往往具有“副作用”,他们会以某种方式改变系统的状态。例如,从网站下一个订单,购买一大堆罐装垃圾并运送到家。 尽管 HTTP 标准明确指出 POST <em>总是</em> 要导致副作用,而 GET 请求 <em>从来不会</em> 导致副作用。但没有什么办法能阻止 GET 和 POST 请求的副作用。数据也可以在 HTTP GET 请求中传递,只要把数据编码到 URL 中即可。</p>
<p>做法如下所示:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span> <span class="nn">urllib.request</span>
<span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span> <span class="nn">urllib.parse</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">data</span> <span class="o">=</span> <span class="p">{}</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">data</span><span class="p">[</span><span class="s1">&#39;name&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="s1">&#39;Somebody Here&#39;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">data</span><span class="p">[</span><span class="s1">&#39;location&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="s1">&#39;Northampton&#39;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">data</span><span class="p">[</span><span class="s1">&#39;language&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="s1">&#39;Python&#39;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">url_values</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">parse</span><span class="o">.</span><span class="n">urlencode</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="nb">print</span><span class="p">(</span><span class="n">url_values</span><span class="p">)</span> <span class="c1"># The order may differ from below. </span>
<span class="go">name=Somebody+Here&amp;language=Python&amp;location=Northampton</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">url</span> <span class="o">=</span> <span class="s1">&#39;http://www.example.com/example.cgi&#39;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">full_url</span> <span class="o">=</span> <span class="n">url</span> <span class="o">+</span> <span class="s1">&#39;?&#39;</span> <span class="o">+</span> <span class="n">url_values</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">data</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">full_url</span><span class="p">)</span>
</pre></div>
</div>
<p>请注意,完整的 URL 是通过在其中添加 <code class="docutils literal notranslate"><span class="pre">?</span></code> 创建的,后面跟着经过编码的数据。</p>
</section>
<section id="headers">
<h3>HTTP 头部信息<a class="headerlink" href="#headers" title="永久链接至标题"></a></h3>
<p>下面介绍一个具体的 HTTP 头部信息,以此说明如何在 HTTP 请求加入头部信息。</p>
<p>有些网站 <a class="footnote-reference brackets" href="#id8" id="id1">1</a> 不愿被程序浏览到,或者要向不同的浏览器发送不同版本 <a class="footnote-reference brackets" href="#id9" id="id2">2</a> 的网页。默认情况下urllib 将自身标识为“Python-urllib/xy”其中 <code class="docutils literal notranslate"><span class="pre">x</span></code><code class="docutils literal notranslate"><span class="pre">y</span></code> 是 Python 版本的主、次版本号,例如 <code class="docutils literal notranslate"><span class="pre">Python-urllib/2.5</span></code>),这可能会让网站不知所措,或者干脆就使其无法正常工作。浏览器是通过头部信息 <code class="docutils literal notranslate"><span class="pre">User-Agent</span></code> <a class="footnote-reference brackets" href="#id10" id="id3">3</a> 来标识自己的。在创建 Request 对象时,可以传入字典形式的头部信息。以下示例将生成与之前相同的请求,只是将自身标识为某个版本的 Internet Explorer <a class="footnote-reference brackets" href="#id11" id="id4">4</a> </p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">urllib.parse</span>
<span class="kn">import</span> <span class="nn">urllib.request</span>
<span class="n">url</span> <span class="o">=</span> <span class="s1">&#39;http://www.someserver.com/cgi-bin/register.cgi&#39;</span>
<span class="n">user_agent</span> <span class="o">=</span> <span class="s1">&#39;Mozilla/5.0 (Windows NT 6.1; Win64; x64)&#39;</span>
<span class="n">values</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;name&#39;</span><span class="p">:</span> <span class="s1">&#39;Michael Foord&#39;</span><span class="p">,</span>
<span class="s1">&#39;location&#39;</span><span class="p">:</span> <span class="s1">&#39;Northampton&#39;</span><span class="p">,</span>
<span class="s1">&#39;language&#39;</span><span class="p">:</span> <span class="s1">&#39;Python&#39;</span> <span class="p">}</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;User-Agent&#39;</span><span class="p">:</span> <span class="n">user_agent</span><span class="p">}</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">parse</span><span class="o">.</span><span class="n">urlencode</span><span class="p">(</span><span class="n">values</span><span class="p">)</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">&#39;ascii&#39;</span><span class="p">)</span>
<span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="p">,</span> <span class="n">headers</span><span class="p">)</span>
<span class="k">with</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
<span class="n">the_page</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</pre></div>
</div>
<p>响应对象也有两个很有用的方法。请参阅有关 <a class="reference internal" href="#info-and-geturl">info 和 geturl</a> 部分,了解出现问题时会发生什么。</p>
</section>
</section>
<section id="handling-exceptions">
<h2>异常的处理<a class="headerlink" href="#handling-exceptions" title="永久链接至标题"></a></h2>
<p>如果 <em>urlopen</em> 无法处理响应信息,就会触发 <code class="xref py py-exc docutils literal notranslate"><span class="pre">URLError</span></code> 。尽管与通常的 Python API 一样,也可能触发 <a class="reference internal" href="../library/exceptions.html#ValueError" title="ValueError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">ValueError</span></code></a><a class="reference internal" href="../library/exceptions.html#TypeError" title="TypeError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">TypeError</span></code></a> 等内置异常。</p>
<p><code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code><code class="xref py py-exc docutils literal notranslate"><span class="pre">URLError</span></code> 的子类,当 URL 是 HTTP 的情况时将会触发。</p>
<p>上述异常类是从 <a class="reference internal" href="../library/urllib.error.html#module-urllib.error" title="urllib.error: Exception classes raised by urllib.request."><code class="xref py py-mod docutils literal notranslate"><span class="pre">urllib.error</span></code></a> 模块中导出的。</p>
<section id="urlerror">
<h3>URLError<a class="headerlink" href="#urlerror" title="永久链接至标题"></a></h3>
<p>触发 URLError 的原因,通常是网络不通(或者没有到指定服务器的路由),或者指定的服务器不存在。这时触发的异常会带有一个 reason 属性,是一个包含错误代码和文本错误信息的元组。</p>
<p>例如:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="s1">&#39;http://www.pretend_server.org&#39;</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="k">try</span><span class="p">:</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
<span class="gp">... </span><span class="k">except</span> <span class="n">urllib</span><span class="o">.</span><span class="n">error</span><span class="o">.</span><span class="n">URLError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
<span class="gp">...</span>
<span class="go">(4, &#39;getaddrinfo failed&#39;)</span>
</pre></div>
</div>
</section>
<section id="httperror">
<h3>HTTPError<a class="headerlink" href="#httperror" title="永久链接至标题"></a></h3>
<p>从服务器返回的每个 HTTP 响应都包含一个数字的 “状态码”。有时该状态码表明服务器无法完成该请求。默认的处理函数将会处理这其中的一部分响应。如若响应是“redirection”这是要求客户端从另一 URL 处获取数据urllib 将会自行处理。对于那些无法处理的状况urlopen 将会引发 <code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code> 。典型的错误包括“404”页面无法找到、“403”请求遭拒绝和“401”需要身份认证</p>
<p>全部的 HTTP 错误码请参阅 <span class="target" id="index-9"></span><a class="rfc reference external" href="https://tools.ietf.org/html/rfc2616.html"><strong>RFC 2616</strong></a> 。</p>
<p><code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code> 实例将包含一个整数型的“code”属性对应于服务器发来的错误。</p>
<section id="error-codes">
<h4>错误代码<a class="headerlink" href="#error-codes" title="永久链接至标题"></a></h4>
<p>由于默认处理函数会自行处理重定向300 以内的错误码),而且 100--299 的状态码表示成功,因此通常只会出现 400--599 的错误码。</p>
<p><a class="reference internal" href="../library/http.server.html#http.server.BaseHTTPRequestHandler.responses" title="http.server.BaseHTTPRequestHandler.responses"><code class="xref py py-attr docutils literal notranslate"><span class="pre">http.server.BaseHTTPRequestHandler.responses</span></code></a> 是很有用的响应码字典,其中给出了 <span class="target" id="index-10"></span><a class="rfc reference external" href="https://tools.ietf.org/html/rfc2616.html"><strong>RFC 2616</strong></a> 用到的所有响应代码。为方便起见,将此字典转载如下:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Table mapping response codes to messages; entries have the</span>
<span class="c1"># form {code: (shortmessage, longmessage)}.</span>
<span class="n">responses</span> <span class="o">=</span> <span class="p">{</span>
<span class="mi">100</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Continue&#39;</span><span class="p">,</span> <span class="s1">&#39;Request received, please continue&#39;</span><span class="p">),</span>
<span class="mi">101</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Switching Protocols&#39;</span><span class="p">,</span>
<span class="s1">&#39;Switching to new protocol; obey Upgrade header&#39;</span><span class="p">),</span>
<span class="mi">200</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;OK&#39;</span><span class="p">,</span> <span class="s1">&#39;Request fulfilled, document follows&#39;</span><span class="p">),</span>
<span class="mi">201</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Created&#39;</span><span class="p">,</span> <span class="s1">&#39;Document created, URL follows&#39;</span><span class="p">),</span>
<span class="mi">202</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Accepted&#39;</span><span class="p">,</span>
<span class="s1">&#39;Request accepted, processing continues off-line&#39;</span><span class="p">),</span>
<span class="mi">203</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Non-Authoritative Information&#39;</span><span class="p">,</span> <span class="s1">&#39;Request fulfilled from cache&#39;</span><span class="p">),</span>
<span class="mi">204</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;No Content&#39;</span><span class="p">,</span> <span class="s1">&#39;Request fulfilled, nothing follows&#39;</span><span class="p">),</span>
<span class="mi">205</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Reset Content&#39;</span><span class="p">,</span> <span class="s1">&#39;Clear input form for further input.&#39;</span><span class="p">),</span>
<span class="mi">206</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Partial Content&#39;</span><span class="p">,</span> <span class="s1">&#39;Partial content follows.&#39;</span><span class="p">),</span>
<span class="mi">300</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Multiple Choices&#39;</span><span class="p">,</span>
<span class="s1">&#39;Object has several resources -- see URI list&#39;</span><span class="p">),</span>
<span class="mi">301</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Moved Permanently&#39;</span><span class="p">,</span> <span class="s1">&#39;Object moved permanently -- see URI list&#39;</span><span class="p">),</span>
<span class="mi">302</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Found&#39;</span><span class="p">,</span> <span class="s1">&#39;Object moved temporarily -- see URI list&#39;</span><span class="p">),</span>
<span class="mi">303</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;See Other&#39;</span><span class="p">,</span> <span class="s1">&#39;Object moved -- see Method and URL list&#39;</span><span class="p">),</span>
<span class="mi">304</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Not Modified&#39;</span><span class="p">,</span>
<span class="s1">&#39;Document has not changed since given time&#39;</span><span class="p">),</span>
<span class="mi">305</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Use Proxy&#39;</span><span class="p">,</span>
<span class="s1">&#39;You must use proxy specified in Location to access this &#39;</span>
<span class="s1">&#39;resource.&#39;</span><span class="p">),</span>
<span class="mi">307</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Temporary Redirect&#39;</span><span class="p">,</span>
<span class="s1">&#39;Object moved temporarily -- see URI list&#39;</span><span class="p">),</span>
<span class="mi">400</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Bad Request&#39;</span><span class="p">,</span>
<span class="s1">&#39;Bad request syntax or unsupported method&#39;</span><span class="p">),</span>
<span class="mi">401</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Unauthorized&#39;</span><span class="p">,</span>
<span class="s1">&#39;No permission -- see authorization schemes&#39;</span><span class="p">),</span>
<span class="mi">402</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Payment Required&#39;</span><span class="p">,</span>
<span class="s1">&#39;No payment -- see charging schemes&#39;</span><span class="p">),</span>
<span class="mi">403</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Forbidden&#39;</span><span class="p">,</span>
<span class="s1">&#39;Request forbidden -- authorization will not help&#39;</span><span class="p">),</span>
<span class="mi">404</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Not Found&#39;</span><span class="p">,</span> <span class="s1">&#39;Nothing matches the given URI&#39;</span><span class="p">),</span>
<span class="mi">405</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Method Not Allowed&#39;</span><span class="p">,</span>
<span class="s1">&#39;Specified method is invalid for this server.&#39;</span><span class="p">),</span>
<span class="mi">406</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Not Acceptable&#39;</span><span class="p">,</span> <span class="s1">&#39;URI not available in preferred format.&#39;</span><span class="p">),</span>
<span class="mi">407</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Proxy Authentication Required&#39;</span><span class="p">,</span> <span class="s1">&#39;You must authenticate with &#39;</span>
<span class="s1">&#39;this proxy before proceeding.&#39;</span><span class="p">),</span>
<span class="mi">408</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Request Timeout&#39;</span><span class="p">,</span> <span class="s1">&#39;Request timed out; try again later.&#39;</span><span class="p">),</span>
<span class="mi">409</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Conflict&#39;</span><span class="p">,</span> <span class="s1">&#39;Request conflict.&#39;</span><span class="p">),</span>
<span class="mi">410</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Gone&#39;</span><span class="p">,</span>
<span class="s1">&#39;URI no longer exists and has been permanently removed.&#39;</span><span class="p">),</span>
<span class="mi">411</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Length Required&#39;</span><span class="p">,</span> <span class="s1">&#39;Client must specify Content-Length.&#39;</span><span class="p">),</span>
<span class="mi">412</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Precondition Failed&#39;</span><span class="p">,</span> <span class="s1">&#39;Precondition in headers is false.&#39;</span><span class="p">),</span>
<span class="mi">413</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Request Entity Too Large&#39;</span><span class="p">,</span> <span class="s1">&#39;Entity is too large.&#39;</span><span class="p">),</span>
<span class="mi">414</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Request-URI Too Long&#39;</span><span class="p">,</span> <span class="s1">&#39;URI is too long.&#39;</span><span class="p">),</span>
<span class="mi">415</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Unsupported Media Type&#39;</span><span class="p">,</span> <span class="s1">&#39;Entity body in unsupported format.&#39;</span><span class="p">),</span>
<span class="mi">416</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Requested Range Not Satisfiable&#39;</span><span class="p">,</span>
<span class="s1">&#39;Cannot satisfy request range.&#39;</span><span class="p">),</span>
<span class="mi">417</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Expectation Failed&#39;</span><span class="p">,</span>
<span class="s1">&#39;Expect condition could not be satisfied.&#39;</span><span class="p">),</span>
<span class="mi">500</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Internal Server Error&#39;</span><span class="p">,</span> <span class="s1">&#39;Server got itself in trouble&#39;</span><span class="p">),</span>
<span class="mi">501</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Not Implemented&#39;</span><span class="p">,</span>
<span class="s1">&#39;Server does not support this operation&#39;</span><span class="p">),</span>
<span class="mi">502</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Bad Gateway&#39;</span><span class="p">,</span> <span class="s1">&#39;Invalid responses from another server/proxy.&#39;</span><span class="p">),</span>
<span class="mi">503</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Service Unavailable&#39;</span><span class="p">,</span>
<span class="s1">&#39;The server cannot process the request due to a high load&#39;</span><span class="p">),</span>
<span class="mi">504</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;Gateway Timeout&#39;</span><span class="p">,</span>
<span class="s1">&#39;The gateway server did not receive a timely response&#39;</span><span class="p">),</span>
<span class="mi">505</span><span class="p">:</span> <span class="p">(</span><span class="s1">&#39;HTTP Version Not Supported&#39;</span><span class="p">,</span> <span class="s1">&#39;Cannot fulfill request.&#39;</span><span class="p">),</span>
<span class="p">}</span>
</pre></div>
</div>
<p>当触发错误时,服务器通过返回 HTTP 错误码 <em></em> 错误页面进行响应。可以将 <code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code> 实例用作返回页面的响应。这意味着除了 code 属性之外,错误对象还像 <code class="docutils literal notranslate"><span class="pre">urllib.response</span></code> 模块返回的那样具有 read、geturl 和 info 方法:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="s1">&#39;http://www.python.org/fish.html&#39;</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="k">try</span><span class="p">:</span>
<span class="gp">... </span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
<span class="gp">... </span><span class="k">except</span> <span class="n">urllib</span><span class="o">.</span><span class="n">error</span><span class="o">.</span><span class="n">HTTPError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">code</span><span class="p">)</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">read</span><span class="p">())</span>
<span class="gp">...</span>
<span class="go">404</span>
<span class="go">b&#39;&lt;!DOCTYPE html PUBLIC &quot;-//W3C//DTD XHTML 1.0 Transitional//EN&quot;</span>
<span class="go"> &quot;http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd&quot;&gt;\n\n\n&lt;html</span>
<span class="go"> ...</span>
<span class="go"> &lt;title&gt;Page Not Found&lt;/title&gt;\n</span>
<span class="go"> ...</span>
</pre></div>
</div>
</section>
</section>
<section id="wrapping-it-up">
<h3>总之<a class="headerlink" href="#wrapping-it-up" title="永久链接至标题"></a></h3>
<p>若要准备处理 <code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code>  <em></em> <code class="xref py py-exc docutils literal notranslate"><span class="pre">URLError</span></code> ,有两种简单的方案。推荐使用第二种方案。</p>
<section id="number-1">
<h4>第一种方案<a class="headerlink" href="#number-1" title="永久链接至标题"></a></h4>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">urllib.request</span> <span class="kn">import</span> <span class="n">Request</span><span class="p">,</span> <span class="n">urlopen</span>
<span class="kn">from</span> <span class="nn">urllib.error</span> <span class="kn">import</span> <span class="n">URLError</span><span class="p">,</span> <span class="n">HTTPError</span>
<span class="n">req</span> <span class="o">=</span> <span class="n">Request</span><span class="p">(</span><span class="n">someurl</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
<span class="k">except</span> <span class="n">HTTPError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;The server couldn</span><span class="se">\&#39;</span><span class="s1">t fulfill the request.&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Error code: &#39;</span><span class="p">,</span> <span class="n">e</span><span class="o">.</span><span class="n">code</span><span class="p">)</span>
<span class="k">except</span> <span class="n">URLError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;We failed to reach a server.&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Reason: &#39;</span><span class="p">,</span> <span class="n">e</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="c1"># everything is fine</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p><code class="docutils literal notranslate"><span class="pre">except</span> <span class="pre">HTTPError</span></code> <em>必须</em> 首先处理,否则 <code class="docutils literal notranslate"><span class="pre">except</span> <span class="pre">URLError</span></code> 将会 <em>同时</em> 捕获 <code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code></p>
</div>
</section>
<section id="number-2">
<h4>第二种方案<a class="headerlink" href="#number-2" title="永久链接至标题"></a></h4>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">urllib.request</span> <span class="kn">import</span> <span class="n">Request</span><span class="p">,</span> <span class="n">urlopen</span>
<span class="kn">from</span> <span class="nn">urllib.error</span> <span class="kn">import</span> <span class="n">URLError</span>
<span class="n">req</span> <span class="o">=</span> <span class="n">Request</span><span class="p">(</span><span class="n">someurl</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
<span class="k">except</span> <span class="n">URLError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">e</span><span class="p">,</span> <span class="s1">&#39;reason&#39;</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;We failed to reach a server.&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Reason: &#39;</span><span class="p">,</span> <span class="n">e</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
<span class="k">elif</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">e</span><span class="p">,</span> <span class="s1">&#39;code&#39;</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;The server couldn</span><span class="se">\&#39;</span><span class="s1">t fulfill the request.&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Error code: &#39;</span><span class="p">,</span> <span class="n">e</span><span class="o">.</span><span class="n">code</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="c1"># everything is fine</span>
</pre></div>
</div>
</section>
</section>
</section>
<section id="info-and-geturl">
<h2>info 和 geturl 方法<a class="headerlink" href="#info-and-geturl" title="永久链接至标题"></a></h2>
<p>由 urlopen (或者 <code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code> 实例)所返回的响应包含两个有用的方法: <code class="xref py py-meth docutils literal notranslate"><span class="pre">info()</span></code><code class="xref py py-meth docutils literal notranslate"><span class="pre">geturl()</span></code>,该响应由模块 <a class="reference internal" href="../library/urllib.request.html#module-urllib.response" title="urllib.response: Response classes used by urllib."><code class="xref py py-mod docutils literal notranslate"><span class="pre">urllib.response</span></code></a> 定义。</p>
<p><strong>geturl</strong> ——返回所获取页面的真实 URL。该方法很有用因为 <code class="docutils literal notranslate"><span class="pre">urlopen</span></code> (或 opener 对象)可能已经经过了一次重定向。已获取页面的 URL 未必就是所请求的 URL 。</p>
<p><strong>info</strong> - 该方法返回一个类似字典的对象描述了所获取的页面特别是由服务器送出的头部信息headers 。目前它是一个 <code class="xref py py-class docutils literal notranslate"><span class="pre">http.client.HTTPMessage</span></code> 实例。</p>
<p>典型的 HTTP 头部信息包括“Content-length”、“Content-type”等。有关 HTTP 头部信息的清单,包括含义和用途的简要说明,请参阅 <a class="reference external" href="http://jkorpela.fi/http.html">HTTP Header 快速参考</a></p>
</section>
<section id="openers-and-handlers">
<h2>Opener 和 Handler<a class="headerlink" href="#openers-and-handlers" title="永久链接至标题"></a></h2>
<p>当获取 URL 时,会用到了一个 opener一个类名可能经过混淆的 <a class="reference internal" href="../library/urllib.request.html#urllib.request.OpenerDirector" title="urllib.request.OpenerDirector"><code class="xref py py-class docutils literal notranslate"><span class="pre">urllib.request.OpenerDirector</span></code></a> 的实例)。通常一直会用默认的 opener ——通过 <code class="docutils literal notranslate"><span class="pre">urlopen</span></code> ——但也可以创建自定义的 opener 。opener 会用到 handler。所有的“繁重工作”都由 handler 完成。每种 handler 知道某种 URL 方案http、ftp 等)的 URL 的打开方式,或是某方面 URL 的打开方式,例如 HTTP 重定向或 HTTP cookie。</p>
<p>若要用已安装的某个 handler 获取 URL需要创建一个 opener 对象,例如处理 cookie 的 opener或对重定向不做处理的 opener。</p>
<p>若要创建 opener请实例化一个 <code class="docutils literal notranslate"><span class="pre">OpenerDirector</span></code> ,然后重复调用 <code class="docutils literal notranslate"><span class="pre">.add_handler(some_handler_instance)</span></code></p>
<p>或者也可以用 <code class="docutils literal notranslate"><span class="pre">build_opener</span></code> ,这是个用单次调用创建 opener 对象的便捷函数。<code class="docutils literal notranslate"><span class="pre">build_opener</span></code> 默认会添加几个 handler不过还提供了一种快速添加和/或覆盖默认 handler 的方法。</p>
<p>可能还需要其他类型的 handler以便处理代理、身份认证和其他常见但稍微特殊的情况。</p>
<p><code class="docutils literal notranslate"><span class="pre">install_opener</span></code> 可用于让 <code class="docutils literal notranslate"><span class="pre">opener</span></code> 对象成为(全局)默认 opener。这意味着调用 <code class="docutils literal notranslate"><span class="pre">urlopen</span></code> 时会采用已安装的 opener。</p>
<p>opener 对象带有一个 <code class="docutils literal notranslate"><span class="pre">`open</span></code> 方法,可供直接调用以获取 url方式与 <code class="docutils literal notranslate"><span class="pre">urlopen</span></code> 函数相同。除非是为了调用方便,否则没必要去调用 <code class="docutils literal notranslate"><span class="pre">install_opener</span></code></p>
</section>
<section id="id5">
<h2>基本认证<a class="headerlink" href="#id5" title="永久链接至标题"></a></h2>
<p>为了说明 handler 的创建和安装过程,会用到 <code class="docutils literal notranslate"><span class="pre">HTTPBasicAuthHandler</span></code> 。有关该主题的更详细的介绍——包括基本身份认证的工作原理——请参阅 <a class="reference external" href="http://www.voidspace.org.uk/python/articles/authentication.shtml">Basic Authentication Tutorial</a></p>
<p>如果需要身份认证,服务器会发送一条请求身份认证的头部信息(以及 401 错误代码。这条信息中指明了身份认证方式和“安全区域realm”。格式如下所示<code class="docutils literal notranslate"><span class="pre">WWW-Authenticate:</span> <span class="pre">SCHEME</span> <span class="pre">realm=&quot;REALM&quot;</span></code></p>
<p>例如</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>WWW-Authenticate: Basic realm=&quot;cPanel Users&quot;
</pre></div>
</div>
<p>然后,客户端应重试发起请求,请求数据中的头部信息应包含安全区域对应的用户名和密码。这就是“基本身份认证”。为了简化此过程,可以创建 <code class="docutils literal notranslate"><span class="pre">HTTPBasicAuthHandler</span></code> 的一个实例及使用它的 opener。</p>
<p><code class="docutils literal notranslate"><span class="pre">HTTPBasicAuthHandler</span></code> 用一个名为密码管理器的对象来管理 URL、安全区域与密码、用户名之间的映射关系。如果知道确切的安全区域来自服务器发送的身份认证头部信息那就可以用到 <code class="docutils literal notranslate"><span class="pre">HTTPPasswordMgr</span></code> 。通常人们并不关心安全区域是什么,这时用 <code class="docutils literal notranslate"><span class="pre">HTTPPasswordMgrWithDefaultRealm</span></code> 就很方便,允许为 URL 指定默认的用户名和密码。当没有为某个安全区域提供用户名和密码时,就会用到默认值。下面用 <code class="docutils literal notranslate"><span class="pre">None</span></code> 作为 <code class="docutils literal notranslate"><span class="pre">add_password</span></code> 方法的安全区域参数,表明采用默认用户名和密码。</p>
<p>首先需要身份认证的是顶级 URL。比传给 .add_password() 的 URL 级别“更深”的 URL 也会得以匹配:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># create a password manager</span>
<span class="n">password_mgr</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">HTTPPasswordMgrWithDefaultRealm</span><span class="p">()</span>
<span class="c1"># Add the username and password.</span>
<span class="c1"># If we knew the realm, we could use it instead of None.</span>
<span class="n">top_level_url</span> <span class="o">=</span> <span class="s2">&quot;http://example.com/foo/&quot;</span>
<span class="n">password_mgr</span><span class="o">.</span><span class="n">add_password</span><span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="n">top_level_url</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span>
<span class="n">handler</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">HTTPBasicAuthHandler</span><span class="p">(</span><span class="n">password_mgr</span><span class="p">)</span>
<span class="c1"># create &quot;opener&quot; (OpenerDirector instance)</span>
<span class="n">opener</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">build_opener</span><span class="p">(</span><span class="n">handler</span><span class="p">)</span>
<span class="c1"># use the opener to fetch a URL</span>
<span class="n">opener</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">a_url</span><span class="p">)</span>
<span class="c1"># Install the opener.</span>
<span class="c1"># Now all calls to urllib.request.urlopen use our opener.</span>
<span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">install_opener</span><span class="p">(</span><span class="n">opener</span><span class="p">)</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>在以上例子中,只向 <code class="docutils literal notranslate"><span class="pre">build_opener</span></code> 给出了 <code class="docutils literal notranslate"><span class="pre">HTTPBasicAuthHandler</span></code> 。默认情况下opener 会有用于处理常见状况的 handler ——<code class="docutils literal notranslate"><span class="pre">ProxyHandler</span></code> (如果设置代理的话,比如设置了环境变量 <span class="target" id="index-11"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">http_proxy</span></code> <code class="docutils literal notranslate"><span class="pre">UnknownHandler</span></code><code class="docutils literal notranslate"><span class="pre">HTTPHandler</span></code><code class="docutils literal notranslate"><span class="pre">HTTPDefaultErrorHandler</span></code><code class="docutils literal notranslate"><span class="pre">HTTPRedirectHandler</span></code><code class="docutils literal notranslate"><span class="pre">FTPHandler</span></code><code class="docutils literal notranslate"><span class="pre">FileHandler</span></code><code class="docutils literal notranslate"><span class="pre">DataHandler</span></code><code class="docutils literal notranslate"><span class="pre">HTTPErrorProcessor</span></code></p>
</div>
<p><code class="docutils literal notranslate"><span class="pre">top_level_url</span></code> 其实 <em>要么</em> 是一条完整的 URL包括 “http:” 部分和主机名及可选的端口号),比如 <code class="docutils literal notranslate"><span class="pre">&quot;http://example.com/&quot;</span></code> <em>要么</em> 是一条“访问权限”(即主机名,及可选的端口号),比如 <code class="docutils literal notranslate"><span class="pre">&quot;example.com&quot;</span></code><code class="docutils literal notranslate"><span class="pre">&quot;example.com:8080&quot;</span></code> (后一个示例包含了端口号)。访问权限 <strong>不得</strong> 包含“用户信息”部分——比如 <code class="docutils literal notranslate"><span class="pre">&quot;joe:password&#64;example.com&quot;</span></code> 就不正确。</p>
</section>
<section id="proxies">
<h2>代理<a class="headerlink" href="#proxies" title="永久链接至标题"></a></h2>
<p><strong>urllib</strong> 将自动检测并使用代理设置。 这是通过 <code class="docutils literal notranslate"><span class="pre">ProxyHandler</span></code> 实现的,当检测到代理设置时,是正常 handler 链中的一部分。通常这是一件好事,但有时也可能会无效 <a class="footnote-reference brackets" href="#id12" id="id6">5</a>。 一种方案是配置自己的 <code class="docutils literal notranslate"><span class="pre">ProxyHandler</span></code> ,不要定义代理。 设置的步骤与 <a class="reference internal" href="#basic-authentication">Basic Authentication</a> handler 类似:</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">proxy_support</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">ProxyHandler</span><span class="p">({})</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">opener</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">build_opener</span><span class="p">(</span><span class="n">proxy_support</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">install_opener</span><span class="p">(</span><span class="n">opener</span><span class="p">)</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>目前 <code class="docutils literal notranslate"><span class="pre">urllib.request</span></code> <em>尚不</em> 支持通过代理抓取 <code class="docutils literal notranslate"><span class="pre">https</span></code> 链接地址。 但此功能可以通过扩展 urllib.request 来启用,如以下例程所示 <a class="footnote-reference brackets" href="#id13" id="id7">6</a></p>
</div>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>如果设置了 <code class="docutils literal notranslate"><span class="pre">REQUEST_METHOD</span></code> 变量,则会忽略 <code class="docutils literal notranslate"><span class="pre">HTTP_PROXY</span></code> ;参阅 <a class="reference internal" href="../library/urllib.request.html#urllib.request.getproxies" title="urllib.request.getproxies"><code class="xref py py-func docutils literal notranslate"><span class="pre">getproxies()</span></code></a> 文档。</p>
</div>
</section>
<section id="sockets-and-layers">
<h2>套接字与分层<a class="headerlink" href="#sockets-and-layers" title="永久链接至标题"></a></h2>
<p>Python 获取 Web 资源的能力是分层的。urllib 用到的是 <a class="reference internal" href="../library/http.client.html#module-http.client" title="http.client: HTTP and HTTPS protocol client (requires sockets)."><code class="xref py py-mod docutils literal notranslate"><span class="pre">http.client</span></code></a> 库,而后者又用到了套接字库。</p>
<p>从 Python 2.3 开始,可以指定套接字等待响应的超时时间。这对必须要读到网页数据的应用程序会很有用。默认情况下,套接字模块 <em>不会超时</em> 并且可以挂起。目前,套接字超时机制未暴露给 http.client 或 urllib.request 层使用。不过可以为所有套接字应用设置默认的全局超时。</p>
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">socket</span>
<span class="kn">import</span> <span class="nn">urllib.request</span>
<span class="c1"># timeout in seconds</span>
<span class="n">timeout</span> <span class="o">=</span> <span class="mi">10</span>
<span class="n">socket</span><span class="o">.</span><span class="n">setdefaulttimeout</span><span class="p">(</span><span class="n">timeout</span><span class="p">)</span>
<span class="c1"># this call to urllib.request.urlopen now uses the default timeout</span>
<span class="c1"># we have set in the socket module</span>
<span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="s1">&#39;http://www.voidspace.org.uk&#39;</span><span class="p">)</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
</pre></div>
</div>
</section>
<hr class="docutils" />
<section id="footnotes">
<h2>备注<a class="headerlink" href="#footnotes" title="永久链接至标题"></a></h2>
<p>这篇文档由 John Lee 审订。</p>
<dl class="footnote brackets">
<dt class="label" id="id8"><span class="brackets"><a class="fn-backref" href="#id1">1</a></span></dt>
<dd><p>例如 Google。</p>
</dd>
<dt class="label" id="id9"><span class="brackets"><a class="fn-backref" href="#id2">2</a></span></dt>
<dd><p>对于网站设计而言,探测不同的浏览器是非常糟糕的做法——更为明智的做法是采用 web 标准构建网站。不幸的是,很多网站依然向不同的浏览器发送不同版本的网页。</p>
</dd>
<dt class="label" id="id10"><span class="brackets"><a class="fn-backref" href="#id3">3</a></span></dt>
<dd><p>MSIE 6 的 user-agent 信息是 <em>“Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)”</em></p>
</dd>
<dt class="label" id="id11"><span class="brackets"><a class="fn-backref" href="#id4">4</a></span></dt>
<dd><p>有关 HTTP 请求的头部信息,详情请参阅 <a class="reference internal" href="#quick-reference-to-http-headers">Quick Reference to HTTP Headers</a></p>
</dd>
<dt class="label" id="id12"><span class="brackets"><a class="fn-backref" href="#id6">5</a></span></dt>
<dd><p>本人必须使用代理才能在工作中访问互联网。如果尝试通过代理获取 <em>localhost</em> URL将会遭到阻止。IE 设置为代理模式urllib 就会获取到配置信息。为了用 localhost 服务器测试脚本,我必须阻止 urllib 使用代理。</p>
</dd>
<dt class="label" id="id13"><span class="brackets"><a class="fn-backref" href="#id7">6</a></span></dt>
<dd><p>urllib 的 SSL 代理 openerCONNECT  方法): <a class="reference external" href="https://code.activestate.com/recipes/456195/">ASPN Cookbook Recipe</a></p>
</dd>
</dl>
</section>
</section>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="../contents.html">目录</a></h3>
<ul>
<li><a class="reference internal" href="#">如何利用 urllib 包获取网络资源</a><ul>
<li><a class="reference internal" href="#introduction">概述</a></li>
<li><a class="reference internal" href="#fetching-urls">获取 URL 资源</a><ul>
<li><a class="reference internal" href="#data">数据</a></li>
<li><a class="reference internal" href="#headers">HTTP 头部信息</a></li>
</ul>
</li>
<li><a class="reference internal" href="#handling-exceptions">异常的处理</a><ul>
<li><a class="reference internal" href="#urlerror">URLError</a></li>
<li><a class="reference internal" href="#httperror">HTTPError</a><ul>
<li><a class="reference internal" href="#error-codes">错误代码</a></li>
</ul>
</li>
<li><a class="reference internal" href="#wrapping-it-up">总之</a><ul>
<li><a class="reference internal" href="#number-1">第一种方案</a></li>
<li><a class="reference internal" href="#number-2">第二种方案</a></li>
</ul>
</li>
</ul>
</li>
<li><a class="reference internal" href="#info-and-geturl">info 和 geturl 方法</a></li>
<li><a class="reference internal" href="#openers-and-handlers">Opener 和 Handler</a></li>
<li><a class="reference internal" href="#id5">基本认证</a></li>
<li><a class="reference internal" href="#proxies">代理</a></li>
<li><a class="reference internal" href="#sockets-and-layers">套接字与分层</a></li>
<li><a class="reference internal" href="#footnotes">备注</a></li>
</ul>
</li>
</ul>
<h4>上一个主题</h4>
<p class="topless"><a href="unicode.html"
title="上一章">Unicode 指南</a></p>
<h4>下一个主题</h4>
<p class="topless"><a href="argparse.html"
title="下一章">argparse 教程</a></p>
<div role="note" aria-label="source link">
<h3>本页</h3>
<ul class="this-page-menu">
<li><a href="../bugs.html">报告 Bug</a></li>
<li>
<a href="https://github.com/python/cpython/blob/3.8/Doc/howto/urllib2.rst"
rel="nofollow">显示源代码
</a>
</li>
</ul>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>导航</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="总目录"
>索引</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python 模块索引"
>模块</a> |</li>
<li class="right" >
<a href="argparse.html" title="argparse 教程"
>下一页</a> |</li>
<li class="right" >
<a href="unicode.html" title="Unicode 指南"
>上一页</a> |</li>
<li><img src="../_static/py.svg" alt="python logo" style="vertical-align: middle; margin-top: -1px"/></li>
<li><a href="https://www.python.org/">Python</a> &#187;</li>
<li class="switchers">
<div class="language_switcher_placeholder"></div>
<div class="version_switcher_placeholder"></div>
</li>
<li>
</li>
<li id="cpython-language-and-version">
<a href="../index.html">3.8.20 Documentation</a> &#187;
</li>
<li class="nav-item nav-item-1"><a href="index.html" >Python 指南</a> &#187;</li>
<li class="right">
<div class="inline-search" role="search">
<form class="inline-search" action="../search.html" method="get">
<input placeholder="快速搜索" aria-label="快速搜索" type="text" name="q" />
<input type="submit" value="转向" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
|
</li>
</ul>
</div>
<div class="footer">
&copy; <a href="../copyright.html">版权所有</a> 2001-2024, Python Software Foundation.
<br />
This page is licensed under the Python Software Foundation License Version 2.
<br />
Examples, recipes, and other code in the documentation are additionally licensed under the Zero Clause BSD License.
<br />
<br />
The Python Software Foundation is a non-profit corporation.
<a href="https://www.python.org/psf/donations/">Please donate.</a>
<br />
<br />
最后更新于 12月 09, 2024.
<a href="https://docs.python.org/3/bugs.html">Found a bug</a>?
<br />
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 2.4.4.
</div>
</body>
</html>