urllib.request.urlopen raise an exception when URL contains credentials

[mauricelambert]

Bug report

Bug description:

When using urllib.request.urlopen() with a URL that contains credentials in the form of http://user:password@host/, the credentials are not properly removed from the host field when making the request. This results in errors during the connection attempt, as the hostname contains the credentials.

Bug

>>> from urllib.request import urlopen
>>> urlopen("http://test:test@example.com/")
Traceback (most recent call last):
  File "/usr/lib/python3.10/http/client.py", line 890, in _get_hostport
    port = int(host[i+1:])
ValueError: invalid literal for int() with base 10: 'test@example.com'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 1377, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib/python3.10/urllib/request.py", line 1317, in do_open
    h = http_class(host, timeout=req.timeout, **http_conn_args)
  File "/usr/lib/python3.10/http/client.py", line 852, in __init__
    (self.host, self.port) = self._get_hostport(host, port)
  File "/usr/lib/python3.10/http/client.py", line 895, in _get_hostport
    raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
http.client.InvalidURL: nonnumeric port: 'test@example.com'
>>> urlopen("http://test:test@example.com:80/")
Traceback (most recent call last):
  File "/usr/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/lib/python3.10/http/client.py", line 942, in connect
    self.sock = self._create_connection(
  File "/usr/lib/python3.10/socket.py", line 824, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 1377, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib/python3.10/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>
>>> 

CPython versions tested on:

3.10

Operating systems tested on:

Linux

Linked PRs

• gh-78809: Fix urlopen crash using credentials in URL #140919

[picnixz]

This is a duplicate of #78809 actually.

[mauricelambert]

@picnixz the issue seems to have been open since 2018, shouldn’t we restart the discussion on the topic ?
In the meantime, while waiting for an implementation, if it’s going to take a while, shouldn’t we add a raise NotImplementedError ? I find that the error message isn’t very clear, and the source of the problem is hard to identify at the moment.

[picnixz]

the issue seems to have been open since 2018, shouldn’t we restart the discussion on the topic ?

Yes, but such discussion can happen on https://discuss.python.org/c/ideas/6. Should we reject it? do it automatically? silently fail? we also don't want duplicate issues even if they are old.

In the meantime, while waiting for an implementation, if it’s going to take a while, shouldn’t we add a raise NotImplementedError

That's not the policy we have as this would change whatever try-catch: ValueError that people may have.

I find that the error message isn’t very clear, and the source of the problem is hard to identify at the moment.

I agree but please discuss it on the old issue.