Skip to content

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters #22219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vstinner merged 12 commits into from
Oct 14, 2020
Merged

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters #22219

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
8ae6408
encodings.normalize_encoding() should ignore non-ASCII letters
shihai1991 Sep 12, 2020
0fcafb8
update test
shihai1991 Sep 12, 2020
dea24d8
Revert "update test"
shihai1991 Sep 12, 2020
dbb0062
apply victor's comment
shihai1991 Oct 3, 2020
3fa221f
apply victor's comment
shihai1991 Oct 9, 2020
a69eef8
Merge remote-tracking branch 'origin/master' into bpo_39337_new_2
shihai1991 Oct 9, 2020
2e73d13
apply victor's comments
shihai1991 Oct 10, 2020
95c1d98
apply victor's comment
shihai1991 Oct 12, 2020
03bfd9b
apply victor's comment
shihai1991 Oct 12, 2020
38f28bd
apply victor's comment
shihai1991 Oct 13, 2020
5982784
apply victor's comment
shihai1991 Oct 13, 2020
4ecb8a1
apply victor's comment
shihai1991 Oct 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion Lib/encodings/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,8 @@ def normalize_encoding(encoding):
if c.isalnum() or c == '.':
if punct and chars:
chars.append('_')
chars.append(c)
if c.isascii():
chars.append(c)
Copy link
Copy Markdown
Member

@vstinner vstinner Oct 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to ask you to add a ".. versionchanged:: 3.10" entry in the documentation, but then I noticed that the encodings module was never documented! Oh!

Copy link
Copy Markdown
Member Author

@shihai1991 shihai1991 Oct 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If end user will use this function or module, I can try to create the doc, but I need some time to do it :)

Copy link
Copy Markdown
Member

@vstinner vstinner Oct 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can and must be addressed in a separated PR anymore. The lack of documentation should not hold this change.

Copy link
Copy Markdown
Member Author

@shihai1991 shihai1991 Oct 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, copy that.

punct = False
else:
punct = True
Expand Down
16 changes: 16 additions & 0 deletions Lib/test/test_codecs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3440,5 +3440,21 @@ def search_function(encoding):
self.assertEqual(NOT_FOUND, codecs.lookup('a\xe9\u20ac-8'))


class EncodingNormalizationTest(unittest.TestCase):
Comment thread
shihai1991 marked this conversation as resolved.
Outdated

def test_normalization(self):
# encodings.normalize_encoding() ignores non-ASCII letters.
out = encodings.normalize_encoding('utf\xE9\u20AC\U0010ffff-8')
self.assertEqual(out, 'utf_8')
out = encodings.normalize_encoding('utf_8')
self.assertEqual(out, 'utf_8')
Comment thread
shihai1991 marked this conversation as resolved.
Outdated
out = encodings.normalize_encoding('utf 8')
self.assertEqual(out, 'utf_8')
Comment thread
shihai1991 marked this conversation as resolved.
Outdated
out = encodings.normalize_encoding('UTF 8')
Copy link
Copy Markdown
Member

@vstinner vstinner Oct 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind to add a comment to explain that the function does not convert upper case letters to lower case letters, just to make the purpose of this test even more explicit?

Copy link
Copy Markdown
Member Author

@shihai1991 shihai1991 Oct 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I don't know how to exact explain it~
I found a case in https://github.com/python/cpython/blob/master/Lib/locale.py#L358.
Looks like It's fine to update encodings.normalize_encoding() to conver to lower-case.

Copy link
Copy Markdown
Member Author

@shihai1991 shihai1991 Oct 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just describe the fact, Lol~

Copy link
Copy Markdown
Member Author

@shihai1991 shihai1991 Oct 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can enhance this encodings.normalize_encoding()? I am not sure~

self.assertEqual(out, 'UTF_8')
out = encodings.normalize_encoding('utf...8')
Comment thread
shihai1991 marked this conversation as resolved.
Outdated
self.assertEqual(out, 'utf...8')


if __name__ == "__main__":
unittest.main()
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Library/2020-09-13-02-02-18.bpo-39337.L3NXTt.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
:func:`encodings.normalize_encoding` now ignores non-ASCII letters.
Comment thread
shihai1991 marked this conversation as resolved.
Outdated