Repository navigation
euc_kr char '0x3164' decode('ksx1001') cause UnicodeDecodeError #101863
Description
Activity
@ezio-melotti (as an active Unicode expert)
I will take a look at this issue as the internal OSS sprint issue.
I will delegate this issue to my mentees.I will try to resolve this issue.
Reacted by Donghee NaReacted by Donghee NaI'm not a python expert.
str.encode()andstr.decode()is a builtin fun?I found the coding logic maybe here:
https://github.com/python/cpython/tree/main/Lib/encodings
https://github.com/python/cpython/blob/main/Lib/encodings/euc_kr.py#L7-L10
that register a codec named 'euc_kr' to provide
encode('euc_kr')and
import _codecs_kris a c source?https://github.com/python/cpython/blob/main/Modules/cjkcodecs/_codecs_kr.c#L7-L9
and
mappings_kr.hhttps://github.com/python/cpython/blob/main/Modules/cjkcodecs/mappings_kr.h
It seems that some arrays are provided here to encode/decode query.
static const ucs2_t __ksx1001_decmap[8264] static const struct dbcs_index ksx1001_decmap[256] static const ucs2_t __cp949ext_decmap[9650] static const struct dbcs_index cp949ext_decmap[256] static const DBCHAR __cp949_encmap[33133] static const struct unim_index cp949_encmap[256]and
mappings_kr.his Generate by https://github.com/python/cpython/blob/main/Tools/unicode/genmap_korean.pyhttps://github.com/python/cpython/blob/main/Tools/unicode/genmap_korean.py#L12-L58
I don't understand why the
cp949encmapsize not equals theksx1001decmapsize?Is
euc_krbi-directional in rules? That if can be encoded, it must can be decoded?Is CP949 equals to ksx1001?
I don't understand why the cp949encmap size not equals the ksx1001decmap size?
Because CP949 is a kind of extended version of KSX1001
iterate unicode :
def main(): encode_count = 0 decode_count = 0 for code_point in range(0, 0x10FFFF + 1): c = chr(code_point) try: raw = c.encode('ksx1001') encode_count += 1 try: raw.decode('ksx1001') decode_count += 1 except: pass except: pass print(f'encode: {encode_count}') print(f'decode: {decode_count}')
encode: 17176 decode: 17175iterate ksx1001 :
def main(): decode_count = 0 for zone_1 in range(1, 94 + 1): for zone_2 in range(1, 94 + 1): try: bytes([zone_1 + 0xA0, zone_2 + 0xA0]).decode('ksx1001') decode_count += 1 except UnicodeDecodeError: pass print(f'decode: {decode_count}')
decode: 8225Reacted by Donghee Nadef main(): not_2_bytes_alphabet = list() not_2_bytes_count = 0 ksx1001_other_count = 0 ksx1001_syllable_count = 0 ksx1001_word_count = 0 ksx1001_but_not_a_char_count = 0 for code_point in range(0, 0x10FFFF + 1): c = chr(code_point) try: raw = c.encode('ksx1001') except UnicodeEncodeError: continue if len(raw) != 2: not_2_bytes_count += 1 not_2_bytes_alphabet.append(c) continue zone_1 = raw[0] - 0xA0 if 1 <= zone_1 <= 12: ksx1001_other_count += 1 elif 16 <= zone_1 <= 40: ksx1001_syllable_count += 1 elif 42 <= zone_1 <= 93: ksx1001_word_count += 1 else: # zone_1 is 13~15, 41, 94 ksx1001_but_not_a_char_count += 1 print(f'ksx other + syllable + word :{ksx1001_other_count + ksx1001_syllable_count + ksx1001_word_count}') print(f'other: {ksx1001_other_count}') print(f'syllable: {ksx1001_syllable_count}') print(f'word: {ksx1001_word_count}') print(f'in ksx but not a char: {ksx1001_but_not_a_char_count}') print(f'not 2 len bytes: {not_2_bytes_count}') print(f'not 2 len bytes + syllable: {not_2_bytes_count + ksx1001_syllable_count}') print('unicode all syllable: 11172') print(''.join(not_2_bytes_alphabet))
ksx other + syllable + word :8226 # 8225 + 1(Hangul Filler) other: 988 syllable: 2350 word: 4888 in ksx but not a char: 0 not 2 len bytes: 8950 # <---- what's this ? Include other syllables not in ksx1001 2350? not 2 len bytes + syllable: 11300 unicode all syllable: 11172not_2_bytes_alphabet:
������� ������������������ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~�갂갃갅갆갋 ...... # Behind them are all syllablesFrom my investigation so far, there are some unclear aspects of the decoding process related to Korean characters encoded in EUC-KR.
FYI, the KS X 1001 standard defines the Korean character set, while EUC-KR is an encoding method for this set. In Python, "ksx1001" is an alias of "euc_kr".
Korean alphabet, also known as Hangul, expresses one syllable consisting of consonants and vowels as one letter. The KS X 1001 standard defines a precomposed set of characters that correspond to one syllable rather than individual consonants and vowels. However, to express a character not included in this set, a method of composing consonants and vowels is defined. The "Hangul Filler" character is used for this purpose, and it, along with the following three consonants and vowels, are arranged to express one character in a total of 8 bytes.
The KS X 1001 standard does not provide any specific guidelines other than the method for expressing one character using the "Hangul Filler".
Korean characters not included in this standard are decomposed into consonants and vowels, and the Korean alphabet code value of the special character area corresponding to each part is used for information exchange. To differentiate it from information exchange for individual consonant and vowel characters, one "Hangul Filler" is added in front of each syllable.
The main issue is whether to allow an independent "Hangul Filler" that is not followed by valid consonants and vowels to compose a single character. Currently, the EUC-KR encoder does not validate strings containing "Hangul Filler" and just convert "Hangul Filler" of KS X 1001 to the corresponding Unicode "Hangul Filler".
I think there are three possible solutions to this issue, which need to be discussed:
- Not fix anything: allow encoding to contain independent "Hangul Filler" characters, while decoding will treat an sequence starting with "Hangul Filler" as an error if it does not represent a valid combination of Korean characters.
- Fix the encoder: strings that contain independent "Hangul Filler" will be treated as an error (this may cause new errors in the existing codes).
- Fix the decoder: a valid sequence will be converted into one composed character, and in other cases, the code of each character will be simply converted into the corresponding code in Unicode.
I have tried the third method and confirmed that no errors have occurred.For more information, please refer to the KS X 1001 specification and a detailed description of "Hangul Filler" character.
I prefer supporting the round-trip approach since we already support encoding.
But I would need to listen to the advice from the other core devs :)I have tried the third method and confirmed that no errors have occurred.
Would you like to submit the patch anyway? We may need to analyze the side effect when we support the decoding for the character.
But please submit the patch with a nice test code.- added 2 commits that reference this issue
on Mar 6, 2023 Since the decoding implementation for this case is undefined and difficult to support decoding for this edge case due to streaming issues.
I close this issue as not a bug.If someone think that this is a bug please let me know.
So far, I still haven't got an explanation that
chr(0x3164).encode('ksx1001').decode('ksx1001')will get a exception is or not a expected behavior (Why python do like this? What is the document or implements reference?).At least in my understanding, the encoder should be idempotent (That if some char can be encoded, it should be decoded).
For example there are some control characters. They have special purposes in the display layer, but this does not affect that they can be encoded and decoded.I also deal with other encoding standards ( gb2312, shitf-jis, big5), that have not the similar behaviors (They are all idempotent).
And this problem does not exist in Java. (I will test later in Rust , Go and node-Javascript)
EUC was used to solve computer coding in the early stage, if they can't convert with unicode(utf-8) equivalent, it seems incredible to me.
My opinion, encoding is a storage structure, data should not be changed through coders but keep it as it is.
So for b8choi's proposal 2 and 3, they are not suitable.
If the char is in the scope, coder should not throw a exception. And coder should not changed the data.
(Imagine that you just browse the ksx file in a utf8 system, the data is changed, this is not reasonable)
Ksx can be fully mapped to unicode, convert the hangul is meaningless. There is also a Hangul Filler in Unicode. Should we convert all the Hangul Filler combin chars to hangul in utf-8 data layer?An evidence that may support my opinion, there is a Emoji Combinations in Unicode.
For example: 👩❤️💋👨 = 👩+❤️+💋+👨
This just a display, on data they are equal.Hangul Filler ’s role should be similar to Ideographic Description Characters only effect on display layer.
If the text engine support Hangul Filler, it will display hangul only on display. If not display as char is.
Bug report
char '0x3164' can be
encode('ksx1001'), but can notdecode('ksx1001')The char is
Hangul Compatibility Jamo -> Hangul Fillerhttps://unicode-table.com/en/3164/
The following code is get the zone in ks-x-1001:
https://en.wikipedia.org/wiki/KS_X_1001#Hangul_Filler
other chars in ksx1001 encode an decode is ok, but only this.
Your environment
Linked PRs