Repository navigation
PHP > 8.1 fails to detect Māori macrons #11908
Description
Description
The following code:
<?php
$str = 'Total%20M%C4%81ori%2C31.5%2C33.3%2C31.8%2C33%2C36.4%2C33.2%2C33.2';
$rawstr = rawurldecode($str);
var_dump(
mb_detect_encoding($rawstr, ['UTF-8', 'ISO-8859-1', 'WINDOWS-1251']),
mb_detect_encoding($rawstr, ['ISO-8859-1', 'WINDOWS-1251', 'UTF-8']),
mb_detect_encoding($rawstr, ['WINDOWS-1251', 'UTF-8', 'ISO-8859-1']),
mb_check_encoding($rawstr, 'ISO-8859-1'),
mb_check_encoding($rawstr, 'UTF-8'),
mb_check_encoding($rawstr, 'WINDOWS-1251'),
);Resulted in this output:
string(12) "Windows-1251"
string(12) "Windows-1251"
string(12) "Windows-1251"
bool(true)
bool(true)
bool(true)
But I expected this output instead:
string(5) "UTF-8"
string(5) "UTF-8"
string(5) "UTF-8"
bool(false)
bool(true)
bool(false)
Related issues
- mb_detect_encoding() results for UTF-7 differ between PHP 8.0 and 8.1 (if UTF-7 is present in the encodings list and the string contains '+' character) #10192
mb_detect_encoding()detects UTF-8 emoji byte sequence as ISO-8859-1 since PHP 8.1 #7871- Improve mb_detect_encoding's recognition of Slavic names #8439
- wrong mb_detect_encoding since php8.1 for very simple utf-8 strings #10481
- mb_detect_encoding does not return the first matching encoding anymore #8279
PHP Version
PHP 8.1.22
Operating System
No response
Activity
hormus commentedon Aug 11, 2023
U+0101 UTF-8 hex C481 valid
ISO-8859-1 hex 81 valid
Windows-1251 hex 81 valid
Expected result true
youkidearitai commentedon Aug 11, 2023
It behavior seems PHP >= 8.1 is correct. Because
Windows-1251is valid of0x81, butISO-8859-1is not valid of0x81.Please see below of
Character setsection.0x81is "undefined".0x81is allocated of "Ѓ"0x81(0b10000001) is "undefined" (not allowed).Therefore, PHP >= 8.1 of
mb_detect_encodingis to this extent, correct behavior.youkidearitai commentedon Aug 11, 2023
I mistake,
0xC4 0x81is correct of UTF-8.However, originally I want you to use the character code assuming what you are using.
youkidearitai commentedon Aug 11, 2023
I'm sorry, I mistake again.
0x81is correct ofISO-8859-1. Therefore,mb_check_encoding("\x81", "ISO-8859-1");is returns true.Anyway, this case is detect(guess) encoding is very difficult. My opinion is not change that I want you to use the character code assuming what you are using.
alexdowad commentedon Aug 22, 2023
@RoSk0 Is U+0101 the only accented character which is commonly used when writing about the Maori people?
xurizaemon commentedon Aug 22, 2023
@alexdowad Thanks for asking. The following is as I understand it (of European descent, living in Aotearoa, some beginner study of Te Reo Māori and a colleague of @RoSk0).
Those characters are used when writing in Te Reo Māori - not only when writing about tangata Māori (Māori people).
There are ten vowels in Te Reo Māori - a,e,i,o,u and the long versions ā,ē,ī,ō,ū, sometimes also represented as double-vowels (aa, ee, ...). Incorrect vowels can have significant impact on meaning. There are also the capital forms A,E,I,O,U and Ā,Ē,Ī,Ō,Ū.
Have added some macron examples to #12025
(Corrections welcome please, doing my best to help here but not expert.)
ndossche commentedon Aug 26, 2023
@alexdowad Is it intentional that this issue is still open, or did you forget to close this? 🙂
alexdowad commentedon Aug 26, 2023
@nielsdos Uhhh... I forgot.