Repository navigation

Major overhaul of mbstring (part 1) - #6052

Closed

alexdowad wants to merge 123 commits into

php:masterfrom

alexdowad:cleanup-mbstring-1

Closed

Major overhaul of mbstring (part 1)#6052
alexdowad wants to merge 123 commits into
php:masterfrom
alexdowad:cleanup-mbstring-1

Commits on Nov 3, 2020
- Fix mbstring support for Shift-JIS
  alexdowad
  committed
  af8d6bf
Commits on Nov 4, 2020
- Add test suite for Shift-JIS encoding
  alexdowad
  committed
  ffb01c9
- Add identify filter for Shift-JIS-2004
  alexdowad
  committed
  b96b1f4
- SJIS-2004 encoding conversion: handle invalid (or truncated) 2nd byte for Kanji correctly
  alexdowad
  committed
  714a7d1
- Don't mangle non-Japanese chars which appear after a 'combining' kana in SJIS-2004
  alexdowad
  committed
  9ece2b3
- Add test suite for SJIS-2004 encoding
  alexdowad
  committed
  4bba852
- Add identify filter for MacJapanese (variant of Shift-JIS)
  alexdowad
  committed
  0c0a4b8
- SJIS-mac encoding conversion: handle invalid (or truncated) 2nd byte for Kanji correctly
  alexdowad
  committed
  55dcf81
- Convert Unicode halfwidth Yen sign to MacJapanese halfwidth Yen sign
  alexdowad
  committed
  4feb701
- SJIS-mac encoding conversion: Stop the carnage of innocent Unicode codepoints
  alexdowad
  committed
  6cd472a
- Add test suite for SJIS-mac encoding
  alexdowad
  committed
  0a6e568
- Enhance mbstring support for UCS-4 text
  alexdowad
  committed
  86e927a
- Leading BOM is stripped for UTF-32
  alexdowad
  committed
  431c4b9
- Enhance mbstring support for UCS-2 text
  alexdowad
  committed
  a38a4a0
- Consolidate all single-byte encodings in one source file
  alexdowad
  committed
  675a311
- Fix identify filter for UTF-7
  alexdowad
  committed
  f6e834f
- Minor code cleanup in mbfilter_utf7.c
  alexdowad
  committed
  02fc58e
- Catch and handle errors in UTF-7 text conversion
  alexdowad
  committed
  0879484
- Add test suite for UTF-{7,8,16,32}
  alexdowad
  committed
  a825801
- Add identify filter for modified UTF-7 (for IMAP protocol)
  alexdowad
  committed
  2c0098d
- Add test suite for mUTF-7 (UTF7-IMAP) encoding
  alexdowad
  committed
  7cbe62b
- Add identify filter for CP50220-raw
  alexdowad
  committed
  cbfc02f
- Add identify filter for 'HTML entities' encoding
  alexdowad
  committed
  5d6ecd3
- Add identify filter for 'byte2be' and 'byte2le' encodings
  alexdowad
  committed
  8e9d740
- Add identify filter for 'byte4be' and 'byte4le' encodings
  alexdowad
  committed
  f3e730c
- Add identify filter for uuencode 'character encoding'
  alexdowad
  committed
  fb9a1d9
- Add identify filter for QPrint 'character encoding'
  alexdowad
  committed
  08a064b
- Add identify filter for Base64 'character encoding'
  alexdowad
  committed
  5550c71
- Stricter identification of valid strings in ISO-2022-JP-2004 encoding
  alexdowad
  committed
  32bb947
- ISO-2022-JP-2004 conversion: handle invalid characters correctly
  alexdowad
  committed
  b374321
- Add test suite for ISO-2022-JP-2004 encoding
  alexdowad
  committed
  3dfc88f
- Add comment explaining why ISO-2022-JP-2004, etc strings end with ESC ( B
  alexdowad
  committed
  2e06352
- Stricter identification of valid strings in JIS7/JIS8 and ISO-2022-JP encodings
  alexdowad
  committed
  f3e3ee5
- JIS7/8 encoding: handle invalid 2nd byte for Kanji correctly
  alexdowad
  committed
  1d10932
- Add test suite for JIS7/JIS8 and ISO-2022-JP encodings
  alexdowad
  committed
  5321788
- Update 'East Asian Width' table to comply with Unicode 13.0
  alexdowad
  committed
  92dd38f
- Correct wrong flags for 'byte2be', 'byte2le', 'byte4be', and 'byte4le' encodings
  alexdowad
  committed
  360187e
- Enhance handling of EUC-JP text encoding
  alexdowad
  committed
  938b517
- Add test suite for EUC-JP encoding
  alexdowad
  committed
  eaca866
- Add identify filter for text encoding 'wchar'
  alexdowad
  committed
  ae0f229
- All mbstring encodings have identify filter now
  alexdowad
  committed
  215c146
- Tell compiler which way jumps will usually go in mbfl_memory_device.c
  alexdowad
  committed
  934a2d8
- Simplify code for handling mbstring language aliases
  alexdowad
  committed
  2c6db2c
- Major refactoring and optimization of mbfilter.c
  alexdowad
  committed
  7874786
- Remove useless struct: mbfl_filt_tl_jisx0201_jisx0208_param
  alexdowad
  committed
  c0eeafd
- Minor code cleanup in php_unicode.h
  alexdowad
  committed
  54e189f
- Remove unneeded function mbfl_convert_filter_copy from mbstring
  alexdowad
  committed
  35a2cec
- Optimize mbstring upper/lowercasing: use fast path in more cases
  alexdowad
  committed
  163815a
- Fix buggy support for negative 'start' argument to mb_strimwidth
  alexdowad
  committed
  9930b0c
- Remove broken test for validity of 'start' argument to mb_strimwidth
  alexdowad
  committed
  809351c
- Remove unneeded function mbfl_filt_tl_jisx0201_jisx0208_init
  alexdowad
  committed
  05ad087
- Remove unneeded struct: mbfl_filt_conv_wchar_cp50220_ctx
  alexdowad
  committed
  fab2151
- mbfl_wchar_device_init accepts initial buffer size argument
  alexdowad
  committed
  20de301
- Remove unused COMPAT2 constants from mbstring
  alexdowad
  committed
  a5e5430
- Rename 'COMPAT1' constants for mbstring to 'SPECIAL' (more meaningful name)
  alexdowad
  committed
  db69a19
- Remove unused MBFL_FILT_TL_{HAN2ZEN,ZEN2HAN}_MASK constants
  alexdowad
  committed
  48e390f
- Rename MBFL_FILT_TL_ZEN2HAN_{HIRA2KANA,KANA2HIRA} to accurately reflect what they do
  alexdowad
  committed
  fd03267
- Disallow nonsensical combinations of conversion flags to mb_convert_kana
  alexdowad
  committed
  6a74bb4
- Return values of 'filter flush' functions are never used
  alexdowad
  committed
  e612f38
- Canonicalize flags received by mb_convert_kana
  alexdowad
  committed
  7cf60e0
- Don't redundantly flush filter chain twice in mbfl_convert_filter_flush
  alexdowad
  committed
  d2d0c1c
- Remove unneeded 'filter_ctor' field from filter structs
  alexdowad
  committed
  1ab4f68
- Initialization of encoding identification filters is more concise
  alexdowad
  committed
  d88df51
- Remove unused 'score' field from mbfl_identify_filter struct
  alexdowad
  committed
  a67cb8f
- Rewrite UTF-8 -> wchar conversion to use function pointer as state variable
  alexdowad
  committed
  e48660d
- Optimize wchar -> JIS conversion (mbfl_filt_conv_wchar_jis)
  alexdowad
  committed
  3c510fa
- Optimize encoding of HTML decimal numeric entities
  alexdowad
  committed
  4b53585
- Offset and mask values in mb_encode_numericentity convmap do not affect which ranges apply
  alexdowad
  committed
  951001a
- Test strimwidth with UTF-16LE text
  alexdowad
  committed
  8836e3c
- Test mb_strcut more thoroughly on UTF-16LE text
  alexdowad
  committed
  3a5c5e7
- mb_convert_kana throws an error on unrecognized flags
  alexdowad
  committed
  4269462
- Use mbfl_memory_device_output for emitting newlines in mb_send_mail
  alexdowad
  committed
  4c19d61
- Code cleanup in mbfl_language.c
  alexdowad
  committed
  60483da
- Code cleanup in mb_str_split
  alexdowad
  committed
  f6447ea
- [WIP] Move meat of mb_str_split to mbfl_str_split (in mbfilter.c)
  alexdowad
  committed
  4a74877
- Minor cleanup in mbfilter_byte2.c
  alexdowad
  committed
  e19f9b2
- Minor cleanup in mbfilter_byte4.c
  alexdowad
  committed
  6223925
- Minor cleanup in mbfilter_base64.c
  alexdowad
  committed
  93158db
- Minor cleanup in mbfilter_uuencode.c
  alexdowad
  committed
  0e01d8a
- Code cleanup in mbfilter_htmlent.c
  alexdowad
  committed
  e9d4694
- Code cleanup in mbfilter_qprint.c
  alexdowad
  committed
  bb5fb71
- Combine MBFL_ENCTYPE_WCS{2,4}{BE,LE} constants
  alexdowad
  committed
  e39473d
- Nothing uses return values of identity filter functions
  alexdowad
  committed
  03b57af
- Identify filter functions should take unsigned char, not int
  alexdowad
  committed
  b472672
- Encoding conversion functions don't need to return anything
  alexdowad
  committed
  aeca9e6
- Optimize calls to mb_substr which return entire input string
  alexdowad
  committed
  fe8ad10
- Minor code tweaks in mbfl_convert.c
  alexdowad
  committed
  1ce3045
- Add comment on assertion in mbstring.c which can fail
  alexdowad
  committed
  cef8d9e
- Add 'Windows-932' alias for CP932 text encoding
  alexdowad
  committed
  9274e2a
- Enhance handling of CP932 text encoding
  alexdowad
  committed
  b2a7ad4
- Add test suite for CP932 text encoding
  alexdowad
  committed
  2cee875
- Don't pass invalid JIS X 0208 characters through
  alexdowad
  committed
  490baf6
- Don't pass invalid JIS X 0212 and Windows-CP932 characters through
  alexdowad
  committed
  7100498
- Remove useless mbstring encoding 'CP50220-raw'
  alexdowad
  committed
  5f0226a
- Fix identify filter for CP50220 text encoding
  alexdowad
  committed
  24267b4
- Stricter handling of erroneous input when converting CP5022{0,1,2} text encoding
  alexdowad
  committed
  99320df
- CP5022{0,1,2}: convert characters in ku 0x2D (13th row) correctly
  alexdowad
  committed
  246bba7
- CP5022{0,1,2}: convert Unicode codepoints in 'user' area (0xE000-E757) correctly
  alexdowad
  committed
  2393da2
- Add test suite for CP5022{0,1,2}
  alexdowad
  committed
  cfd3b56
- Combine MBFL_ENCTYPE_MWC2{BE,LE} constants
  alexdowad
  committed
  73cac6a
- Remove useless mbstring encoding 'JIS-ms'
  alexdowad
  committed
  9f5fee9
- Remove unused macros from mbfilter_cp51932.c, mbfilter_iso2022jp_mobile.c
  alexdowad
  committed
  61981ad
- Enhance handling of CP51932 encoding
  alexdowad
  committed
  1bda08d
- Add test suite for CP51932 encoding
  alexdowad
  committed
  b222b4d
- Remove unused function php_mb_safe_strrchr
  alexdowad
  committed
  2540809
- Remove useless constant MBFL_ENCTYPE_MBCS
  alexdowad
  committed
  8ef5543
- Simplify php_mb_zend_encoding_converter
  alexdowad
  committed
  7e18f2f
- Don't check for impossible error condition in mb_strlen
  alexdowad
  committed
  c48d07c
- Remove useless constant MBFL_ERROR_ENCODING
  alexdowad
  committed
  c4cbaa7
- Don't check for impossible error condition in mb_substr_count
  alexdowad
  committed
  a8d947c
- Don't check for impossible error condition in mb_strwidth
  alexdowad
  committed
  951bc0d
- Use stack-allocated buffer in php_mb_chr
  alexdowad
  committed
  0488bbb
- Remove unused function php_mb_mbchar_bytes
  alexdowad
  committed
  2c53090
- No need to null-terminate buffer in php_mb_chr
  alexdowad
  committed
  5234d92
- Fix mbstring support for SJIS-Mobile (DoCoMo, KDDI, and Softbank variants of Shift-JIS)
  alexdowad
  committed
  befbb46
- Add test suite for SJIS-Mobile
  alexdowad
  committed
  347e14c
- CP932: treat truncated multi-byte characters as an error
  alexdowad
  committed
  8da76e5
- Remove duplicate implementation of CP932 from mbstring
  alexdowad
  committed
  0534a1d
- Fix mbstring support for ISO-2022-JP-KDDI encoding
  alexdowad
  committed
  83653b2
- Add test suite for ISO-2022-JP-KDDI encoding
  alexdowad
  committed
  6cef434
- Fix mbstring support for ISO-2022-JP-MS encoding
  alexdowad
  committed
  3534564
- Add test suite for ISO-2022-JP-MS encoding
  alexdowad
  committed
  dadfc70
- Refactoring of UTF-8 with mobile vendor extensions (DoCoMo, KDDI, SoftBank)
  alexdowad
  committed
  25ac652

Commits

Commits on Nov 3, 2020

Commits on Nov 4, 2020