[wp-trac] [WordPress Trac] #62172: Deprecate non-UTF-8 Support

[WordPress Trac]

#62172: Deprecate non-UTF-8 Support
-------------------------+----------------------------
 Reporter:  dmsnell      |      Owner:  (none)
     Type:  enhancement  |     Status:  new
 Priority:  normal       |  Milestone:  Future Release
Component:  General      |    Version:  trunk
 Severity:  normal       |   Keywords:
  Focuses:               |
-------------------------+----------------------------
 WordPress' code and history is full of ambiguity on character encoding.
 When WordPress was formed, many websites and systems still used various
 single-byte region-specific text encodings, and some used more complicated
 shifting encodings, but today, UTF-8 is near-universal and the standard
 recommendation for interoperability between systems.

 Significant complexity in WordPress codebase exists in an attempt to
 properly handle various character encodings. Unfortunately, in many (if
 not most) of these cases, the code is confused on what strings are in what
 encodings and how those need to be transformed in order to make proper
 sense of it all.

 Furthermore, the `blog_charset` appears to have been introduced for the
 purpose of writing a `<meta>` tag on rendered pages to let a browser know
 what encoding to expect, while WordPress itself was to remain agnostic
 with regards to that same encoding. Over time, the option has been used as
 a mechanism for indicating how to transform strings, which doesn't resolve
 any of the problems introduced by working with multiple character
 encodings. (thanks @mdawaffe for the history there).

 In any given WordPress request:
  - data in the database is stored in one of two ways: either as encoded
 text in some character encoding, or as the raw bytes of some encoding
 which are mislabeled as `latin1` so that MySQL doesn't attempt to
 interpret the bytes.
  - data is read from MySQL and possibly transformed from the stored bytes
 into a connection/session-determined encoding and collation, unless a
 query-specified encoding is also provided.
  - PHP source code is stored as UTF-8 or is US-ASCII compatible, making
 string-based operations against possibly-transformed data from the
 database.
  - Various PHP code will read the currently-set locale or
 `default_charset`, `input_encoding`, `output_encoding`, or
 `internal_encoding` and operate differently because of an assumption that
 the bytes on which they are operating is in those other encodings.
  - Files are read from the filesystem which are probably encoded in UTF-8.
  - Query args are parsed and percent-escaping is decoded, whose source
 encoding is not guaranteed to be UTF-8.
  - POST arguments are read, parsed, and percent-decoded, again without
 clarity on which byte encoding they are escaping.
  - HTML named character references are encoded and decoded, which
 translate into different byte sequences based on the configured character
 encoding, often set by `blog_charset`.
  - Various filters and functions in Core, like `wp_spaces_regex()` examine
 specific byte sequences, which are UTF-8-specific, against strings which
 may have the same character sequence but in a different byte sequence.
  - Network requests might be made, which are read and parsed, which may
 come in different encodings according to the `Content-type`.
  - HTML is sent to the browser and a `<meta charset="">` tag is produced
 to instruct the browser how to interpret the bytes it receives. This may
 or may not match the HTML which WordPress is generating, as most block
 code and most filters are hard-coded PHP strings in UTF-8 or are at least
 isomorphic to it up to US-ASCII.

 So as is the case with deprecating XHTML and HTML4 support, deprecating
 UTF-8 is mostly about being honest with ourselves and making space
 officially to remove complex and risky parts of the codebase that often do
 more harm and help. There's a good chance today that WordPress is already
 extremely fragile when working with non-UTF-8 systems, and deprecating it
 would make it possible to fix those existing issues.

 Deprecating UTF-8 means WordPress can stop attempting to support an N-to-M
 text-encoding architecture and replace it with an N-to-1 architecture,
 where strings that need to be converted are converted at the boundary of
 the system while everything inside the system is UTF-8, harmonizing all of
 the different levels of encoding and code.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/62172>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform