Normalize named entities to numeric

We should never be outputting named entities other than the ones in XML,
< > & ", because that will break well-formedness unless
we have a DTD in the doctype, which we don't in HTML5 mode.

I stuck with outputting numeric entities here instead of UTF-8 because
some characters are hard to read in UTF-8 (e.g.,  ).  Maybe it
would be nicer if we decoded to UTF-8 except for whitespace and control
characters, or something like that, but it's a detail.

I'll backport to 1.17 and add RELEASE-NOTES there, which is why I added
the line to HISTORY instead of RELEASE-NOTES.

diff --git a/HISTORY b/HISTORY
index e897e88..3772349 100644 (file)
--- a/HISTORY
+++ b/HISTORY
@@ -455,6 +455,7 @@ general notes.
 * (bug 20244) Installer does not validate SQLite database directory for stable path
 * (bug 1379) Installer directory conflicts with some hosts' configuration panel.
 * (bug 12070) After Installation MySQL was blocked
+* Fix XML well-formedness on a few pages when $wgHtml5 is true (the default)
 
 === API changes in 1.17 ===
 * (bug 22738) Allow filtering by action type on query=logevent.

diff --git a/includes/Sanitizer.php b/includes/Sanitizer.php
index e26c86d..4c99e82 100644 (file)
--- a/includes/Sanitizer.php
+++ b/includes/Sanitizer.php
@@ -1093,7 +1093,8 @@ class Sanitizer {
         * for XML and XHTML specifically. Any stray bits will be
         * &-escaped to result in a valid text fragment.
         *
-        * a. any named char refs must be known in XHTML
+        * a. named char refs can only be < > & ", others are
+        *   numericized (this way we're well-formed even without a DTD)
         * b. any numeric char refs must be legal chars, not invalid or forbidden
         * c. use &#x, not &#X
         * d. fix or reject non-valid attributes
@@ -1130,9 +1131,10 @@ class Sanitizer {
 
        /**
         * If the named entity is defined in the HTML 4.0/XHTML 1.0 DTD,
-        * return the named entity reference as is. If the entity is a
-        * MediaWiki-specific alias, returns the HTML equivalent. Otherwise,
-        * returns HTML-escaped text of pseudo-entity source (eg &foo;)
+        * return the equivalent numeric entity reference (except for the core <
+        * > & "). If the entity is a MediaWiki-specific alias, returns
+        * the HTML equivalent. Otherwise, returns HTML-escaped text of
+        * pseudo-entity source (eg &foo;)
         *
         * @param $name String
         * @return String
@@ -1141,8 +1143,11 @@ class Sanitizer {
                global $wgHtmlEntities, $wgHtmlEntityAliases;
                if ( isset( $wgHtmlEntityAliases[$name] ) ) {
                        return "&{$wgHtmlEntityAliases[$name]};";
-               } elseif( isset( $wgHtmlEntities[$name] ) ) {
+               } elseif ( in_array( $name,
+               array( 'lt', 'gt', 'amp', 'quot' ) ) ) {
                        return "&$name;";
+               } elseif ( isset( $wgHtmlEntities[$name] ) ) {
+                       return "&#{$wgHtmlEntities[$name]};";
                } else {
                        return "&$name;";
                }

diff --git a/tests/parser/parserTests.txt b/tests/parser/parserTests.txt
index 32d5a3c..6c2f9ed 100644 (file)
--- a/tests/parser/parserTests.txt
+++ b/tests/parser/parserTests.txt
@@ -1264,7 +1264,7 @@ Multiplication table
 <caption>Multiplication table
 </caption>
 <tr>
-<th> &times; </th>
+<th> &#215; </th>
 <th> 1 </th>
 <th> 2 </th>
 <th> 3
@@ -1351,7 +1351,7 @@ Nested table
 !! result
 <table border="1">
 <tr>
-<td> &alpha;
+<td> &#945;
 </td>
 <td>
 <table bgcolor="#ABCDEF" border="2">
@@ -1730,7 +1730,7 @@ Non-breaking spaces in title
 !! input
 [[&nbsp; Main &nbsp; Page &nbsp;]]
 !! result
-<p><a href="/wiki/Main_Page" title="Main Page">&nbsp; Main &nbsp; Page &nbsp;</a>
+<p><a href="/wiki/Main_Page" title="Main Page">&#160; Main &#160; Page &#160;</a>
 </p>
 !!end