Vote #72981
完了Diff outputs become ??? in some non ASCII words.
0%
journals
The Problem is, that for example the following diff-lines
<pre><code class="diff">
- часа"
+ часов"
</code></pre>
are parsed in Redmine as UTF-8 like this:
<pre><code>\xD1\x87\xD0\xB0\xD1\x81\xD0<span>\xB0</span>"
\xD1\x87\xD0\xB0\xD1\x81\xD0<span>\xBE\xD0\xB2</span>"
</code></pre>
This is wrong, because the _leading byte_ <code>\xD0</code> is part of the cyrillic 2-Byte character "<code>а</code>" in the <span>-tag, but it's actually outside of the <span>-tag. Therefore charaters will be misinterpreted and will be displayed with "?".
Correct UTF-8 would be:
<pre><code>\xD1\x87\xD0\xB0\xD1\x81<span>\xD0\xB0</span>"
\xD1\x87\xD0\xB0\xD1\x81<span>\xD0\xBE\xD0\xB2</span>"
</code></pre>
So we have for the first line "<code>...<span>\xD0\xB0</span>...</code>" instead of "<code>...\xD0<span>\xB0</span>...</code>". The attached patch searchs for the last _leading byte_, if the unmatching byte is a _continuation byte_ (and not a _leading byte_ or a single character byte).
A _continuation byte_ has the binary format 10xxxxxx, so we can determine it with <code>myContinuationByte.ord.between?(128, 191)</code>
This problem occurs always, when the first determined difference between two bytes are _continuation bytes_. An other example in japanese you find in #13350.
--------------------------------------------------------------------------------
A much better way to fix this problem is to set an UTF-8 encoding. :-)
--------------------------------------------------------------------------------
The affected version is also 2.3 (devel)
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Committed in, thanks.
--------------------------------------------------------------------------------
related_issues
relates,Closed,12640,Russian "about_x_hours" translation change