<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi guys, </div><div class=""><br class=""></div><div class="">I’m afraid I’m still a bit stuck on this, any other ideas what might be causing the problem?</div><div class=""><br class=""></div><div class="">Chris</div><br class=""><div><blockquote type="cite" class=""><div class="">On 6 Jan 2016, at 4:27 AM, Chris Sherlock <<a href="mailto:chris.sherlock79@gmail.com" class="">chris.sherlock79@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Thanks Mark, appreciate these code pointers!</div><div class=""><br class=""></div><div class="">(I’m cc’ing in the mailing list so others can comment)</div><div class=""><br class=""></div><div class="">Chris</div><br class=""><div class=""><blockquote type="cite" class=""><div class="">On 4 Jan 2016, at 8:21 PM, Mark Hung <<a href="mailto:marklh9@gmail.com" class="">marklh9@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><span style="font-size:14px" class=""><br class=""></span></div><span style="font-size:14px" class="">I meant there is a chance for SvParser::GetNextChar() to switch encoding, but yes it is less relevant.</span><div class=""><span style="font-size:14px" class=""><br class=""></span><div class="">Grepping content-type under ucb , there are some suspicious code</div><div class=""><div class=""><a href="http://opengrok.libreoffice.org/xref/core/ucb/source/ucp/webdav-neon/ContentProperties.cxx#454" class="">http://opengrok.libreoffice.org/xref/core/ucb/source/ucp/webdav-neon/ContentProperties.cxx#454</a><br class=""></div><div class=""><a href="http://opengrok.libreoffice.org/xref/core/ucb/source/ucp/webdav/ContentProperties.cxx#471" class="">http://opengrok.libreoffice.org/xref/core/ucb/source/ucp/webdav/ContentProperties.cxx#471</a></div></div><div class=""><br class=""></div><div class="">Which seems incosistent with </div><div class=""><a href="http://opengrok.libreoffice.org/xref/core/sc/source/filter/html/htmlpars.cxx#264" class="">http://opengrok.libreoffice.org/xref/core/sc/source/filter/html/htmlpars.cxx#264</a></div><div class=""><br class=""></div></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">2016-01-04 16:17 GMT+08:00 Chris Sherlock <span dir="ltr" class=""><<a href="mailto:chris.sherlock79@gmail.com" target="_blank" class="">chris.sherlock79@gmail.com</a>></span>:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">Hi Mark,</div><div class=""><br class=""></div>BOM detection is irrelevant here. The HTTP header states that it should be UTF8, but this is not being honoured. <div class=""><br class=""></div><div class="">There is something further down the stack that isn’t recording the HTTP headers. </div><span class="HOEnZb"><font color="#888888" class=""><div class=""><br class=""></div><div class="">Chris</div></font></span><div class=""><div class="h5"><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On 4 Jan 2016, at 4:23 PM, Mark Hung <<a href="mailto:marklh9@gmail.com" target="_blank" class="">marklh9@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Hi Chris,<div class=""><br class=""></div><div class="">As recently I'm working on SvParser and HTMLParser, </div><div class=""><br class=""></div><div class="">There is BOM detection is in SvParser::GetNextChar().</div><div class=""><br class=""></div><div class="">A quick look at eehtml, EditHTMLParser<span class="">::</span><a name="1520bb6d3aa4312b_EditHTMLParser" style="color:rgb(153,0,153);font-weight:bold;font-style:italic" class=""></a>EditHTMLParser seems relevant.</div><div class=""><br class=""></div><div class="">Best regards.</div><div class=""><div class=""><br class=""></div></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">2016-01-04 12:02 GMT+08:00 Chris Sherlock <span dir="ltr" class=""><<a href="mailto:chris.sherlock79@gmail.com" target="_blank" class="">chris.sherlock79@gmail.com</a>></span>:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="">Hey guys, <div class=""><br class=""></div><div class="">Probably nobody saw this because of the time of year (Happy New Year, incidentally!!!).</div><div class=""><br class=""></div><div class="">Just a quick ping to the list to see if anyone can give me some pointers. </div><span class=""><font color="#888888" class=""><div class=""><br class=""></div><div class="">Chris</div></font></span><div class=""><div class=""><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On 30 Dec 2015, at 12:15 PM, Chris Sherlock <<a href="mailto:chris.sherlock79@gmail.com" target="_blank" class="">chris.sherlock79@gmail.com</a>> wrote:</div><br class=""><div class="">Hi guys,<div class=""><br class=""></div><div class="">In bug 95217 - <a href="https://bugs.documentfoundation.org/show_bug.cgi?id=95217" target="_blank" class="">https://bugs.documentfoundation.org/show_bug.cgi?id=95217</a> - Persian test in a webpage encoded as UTF-8 is corrupting.</div><div class=""><br class=""></div><div class="">If I take the webpage and save to an HTML file encoded as UTF8, then there are no problems and the Persian text comes through fine. However, when connecting to a webserver directly, the HTTP header correctly gives the content type as utf8.</div><div class=""><br class=""></div><div class="">I did a test using Charles Proxy with its SSL interception feature turned on and pointed Safari to <a href="https://bugs.documentfoundation.org/attachment.cgi?id=119818" target="_blank" class="">https://bugs.documentfoundation.org/attachment.cgi?id=119818</a></div><div class=""><br class=""></div><div class="">The following headers are gathered:</div><div class=""><br class=""></div><div class=""><pre style="font-size:inherit;width:50em" class="">HTTP/1.1 200 OK Server: nginx/1.2.1 Date: Sat, 26 Dec 2015 01:41:30 GMT Content-Type: text/html; name="text.html"; charset=UTF-8 Content-Length: 982 Connection: keep-alive X-xss-protection: 1; mode=block Content-disposition: inline; filename="text.html" X-content-type-options: nosniff</pre></div><br class=""><div class="">Some warnings are spat out that it editeng's eehtml can't detect the encoding. I initially thought it was looking for a BOM, which makes no sense for a webpage, but that's wrong. Instead, for some reason the headers don't seem to be processed and the HTML parser is falling back to ISO-8859-1 and not UTF8 as the character encoding.</div><div class=""><br class=""></div><div class="">We seem to use Neon to make the GET request to the webserver. A few observations:</div><div class=""><br class=""></div><div class="">1. We detect a server OK response as an error</div><div class="">2. (Probably more to the point) I believe PROPFIND is being used, but actually even though the function being used indicates a PROPFIND verb is used a GET is used as is normal but the headers aren't being stored. This ,Evans that when the parser looks for the headers to find the encoding it's not finding anything, resulting in a fallback to ISO-8859-1.</div><div class=""><br class=""></div><div class="">One easy thing (doesn't solve the root issue) is that wouldn't it be a better idea to fallback to UTF8 and not ISO-8859-1, given ISO-8859-1 is really just a subset of UTF-8?</div><div class=""><br class=""></div><div class="">Any pointers on how to get to the bottom of this would be appreciated, I'm honestly not up on webdav or Neon.</div><div class=""><br class=""></div><div class="">Chris Sherlock</div> </div></blockquote></div><br class=""></div></div></div></div><br class="">_______________________________________________<br class=""> LibreOffice mailing list<br class=""> <a href="mailto:LibreOffice@lists.freedesktop.org" target="_blank" class="">LibreOffice@lists.freedesktop.org</a><br class=""> <a href="http://lists.freedesktop.org/mailman/listinfo/libreoffice" rel="noreferrer" target="_blank" class="">http://lists.freedesktop.org/mailman/listinfo/libreoffice</a><br class=""> <br class=""></blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class=""><div dir="ltr" class=""><div class="">Mark Hung<br class=""></div></div></div> </div> </div></blockquote></div><br class=""></div></div></div></div></blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class="gmail_signature"><div dir="ltr" class=""><div class="">Mark Hung<br class=""></div></div></div> </div> </div></blockquote></div><br class=""></div></div></blockquote></div><br class=""></body></html>