Adding Extension for Experimental Thai Spelling

[Nathan Wells]

Hello Again,

Thank you all for your input!

This is a deeper problem than I first thought...sorry for the delayed
response, but I hope a solution can be found, even though the current ICU
breakiterator is not at 100% for Khmer.

Here are some new ideas, ordered by desirability, with number one being the
most desired, to number three being the least.

1) When a zero-width space is detected (U+200B), shut off ICU breakiterator
for Khmer spell checking for characters following the zero-width space
until encounters real space (U+0020) or end of sentence (detect end of
sentence using ICU Sentence Boundary).

2) Disable use of ICU breakiterator for Khmer spell checking by default,
but allow users to enable it by adding a check-box to enable ICU
breakiterator in the Tools > Options > Language Settings > Writing Aids >
Options dialogue when a Khmer Hunspell dictionary is present (
http://extensions.libreoffice.org/extension-center/khmer-spelling-checker-sbbic-version
 ).

3) Disable use of ICU breakiterator for Khmer spell checking until the ICU
breakiterator for Khmer is more accurate.

Currently, with the ICU breakiterator for Khmer enabled in LibreOffice 3.6
it causes a lot of spelling errors to go unnoticed since the ICU
breakiterator breaks words up incorrectly. So hopfully we can find a
solution that will work with the current ICU breakiterator - though with
ICU 50.1 the breakiterator for Khmer will have some improvements. But I do
feel if solution 1 or 2 (or if someone else has better ideas) cannot
be implemented the breakiterator for spelling with Khmer should be turned
off in LibreOffice until the ICU breakiterator for Khmer is more accurate.


Thanks again for your help and time, your input is greatly appreciated!

Sincerely,

Nathan



On Thu, Jul 26, 2012 at 4:33 PM, Martin Hosken <martin_hosken at sil.org>wrote:

> Dear All,
>
> > > An automatic word and line breaker is very necessary for Khmer and
> > > Thai because traditionally they have no spaces between words, and so
> > > line-breaking and spell checking require the use of a zero-width space
> > > between words which is counterintuitive for most native speakers, and
> > > so spell checking goes widely unused.
>
> I agree that automatic word breaking is a good thing and I am relieved to
> see that libreoffice does it based on language selection and not on
> automatic language guessing based on scripts. There are more languages that
> use Thai script and Khmer script than just Thai and Khmer. So one of my
> fears is already alleviated :)
>
> > > But now with the ICU code you implemented, Thai and Khmer can be
> > > automatically broken, and the results are quite good. But with its
> > > implementation in the real world, I have found some issues that I
> > > wanted to raise and also suggest possible solutions. I write this as
> > > an end-user, not so much as a programmer, nor do I claim to fully
> > > understand the inner-workings of ICU and LibreOffice (because I don't!
> > > ).
> > >
> > > First, I will do my best to explain the current results of the ICU
> > > break iterator with Khmer:
> > >
> > > Input sentence: មានប្រាជ្ញាឈ្លាសវៃឈ្មោះសិវកឥវលិយៈ
> > >
> > > Current ICU line-breaking: មាន|ប្រាជ្ញាឈ្លាស|វៃ|ឈ្មោះ|សិវ|កឥ|វលិ|យៈ
> > >
> > > Compared with the sentence manually broken: មាន|ប្រាជ្ញា|ឈ្លាសវៃ|
> > > ឈ្មោះ|សិវកឥវលិយៈ
> > >
> > > The differences should be clear – the ICU break iterator does not
> > > break the words with 100% accuracy.
> > >
> > > One possible solution to this issue is by how the ICU Break Iterator
> > > interacts with zero-width spaces (U+200B) in LibreOffice. Before ICU
> > > code was enabled to automatically break Khmer, if an end-user wanted
> > > to spell check Khmer, they had to manually place U+200B characters to
> > > separate words. This solution worked quite well, but was
> > > counterintuitive to most native speakers, because Khmer has no spaces
> > > (as stated before). But with this solution, an end-user could be sure
> > > that their document was broken with 100% accuracy, if there was no
> > > human error (something automatic solutions cannot do – it is more
> > > along the lines of 80% accurate). What I propose, is that the break
> > > iterator code in LibreOffice looks for U+200B characters in a given
> > > string and considers them as a sign to NOT automatically break, but to
> > > allow the end-user full control to manually break words. Let me
> > > explain:
> > >
> > >      1. The code starts processing the text and automatically breaking
> > >         it until it comes across a U+200B character. If one is found,
> > >         it searches to see if there are any additional U+200B or U
> > >         +0020 characters in the following 20 characters (or so), and
> > >         if there are, the break iterator skips over those characters
> > >         and starts again from the second U+200B character (or U+0020,
> > >         but a U+0020 character would only signify the “close” of the
> > >         manual break because sometimes a phrase will end and there
> > >         will be an actual space – so if the word that the user wants
> > >         to manually break has a “real” U+0020 space at the end of it,
> > >         then the user does not need to put an additional U+200B
> > >         character to close it) which then repeats, looking for U+200B
> > >         characters etc.
> > >
> > >      2. This would allow end-users to choose to manually break their
> > >         whole document so they can have precise control, as well as
> > >         allow end-users to place U+200B characters around names of
> > >         people, places or transliterations in order to tell the break
> > >         iterator to not try to break those words.
>
> In principle I like this approach. I like the idea of being able to force
> breaks and non-breaks. But I don't think we are quite there with this
> solution yet. Here are my difficulties with it:
>
> 1. use of U+2060 makes string searching and spell checking harder (unless
> WJ chars are stripped for searching and spell checking). They are not part
> of the spelling of a word, so their introduction in the underlying text
> stream is problematic for other text processing processes (like searching
> as mentioned). This is less of an issue for U+200B ZWSP because that occurs
> between words and searching across word boundaries is a rarer activity.
> Likewise spell checking across word boundaries isn't really needed.
>
> 2. How do we come up with the range of what is considered a word between
> two zwsp chars as opposed to two words? How close to the end of a string
> must a zwsp occur to disable all breaking before the end of the string?
> does "abcdef<zwsp>uvwxyz" block all breaks in the string? I think we need
> to think harder (deeper) about the use of zwsp in this way and see if we
> can come up with something with a little less ambiguity. Having said that,
> I think we are going to have to think really hard, because I don't think
> this is an easy problem.
>
> > >      4. I then notice that "ម្នាក់ទៀត" line breaks together (since the
> > >         automatic line-breaking breaks them as one word. And I decide
> > >         I would rather line-break after “ម្នាក់” rather than have both
> > >         words break connected to each other, so I place a zero-width
> > >         space between the words:
> > >         មាន​ប្រាជ្ញាឈ្លាស​វៃ​ឈ្មោះ<zw>សិវ​កឥ​វលិ​យៈ<sp>​អ្នកប្រាជ្ញ
> > >         ម្នាក់<zw>ទៀត​ដែល​ល្បីល្បាញ​ជាងគេ
> > >         the automatic break iterator comes to the zero width space and
> > >         then stops automatically breaking and look ahead to see if
> > >         there is a zero-width space or a “real” space within 20
> > >         characters (this number might need refining, but I think 20
> > >         characters would be enough). As there are no zero-width or
> > >         “real” spaces within 20 characters, the break iterator then
> > >         goes back to the previous zero-width and starts breaking
> > >         starting from the zero-width character.
>
> Now what happens if I want to put zw around a word that occurs < 20 chars
> after my last zw? The on off nature of the zw has now been inverted. One
> option is to say that zw must always occur in pairs and you would have to
> bracket your first or second word there. But then management of which zw is
> on and which is off will get confusing for users.
>
> An alternative model is to weight breakpoints. An explicit breakpoint
> weighs more highly than an automatically generated one. Then when it comes
> to line breaking the weight of a breakpoint counts towards its choice as to
> the actual break. For example if we say an explicit break is 2 and an
> automatic is 1. Then we might use a square rule for distance and say: an
> explicit break is preferred if it occurs closer to the end of a line than
> 4x the distance to the last automatic break on the line. Or somesuch.
>
> Yours,
> Martin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/libreoffice/attachments/20120927/329f3eb7/attachment-0001.html>