Character classification

[Chris Sherlock]

> On 23 Mar 2017, at 10:47 pm, Stephan Bergmann <sbergman at redhat.com> wrote:
> 
> C character classification and case mapping functions in <ctype.h> are notoriously hard to use.

[snip]

> I audited all uses of such functions (outside of external/ source blobs) on recent master:
> 
> In C++ code, I replaced them with calls to corresponding rtl/character.hxx functions (and adding missing casts from char to unsigned char where necessary; also see <https://cgit.freedesktop.org/libreoffice/core/commit/?id=7778d9f51bd1f4d086cafe95995406c3157afb89> "Prevent calls to rtl/character.hxx functions with (signed) char arguments").  A function corresponding to isspace was missing and has been added with <https://cgit.freedesktop.org/libreoffice/core/commit/?id=f5c93d4149e7ae967e98dbce72528a04a204ca95> "Use rtl::isAscii* instead of ctype.h is* (and fix passing plain char) and add rtl::isAsciiWhiteSpace".  Any calls thus replaced appeared to either never pass in EOF or have been adapted accordingly (<https://cgit.freedesktop.org/libreoffice/core/commit/?id=4a3f2cb747b2553485f48dc440e141e30ade5a70> "Fix some usage of std::istream unformatted input in hwpfilter/source/hwpeq.cxx").  (And there had been no uses of the corresponding std-namespaced functions from <cctype>

Nice work! When I get to the point where I write about the RTL I'll make sure I note this in the part I write about string handling.

> What remains is the source of the five C programs
> 
>  rsc/Executable_rsc.mk (rsc/source/rscpp/cpp{2,3,5,6}.c)
>  shell/Executable_uri_encode.mk (shell/source/unix/misc/uri-encode.c)
>  solenv/Executable_concat-deps.mk (solenv/bin/concat-deps.c)
>  soltools/Executable_cpp.mk (soltools/cpp/_{tokens,unix}.c)
>  soltools/Executable_mkdepend.mk (soltools/mkdepend/{cppsetup,ifparser,parse}.c)
> 
> For one, I have added any casts from char to unsigned char where missing.  (But note that in some cases the input already was of the expected form.)

So the recommendation is to avoid C string functions in LibreOffice code in future?

I realise this may be a silly question, but does this mean we have a portable, cross-culture string handling module that makes things like character case handling consistent across platforms?

If so, I've always wondered what effort it would take to hive off our string handling code to its own project - there is a lot of thought and decades worth of testing for this C and C++ code. Perhaps it might help other projects outside of LibreOffice in the future...

> For another, with a recent set of commits to master I have removed all but one call to setlocale from the LO code base itself.  (The remaining one is in SetSystemLocale in vcl/unx/generic/app/i18n_im.cxx, and smells like it is necessary for proper IME support in VCL-based applications on Linux.  None of those five C programs should be affected by it.)  So barring any calls to setlocale in external code, and ignoring the somewhat fuzzy definition of isprint as called from rsc/source/rscpp/cpp{5,6}.c, those five C programs should not (any longer) be affected by locale issues.

Was this done because of the character casing challenges mentioned above? Or was calling on this causing problems elsewhere?

I'm assuming setting locales is frowned on... 

> Note that there are three remaining calls to toupper/tolower that are currently being addressed by <https://gerrit.libreoffice.org/#/c/35303/> "tdf#99589 change tolower/toupper to easytolower/upper."
> 
> Please avoid adding any new uses of any of those functions.

Nice work! Please pardon me if any of my questions are silly, I am relatively inexperienced with topics related to localization.

Chris