[Mesa-dev] [PATCH v3] python: Rework bytes/unicode string handling

[Jose Fonseca]

On 17/08/18 14:03, Mathieu Bridon wrote:
> On Fri, 2018-08-17 at 13:29 +0100, Jose Fonseca wrote:
>> This change caused one of our MSVC build machines to fail with
>>
>> scons: Building targets ...
>>     Generating build\windows-x86-debug\util\xmlpool\options.h ...
>> Traceback (most recent call last):
>>     File "src\util\xmlpool\gen_xmlpool.py", line 221, in <module>
>>       print(line, end='')
>> UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c'
>> in
>> position 68: ordinal not in range(128)
>> scons: *** [build\windows-x86-debug\util\xmlpool\options.h] Error 1
> 
> Argh!
> 
> I think that's because we're now printing a unicode string. On Python 3
> that's the right thing to do, if we printed the encoded byte string
> we'd get the "b'…'" representation, which we certainly don't want:
> 
>>>> print(u'août')
> août
>>>> print(u'août'.encode('utf-8'))
> b'ao\xc3\xbbt'
> 
> 
> But on Python 2, we should really print the byte string though. Python
> 2 helpfully tries to encode the unicode string automatically, using its
> default encoding: ASCII.
> 
> That obviously fails when the string contains non-ascii character.
> 
> I'll send a patch ASAP.

Thanks!

>> Setting PYTHONIOENCODING=utf-8 helps, but then bad things still
>> happen when the output is loaded src/gallium/auxiliary/pipe-loader/
>>
>>
>> But the fact is that everything was working before.
>>
>>
>> Perhaps a solution is to just start using Python 3 for the
>> generation scripts, as it might yield more consistent results.
> 
> That's a possibility, but then it means you need both Python 2 (for
> Scons) and Python 3 (for the scripts). Requiring 2 Python stacks to
> build a C codebase is pretty terrible. :-/

True.  I think we can all agree the ideal is to just always Python 3. 
I'm just wondering if that wouldn't be a

The hurdle with SCons is that upgrading to Python 3 also requires 
updating all SConscripts and updagre SCons version itself, in lockstep..

Jose