[Piglit] [RFC PATCH 2/2] cmake: build the util libraries as shared under Windows

[Jose Fonseca]

On 23/11/14 22:48, Jose Fonseca wrote:
> On 23/11/14 21:41, Emil Velikov wrote:
>> On 23/11/14 20:14, Jose Fonseca wrote:
>>> On 23/11/14 18:11, Emil Velikov wrote:
>>>> On 23/11/14 11:18, Jose Fonseca wrote:
>>>>> On 22/11/14 22:26, Emil Velikov wrote:
>>>>>> Rather than rebuilding every single test as we change the util
>>>>>> libraries and increase the size of each test by ~9MiB (as noticed
>>>>>> in the mingw-w64 build), just revert to shared piglitutil* libs.
>>>>>
>>>>> This is great. I actually tried to do this, a year ago or so, but I
>>>>> gave
>>>>> up half way through trying to overcome "missing symbols". I don't know
>>>>> if changes in piglit made this easier or if you found a better
>>>>> solution,
>>>>> but I'm happy you worked it out.
>>>>>
>>>> The compile-time missing symbols should be resolved with patch
>>>> 01/02. If
>>>> there are any missing at runtime, can you provide any information -
>>>> which symbol(s) on what Windows, is piglit/glut/etc build via
>>>> mingw{32,-w64} or MSVC, etc. ?
>>>>
>>>>>> TODO:
>>>>>>     - Check if the DLLs are installed and where.
>>>>>>     - If there is one or more executables linking against them, but
>>>>>> located
>>>>>> in a different folder, see the next todo note.
>>>>>>     - Modify the python scripts to set PATH to the location of the
>>>>>> DLLs.
>>>>>> Alternatively apply any of the other solutions as mentioned here
>>>>>
>>>>> The solution is to put all .dlls on the bin directory, next to the
>>>>> executables.
>>>>>
>>>>> And I think that the
>>>>>
>>>>>      set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${piglit_BINARY_DIR}/bin)
>>>>>
>>>> I'm not sure if that will collide with the install() directive below,
>>>> thus the notes and contingency plan :P
>>>>
>>>>> on piglit's top CMakeLists.txt.
>>>>>
>>>>> That said, I think I'd like first to get piglit + waffle on windows
>>>>> wokring reliably, before DLL'ifying the libraries.  This way I don't
>>>>> have to worry about testing with glut.
>>>>>
>>>> DLL'ifying the libraries is not related to using glut/waffle. Thus one
>>>> can just have it ticked off (confirm that it works ok), for either (or
>>>> both) builds and worry about the waffle WGL bugs as we uncover them :)
>>>>
>>>> My current (waffle) builds are
>>>>    - Linux: (GCC default build) ~1150 files, 33MiB in size
>>>>    - Windows: (MSVC default build) ~1030 files, 4.9GiB in size.
>>>> ~70% of which are the debug symbols.
>>>>
>>>> So it's approx 45 times larger, when we ignore the debug symbols and
>>>> that Windows has ~10% tests less. Or 170 times if we consider it.
>>>>
>>>> Hope that's enough to convince you that it might be nice to have this
>>>> earlier rather than later :)
>>>
>>> I don't mind to get it in once all problems are ironed out.  But your
>>> commit message highlitghted some rough edges.   Furthermore this is not
>>> working for me:
>>>
>>> $ ninja -C build/mingw32
>>> ninja: Entering directory `build/mingw32'
>>> [120/2334] Generating builtin_uniform_tests.list
>>> /home/jfonseca/work/vmware/tests/piglit/generated_tests/builtin_function.py:1262:
>>>
>>> RuntimeWarning: overflow encountered in uint_scalars
>>>    f('op-assign-sub', 2, 110, lambda x, y: x - y,
>>> match_assignment_operators,
>>> /home/jfonseca/work/vmware/tests/piglit/generated_tests/builtin_function.py:1304:
>>>
>>> RuntimeWarning: overflow encountered in uint_scalars
>>>    f('op-sub', 2, 110, lambda x, y: x - y, match_simple_binop,
>>> [173/2334] Generating constant_array_size_tests.list
>>> /home/jfonseca/work/vmware/tests/piglit/generated_tests/builtin_function.py:1262:
>>>
>>> RuntimeWarning: overflow encountered in uint_scalars
>>>    f('op-assign-sub', 2, 110, lambda x, y: x - y,
>>> match_assignment_operators,
>>> /home/jfonseca/work/vmware/tests/piglit/generated_tests/builtin_function.py:1304:
>>>
>>> RuntimeWarning: overflow encountered in uint_scalars
>>>    f('op-sub', 2, 110, lambda x, y: x - y, match_simple_binop,
>>> [192/2334] Linking CXX executable bin/fbo-incomplete.exe
>>> FAILED: : && /usr/bin/i686-w64-mingw32-g++  -Wall -g  -static-libgcc
>>> -static-libstdc++
>>> target_api/gl/tests/fbo/CMakeFiles/fbo-incomplete.dir/fbo-incomplete.cpp.obj
>>>
>>>   -o bin/fbo-incomplete.exe
>>> -Wl,--out-implib,target_api/gl/tests/fbo/libfbo-incomplete.dll.a
>>> -Wl,--major-image-version,0,--minor-image-version,0
>>> /home/jfonseca/projects/sdk/mingw32/lib/libfreeglut.a
>>> target_api/gl/tests/util/libpiglitutil_gl.dll.a -lopengl32 -lglu32
>>> /home/jfonseca/projects/sdk/mingw32/lib/libfreeglut.a -lopengl32
>>> target_api/no_api/tests/util/libpiglitutil.dll.a -lkernel32 -luser32
>>> -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32
>>> -ladvapi32 && :
>>> /usr/lib/gcc/i686-w64-mingw32/4.6/libgcc_eh.a(unwind-sjlj.o):(.text+0x2e0):
>>>
>>> multiple definition of `__Unwind_SjLj_Register'
>>> target_api/gl/tests/util/libpiglitutil_gl.dll.a(d000023.o):(.text+0x0):
>>> first defined here
>>> /usr/lib/gcc/i686-w64-mingw32/4.6/libgcc_eh.a(unwind-sjlj.o):(.text+0x3a0):
>>>
>>> multiple definition of `__Unwind_SjLj_Unregister'
>>> target_api/gl/tests/util/libpiglitutil_gl.dll.a(d000026.o):(.text+0x0):
>>> first defined here
>>> /usr/lib/gcc/i686-w64-mingw32/4.6/libgcc_eh.a(unwind-sjlj.o):(.text+0x6a0):
>>>
>>> multiple definition of `__Unwind_SjLj_Resume'
>>> target_api/gl/tests/util/libpiglitutil_gl.dll.a(d000024.o):(.text+0x0):
>>> first defined here
>>> collect2: ld returned 1 exit status
>>> [192/2334] Building C object
>>> target_api/gl/tests/fbo/CMakeFiles/fbo-incomplete-texture-02.dir/fbo-incomplete-texture-02.c.obj
>>>
>>>
>>> ninja: build stopped: subcommand failed.
>>>
>> Seems like the util_gl library retains/exports the static gcc symbols
>> which later clash with the ones in the executable. A bug shamelessly
>> exposed by the patch. We might need to fiddle with the global
>> --static-lib{gcc,stdc++}.
> [...]
>> The symbol mayhem is already there for non-windows builds, and is seems
>> to work fine.
>
> Linux and Windows implementation of shared objects are quite different
> at a fundamental level.
>
> Shared objects on Linux (and other unices)'s symbols are resolved in a
> global namespace.
>
> Whereas Windows DLL's symbols must be resolved by pairs of DLLs (and
> multiple DLLs can export/import different symbols with the same name).
>
> The problem here is not the "symbol mayhem" per se, but getting the
> right symbols exported/imported from the right DLLs, as on Windows we
> cannot rely on a global namespace to pair up things automatically for us
> (like we do on Linux)
>
>  > On a serious note, all we need is to prefix the few
>> remaining functions (where needed) and use a def file to hide anything
>> else :)
>
> I'm not following.
>
> .defs are used to show _more_ symbols. They can't be used to hide
> symbols AFAIK.  If you want to use a .def, then you should list every
> symbols, and never use declspec(dllexport/dllimport).
>
> Using .def instead of dllexport/dllimport is not impossible, but I
> suspect it won't be practical for piglit, because more and more GL
> symbols will need to be added to the .def as tests for new GL extensions
> are written.    declspec(dllexport/dllimport) share some similarities
> with linux symbol visibility, so it's easier to keep both linux and
> indows build systems (less chances of inadvertently breaking windows
> when updating linux.)
>
>  > True. Silly me built only ~50 tests, thus I've not seen how sticky the
>  > situation currently is. So I'll give this a couple full spins on my
>  > Linux and Windows setup before bringing up the topic again.
>
> Thanks.
>
> Don't get me wrong -- I want to help -- and this issue has been on my
> wishlist for a while.   But I think that using waffle on Windows seems
> closer to reach so I'd like to get it done first.
>
>  > Meanwhile if you do remember some of the "missing symbols" and/or other
>  > blockers please let me know.
>
> Unfortunately I can't find my original patches.
>
>
> I do recall I did something along the lines of your 2nd patch, then
> tried to use dllexport/dllimport so the right symbols got generated. And
> this is where I threw the towel: there are a bundch of DLLs, and using
> dllexport/dllimport for each was just too much.

FWIW, I just found what I had, and I've pushed it to

 
http://cgit.freedesktop.org/~jrfonseca/piglit/commit/?h=win32-dll&id=7c54059ac3f1800bf11df498eb54c1067e09a380

Jose