[Piglit] [RFC PATCH 2/2] cmake: build the util libraries as shared under Windows

[Emil Velikov]

On 23/11/14 20:14, Jose Fonseca wrote:
> On 23/11/14 18:11, Emil Velikov wrote:
>> On 23/11/14 11:18, Jose Fonseca wrote:
>>> On 22/11/14 22:26, Emil Velikov wrote:
>>>> Rather than rebuilding every single test as we change the util
>>>> libraries and increase the size of each test by ~9MiB (as noticed
>>>> in the mingw-w64 build), just revert to shared piglitutil* libs.
>>>
>>> This is great. I actually tried to do this, a year ago or so, but I gave
>>> up half way through trying to overcome "missing symbols". I don't know
>>> if changes in piglit made this easier or if you found a better solution,
>>> but I'm happy you worked it out.
>>>
>> The compile-time missing symbols should be resolved with patch 01/02. If
>> there are any missing at runtime, can you provide any information -
>> which symbol(s) on what Windows, is piglit/glut/etc build via
>> mingw{32,-w64} or MSVC, etc. ?
>>
>>>> TODO:
>>>>    - Check if the DLLs are installed and where.
>>>>    - If there is one or more executables linking against them, but
>>>> located
>>>> in a different folder, see the next todo note.
>>>>    - Modify the python scripts to set PATH to the location of the DLLs.
>>>> Alternatively apply any of the other solutions as mentioned here
>>>
>>> The solution is to put all .dlls on the bin directory, next to the
>>> executables.
>>>
>>> And I think that the
>>>
>>>     set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${piglit_BINARY_DIR}/bin)
>>>
>> I'm not sure if that will collide with the install() directive below,
>> thus the notes and contingency plan :P
>>
>>> on piglit's top CMakeLists.txt.
>>>
>>> That said, I think I'd like first to get piglit + waffle on windows
>>> wokring reliably, before DLL'ifying the libraries.  This way I don't
>>> have to worry about testing with glut.
>>>
>> DLL'ifying the libraries is not related to using glut/waffle. Thus one
>> can just have it ticked off (confirm that it works ok), for either (or
>> both) builds and worry about the waffle WGL bugs as we uncover them :)
>>
>> My current (waffle) builds are
>>   - Linux: (GCC default build) ~1150 files, 33MiB in size
>>   - Windows: (MSVC default build) ~1030 files, 4.9GiB in size.
>> ~70% of which are the debug symbols.
>>
>> So it's approx 45 times larger, when we ignore the debug symbols and
>> that Windows has ~10% tests less. Or 170 times if we consider it.
>>
>> Hope that's enough to convince you that it might be nice to have this
>> earlier rather than later :)
> 
> I don't mind to get it in once all problems are ironed out.  But your
> commit message highlitghted some rough edges.   Furthermore this is not
> working for me:
> 
> $ ninja -C build/mingw32
> ninja: Entering directory `build/mingw32'
> [120/2334] Generating builtin_uniform_tests.list
> /home/jfonseca/work/vmware/tests/piglit/generated_tests/builtin_function.py:1262:
> RuntimeWarning: overflow encountered in uint_scalars
>   f('op-assign-sub', 2, 110, lambda x, y: x - y,
> match_assignment_operators,
> /home/jfonseca/work/vmware/tests/piglit/generated_tests/builtin_function.py:1304:
> RuntimeWarning: overflow encountered in uint_scalars
>   f('op-sub', 2, 110, lambda x, y: x - y, match_simple_binop,
> [173/2334] Generating constant_array_size_tests.list
> /home/jfonseca/work/vmware/tests/piglit/generated_tests/builtin_function.py:1262:
> RuntimeWarning: overflow encountered in uint_scalars
>   f('op-assign-sub', 2, 110, lambda x, y: x - y,
> match_assignment_operators,
> /home/jfonseca/work/vmware/tests/piglit/generated_tests/builtin_function.py:1304:
> RuntimeWarning: overflow encountered in uint_scalars
>   f('op-sub', 2, 110, lambda x, y: x - y, match_simple_binop,
> [192/2334] Linking CXX executable bin/fbo-incomplete.exe
> FAILED: : && /usr/bin/i686-w64-mingw32-g++  -Wall -g  -static-libgcc
> -static-libstdc++
> target_api/gl/tests/fbo/CMakeFiles/fbo-incomplete.dir/fbo-incomplete.cpp.obj
>  -o bin/fbo-incomplete.exe
> -Wl,--out-implib,target_api/gl/tests/fbo/libfbo-incomplete.dll.a
> -Wl,--major-image-version,0,--minor-image-version,0
> /home/jfonseca/projects/sdk/mingw32/lib/libfreeglut.a
> target_api/gl/tests/util/libpiglitutil_gl.dll.a -lopengl32 -lglu32
> /home/jfonseca/projects/sdk/mingw32/lib/libfreeglut.a -lopengl32
> target_api/no_api/tests/util/libpiglitutil.dll.a -lkernel32 -luser32
> -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32
> -ladvapi32 && :
> /usr/lib/gcc/i686-w64-mingw32/4.6/libgcc_eh.a(unwind-sjlj.o):(.text+0x2e0):
> multiple definition of `__Unwind_SjLj_Register'
> target_api/gl/tests/util/libpiglitutil_gl.dll.a(d000023.o):(.text+0x0):
> first defined here
> /usr/lib/gcc/i686-w64-mingw32/4.6/libgcc_eh.a(unwind-sjlj.o):(.text+0x3a0):
> multiple definition of `__Unwind_SjLj_Unregister'
> target_api/gl/tests/util/libpiglitutil_gl.dll.a(d000026.o):(.text+0x0):
> first defined here
> /usr/lib/gcc/i686-w64-mingw32/4.6/libgcc_eh.a(unwind-sjlj.o):(.text+0x6a0):
> multiple definition of `__Unwind_SjLj_Resume'
> target_api/gl/tests/util/libpiglitutil_gl.dll.a(d000024.o):(.text+0x0):
> first defined here
> collect2: ld returned 1 exit status
> [192/2334] Building C object
> target_api/gl/tests/fbo/CMakeFiles/fbo-incomplete-texture-02.dir/fbo-incomplete-texture-02.c.obj
> 
> ninja: build stopped: subcommand failed.
> 
Seems like the util_gl library retains/exports the static gcc symbols
which later clash with the ones in the executable. A bug shamelessly
exposed by the patch. We might need to fiddle with the global
--static-lib{gcc,stdc++}.

> 
> I suspect that this patch series is insufficient on its own.  Just like
> when I tried to do this, we need to use the
> declspec(dllexport)/declspec(dllimort) attributes rigorously to avoid
> exporting too many or too little symbols.  And this is hard work, since
> for every DLL "foo" we need a "-DBUILDING_FOO" define while building,
> and then have in the header
> 
>   #ifdef BUILDING_FOO
>   #define FOO_EXPORT declspec(dllexport)
>   #else
>   #define FOO_EXPORT declspec(dllimport)
>   #endif
> 
> then prefix all exported simbols with FOO_EXPORT.
> 
The symbol mayhem is already there for non-windows builds, and is seems
to work fine. On a serious note, all we need is to prefix the few
remaining functions (where needed) and use a def file to hide anything
else :)

Properly resolving the exported functions and the transitive linking can
be sorted out at a later stage. Once it takes a reasonable about of time
to do the build iterations.

> 
> In short, I agree that shaving time from compile or space from binaries
> would be quite nice, but keeping things building and running without
> problems is critical, and I'm afraid that updating piglit to use DLLs on
> Windows won't be something that can be rushed.  And since I can't afford
> to neglect my other duties I must insist that we deal with one
> disruptive problem at a time -- waffle on Windows, or DLLs on Windows --
> but not both simultaneously.
> 
True. Silly me built only ~50 tests, thus I've not seen how sticky the
situation currently is. So I'll give this a couple full spins on my
Linux and Windows setup before bringing up the topic again.

Meanwhile if you do remember some of the "missing symbols" and/or other
blockers please let me know.

Thanks
Emil