Some further testing shows that the LDFLAGS seem to cause the problem. More specifically the '--as-needed' which is passed to the linker. I don't know how to fix this, but I figure it could become a problem for a lot more users since more and more distros seem to use '--as-needed' by default.