[Libreoffice] DLL rebasing on Windows: WTF?

[Tor Lillqvist]

I wonder if there is anybody among us who knows what actually is the rationale for the rebasing acrobatics that OOo (and thus LibreOffice) does.

(This investigation and message was prompted by bug report fdo#36679, https://bugs.freedesktop.org/show_bug.cgi?id=36679 )

Each Windows executable image (PE file), i.e. .exe or .dll, has in its IMAGE_OPTIONAL_HEADER header (which despite its name is not really optional) the ImageBase field which gives the preferred address at which the executable should be loaded in the address space of a process. This is normally just a hint, though, normally both .exe and .dll files are relocatable, they can and will be relocated if the hinted region of memory is not available. This of course seldom happens for .exe files, the .exe file is after all the first executable image to be loaded into the address space of a fresh process. (Well, ntdll.dll or whatever that contains the actual dynamic loader might be there first, but anyway, the address space if fairly empty when the .exe is loaded.)

The base address of an existing image file can be changed, either using the command-line rebase tool (part of the Windows SDK), or programmatically using the ReBaseImage() function.

On Vista and newer, if the IMAGE_OPTIONAL_HEADER's DllCharacteristics has the IMAGE_DLLCHARACTERISTICS_DYNAMIC_BASE flag, the OS is supposed to ignore the ImageBase hint and instead load the executable at a random address. ("Address space layout randomization") This is a security measure to make it harder for malware to assume code from a loaded DLL is in a known location.

LibreOffice (and OOo) uses the -dynamicbase linker flag which sets this flag in the created executable headers. No preferred base address is specified when linking, they all get the same default base address (0x10000000).

Then, in the penultimate module of the in build process, in postprocess/rebase the rebase.pl script goes through fairly significant pain to rebase each produced DLL. It sets the base address of each LibreOffice DLL (using the rebase command-line tool) so that they don't overlap. OK, the rationale for this is easy to understand, it should make the loading of DLLs into the soffice.bin process faster if no relocation is needed, right?

But wait a moment, we use the -dynamicbase linker switch, so we actually want Windows (well, current ones, Vista and newer) to choose a random base address for each DLL each time it gets loaded, right? Why then bother tediously calculating non-overlapping base addresses for them? Just for XP? OK, I guess that makes sense, many people still run XP.

But... it gets more fun. In the OOo/LO installer, there is a custom action that runs the code in setup_native/source/win32/customactions/rebase/rebase.cxx .

This custom action rebases each DLL back to the default base address, 0x10000000 . This is not done on "server" Windows OSes, for some reason, but obviously the vast majority of our Windows users don't run LibreOffice on a server OS. So the rebasing done in postprocess was mostly pointless as it is reverted on installation anyhow. Are we having fun yet?

Then, to get back to fdo#36679, there are two programs shipped with OOo/LO: rebasegui.exe and rebaseoo.exe. Their source code is in desktop/win32/source/rebase/rebasegui.cxx and rebase.cxx respectively. (It is rebasegui.exe that some quality virus scanner thinks contains a trojan.)

Despite its name, rebasegui.exe doesn't actually do any rebasing. It just takes a read-only look inside a couple of the OOo/LO DLLs (libxml2.dll, vclmi.dll and sal3.dll) and checks if they have the default 0x10000000 base address. The intent is that if one of them has another base address, rebasegui.exe displays the message "Installation is optimized for servers", otherwise "Installation is optimized for clients"

OK, so that sounds as if it would match what the custom action has done. (Even if it still doesn't explain *why* on normal non-server machines the rebasing done at build time is reverted when installing.) rebasegui.exe apparently is a tool to display whether the custom action has done the rebasing or not.

But... it gets even more fun. rebasegui.cxx code has a couple of bugs:

1) It looks for vclmi.dll under the "basis layer" or whatever it is called, in <installation>\basis\program\vclmi.dll. But vclmi.dll is in fact in the "brand" layer (as I think it is called), in <installation>\program\vclmi.dll.

2) It looks for an URE-link file to point to the "ure layer", but unlike basis-link for basis, there is no longer any ure-link file that would contain the actual path to the URE. sal3.dll is located in <installation>\URE\bin\sal3.dll.

So, rebasegui.exe actually fails to open two of the three DLLs it wants to have a peek in. Does it display any error message about this? Bah. It just silently proceeds as if the base address of those DLLs would be zero instead (and thus unequal to 0x10000000), and in the end thus displays the same message "Installation is optimized for servers" in all cases, regardless what the base address of the single DLL it actually manages to open, libxml2.dll, is.

In addition to this fairly useless rebasegui.exe program, there is also rebaseoo.exe, whose intent seems to be to set the base address of all OOo/LO DLls to that same default base address, 0x10000000. It too tries to use a URE-link file so it will for sure fail at least for the DLLs in URE. But what the point of this program is I don't understand either.

So, all this is a huge mystery and mess. The original intent of all this crack is known only to some people in Hamburg. I suggest we just get rid of all of this in LibreOffice. Let's not bother with any rebasing at build time, let's not do any rebasing back to the same default address at installation time, let's not build or distribute the rebaseoo.exe and rebasegui.exe programs.

--tml