Building for ARM systems

[Dan Nicholson]

On Tue, Mar 14, 2017 at 12:40 PM, Aleix Pol <aleixpol at kde.org> wrote:
> On Tue, Mar 14, 2017 at 5:54 PM, Dan Nicholson <nicholson at endlessm.com> wrote:
>> On Tue, Mar 14, 2017 at 8:58 AM, Aleix Pol <aleixpol at kde.org> wrote:
>>> On Tue, Mar 14, 2017 at 11:54 AM, Dan Nicholson <nicholson at endlessm.com> wrote:
>>>> On Mar 14, 2017 3:08 AM, "Alexander Larsson" <alexl at redhat.com> wrote:
>>>>
>>>> On Mon, 2017-03-13 at 22:31 -0500, Dan Nicholson wrote:
>>>>> On Mar 13, 2017 9:19 PM, "Aleix Pol" <aleixpol at kde.org> wrote:
>>>>>
>>>>> $ flatpak build sdk bash
>>>>> execvp bash: No such file or directory
>>>>> $ cat /proc/sys/fs/binfmt_misc/status
>>>>> enabled
>>>>> $ ls -l /proc/sys/fs/binfmt_misc/qemu-arm
>>>>> -rw-r--r-- 1 root root 0 mar 13 14:19 /proc/sys/fs/binfmt_misc/qemu-
>>>>> arm
>>>>>
>>>>> Can you show the contents of that qemu-arm file?
>>>>
>>>> Yeah, it seems like it is dynamically linked and is not finding the
>>>> loader.
>>>>
>>>>
>>>> Nah, I just went through this debugging with someone at Endless yesterday.
>>>> The kernel unfortunately returns ENOENT when it can't handle the binary
>>>> format. Or possibly the ENOENT is because it can't find the interpreter for
>>>> the binfmt. Either way, I'm pretty sure it's not a dynamic linking problem
>>>> unless Aleix just installed the wrong package.
>>>>
>>>> What I suspect is actually the problem is that Debian's qemu-user-static
>>>> package is setting a restrictive mask in the binfmt configuration that
>>>> doesn't have the new F flag needed so that the interpreter is opened
>>>> immediately at registration and kept open. I noticed yesterday that the
>>>> setup on xenial is that the flags are set to OC. I'm guessing this hasn't
>>>> been updated and can't really since it would make the configurations in the
>>>> package dependent on a pretty new kernel. I believe the kernel throws EINVAL
>>>> when an unrecognized option is in the mask.
>>>>
>>>> What Fedora does (I'm pretty sure), and what I think Debian should do, is
>>>> leave the mask blank so the kernel defaults are used. Then you pick up the
>>>> new F flag automatically when the kernel supports it.
>>>>
>>>> So, I'd like to see the actual registered configuration. If it contains OC
>>>> like I suspect, then it needs to be updated to include F or just have the
>>>> mask stripped off and use the kernel defaults.
>>>>
>>>> --
>>>> Dan
>>>
>>> It does include OC, not that I know what it means ^^'.
>>>
>>> $ cat /proc/sys/fs/binfmt_misc/qemu-arm
>>> enabled
>>> interpreter /usr/bin/qemu-arm-static
>>> flags: OC
>>> offset 0
>>> magic 7f454c4601010100000000000000000002002800
>>> mask ffffffffffffff00fffffffffffffffffeffffff
>>
>> Documentation for the entries is at
>> https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html.
>> Here's a temporary workaround you can try where you're clearing the
>> flags so you use the kernel defaults (which should include F on your
>> yakkety kernel).
>>
>> # echo -1 > /proc/sys/fs/binfmt_misc/qemu-arm
>> # echo ':qemu-arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:'
>>> /proc/sys/fs/binfmt_misc/register
>>
>> After this, /proc/sys/fs/binfmt_misc/qemu-arm should show an empty
>> flags, and I bet things work.
>>
>> If you want to make this more permanent, remove the binfmt support
>> file from qemu-user-static and use systemd's binfmt support:
>>
>> # update-binfmts --package qemu-user-static --remove qemu-arm
>> /usr/bin/qemu-arm-static
>> # echo ':qemu-arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:'
>>> /etc/binfmt.d/qemu-arm.conf
>>
>> This should really be fixed in debian's qemu-user-static (drop
>> --credential yes in call to update-binfmts) or binfmt-support (support
>> F flag with some --option) packages.
>
> I'm still getting the same error and now I don't have a
> /proc/sys/fs/binfmt_misc/qemu-arm :(
> (tried both approaches).

Without the configuration, you won't get anywhere. This obviously all
needs to run as root, but you should be able to manually setup the
binfmt via sudo like:

$ echo ':qemu-arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:'
| sudo tee /proc/sys/fs/binfmt_misc/register

If you setup the conf file in /etc/binfmt.d/qemu-arm.conf, then "sudo
systemctl restart systemd-binfmt".

I know this approach works because I have a similarly hacked setup on xenial.

That said, if you install qemu-user-static on recent fedora, it will
just work like Bastien says since the Fedora binfmts don't include any
flags - http://pkgs.fedoraproject.org/cgit/rpms/qemu.git/tree/qemu.binfmt.