Building for ARM systems

[Aleix Pol]

On Tue, Mar 14, 2017 at 7:23 PM, Dan Nicholson <nicholson at endlessm.com> wrote:
> On Tue, Mar 14, 2017 at 12:40 PM, Aleix Pol <aleixpol at kde.org> wrote:
>> On Tue, Mar 14, 2017 at 5:54 PM, Dan Nicholson <nicholson at endlessm.com> wrote:
>>> On Tue, Mar 14, 2017 at 8:58 AM, Aleix Pol <aleixpol at kde.org> wrote:
>>>> On Tue, Mar 14, 2017 at 11:54 AM, Dan Nicholson <nicholson at endlessm.com> wrote:
>>>>> On Mar 14, 2017 3:08 AM, "Alexander Larsson" <alexl at redhat.com> wrote:
>>>>>
>>>>> On Mon, 2017-03-13 at 22:31 -0500, Dan Nicholson wrote:
>>>>>> On Mar 13, 2017 9:19 PM, "Aleix Pol" <aleixpol at kde.org> wrote:
>>>>>>
>>>>>> $ flatpak build sdk bash
>>>>>> execvp bash: No such file or directory
>>>>>> $ cat /proc/sys/fs/binfmt_misc/status
>>>>>> enabled
>>>>>> $ ls -l /proc/sys/fs/binfmt_misc/qemu-arm
>>>>>> -rw-r--r-- 1 root root 0 mar 13 14:19 /proc/sys/fs/binfmt_misc/qemu-
>>>>>> arm
>>>>>>
>>>>>> Can you show the contents of that qemu-arm file?
>>>>>
>>>>> Yeah, it seems like it is dynamically linked and is not finding the
>>>>> loader.
>>>>>
>>>>>
>>>>> Nah, I just went through this debugging with someone at Endless yesterday.
>>>>> The kernel unfortunately returns ENOENT when it can't handle the binary
>>>>> format. Or possibly the ENOENT is because it can't find the interpreter for
>>>>> the binfmt. Either way, I'm pretty sure it's not a dynamic linking problem
>>>>> unless Aleix just installed the wrong package.
>>>>>
>>>>> What I suspect is actually the problem is that Debian's qemu-user-static
>>>>> package is setting a restrictive mask in the binfmt configuration that
>>>>> doesn't have the new F flag needed so that the interpreter is opened
>>>>> immediately at registration and kept open. I noticed yesterday that the
>>>>> setup on xenial is that the flags are set to OC. I'm guessing this hasn't
>>>>> been updated and can't really since it would make the configurations in the
>>>>> package dependent on a pretty new kernel. I believe the kernel throws EINVAL
>>>>> when an unrecognized option is in the mask.
>>>>>
>>>>> What Fedora does (I'm pretty sure), and what I think Debian should do, is
>>>>> leave the mask blank so the kernel defaults are used. Then you pick up the
>>>>> new F flag automatically when the kernel supports it.
>>>>>
>>>>> So, I'd like to see the actual registered configuration. If it contains OC
>>>>> like I suspect, then it needs to be updated to include F or just have the
>>>>> mask stripped off and use the kernel defaults.
>>>>>
>>>>> --
>>>>> Dan
>>>>
>>>> It does include OC, not that I know what it means ^^'.
>>>>
>>>> $ cat /proc/sys/fs/binfmt_misc/qemu-arm
>>>> enabled
>>>> interpreter /usr/bin/qemu-arm-static
>>>> flags: OC
>>>> offset 0
>>>> magic 7f454c4601010100000000000000000002002800
>>>> mask ffffffffffffff00fffffffffffffffffeffffff
>>>
>>> Documentation for the entries is at
>>> https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html.
>>> Here's a temporary workaround you can try where you're clearing the
>>> flags so you use the kernel defaults (which should include F on your
>>> yakkety kernel).
>>>
>>> # echo -1 > /proc/sys/fs/binfmt_misc/qemu-arm
>>> # echo ':qemu-arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:'
>>>> /proc/sys/fs/binfmt_misc/register
>>>
>>> After this, /proc/sys/fs/binfmt_misc/qemu-arm should show an empty
>>> flags, and I bet things work.
>>>
>>> If you want to make this more permanent, remove the binfmt support
>>> file from qemu-user-static and use systemd's binfmt support:
>>>
>>> # update-binfmts --package qemu-user-static --remove qemu-arm
>>> /usr/bin/qemu-arm-static
>>> # echo ':qemu-arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:'
>>>> /etc/binfmt.d/qemu-arm.conf
>>>
>>> This should really be fixed in debian's qemu-user-static (drop
>>> --credential yes in call to update-binfmts) or binfmt-support (support
>>> F flag with some --option) packages.
>>
>> I'm still getting the same error and now I don't have a
>> /proc/sys/fs/binfmt_misc/qemu-arm :(
>> (tried both approaches).
>
> Without the configuration, you won't get anywhere. This obviously all
> needs to run as root, but you should be able to manually setup the
> binfmt via sudo like:
>
> $ echo ':qemu-arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:'
> | sudo tee /proc/sys/fs/binfmt_misc/register
>
> If you setup the conf file in /etc/binfmt.d/qemu-arm.conf, then "sudo
> systemctl restart systemd-binfmt".
>
> I know this approach works because I have a similarly hacked setup on xenial.
>
> That said, if you install qemu-user-static on recent fedora, it will
> just work like Bastien says since the Fedora binfmts don't include any
> flags - http://pkgs.fedoraproject.org/cgit/rpms/qemu.git/tree/qemu.binfmt.

Hi Dan,
Thanks for the patience.

I've been trying it for a bit more, I am getting these contents now,
still getting the same execvp error message.

# cat /proc/sys/fs/binfmt_misc/qemu-arm
enabled
interpreter /usr/bin/qemu-arm-static
flags:
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff

I guess I'll have to go RT(F)M...

Aleix