[Mesa-dev] EXTERNAL: Re: OpenCL Clang/Clover Offline Compilation issue

[Dorrington, Albert]

Hi Tom and Francisco,

When I tried to use Clang from the command line to produce binaries, all I could get was the LLVM IR code, so I adapted my test program to produce a binary using clGetProgramInfo().
(I have been following code examples in book 'OpenCL Programming Guide')

I have been stepping through the existing code in this area, using GDB, for the past few days, trying to get the binary to load successfully, and I have also stepped through the code behind clCreateProgramWithSource() -  so I have started getting familiar with the process that is going on.

I thought, if I generated the binary using clGetProgramInfo() after clBuildProgram() that the binary would be in the same format as would be needed.

So far, I have run into two main issues.
The first is, if there is only one kernel in the binary, it seems that clCreateProgramWithBinary() thinks there are two, due to (I think) an issue with the range() processing.
In the debugger I see a second pair of binary/length fields in the result map, and when the 'return new program()' call is made at the end of clCreateProgramWithBinary() I get a SegFault after the first (only) binary is deserialized.

So, I added a second kernel function to the CL program, and I am able to get through clCreateProgramWithBinary() without crashing, but quickly ran into a second issue.

My code currently calls in the order:
	clCreateProgramWithBinary();
	clBuildProgram();
	clCreateKernel();
	
The clCreateKernel() call fails with a -46/Invalid Kernel Name; stepping through the debugger, I believe I can see the two loaded kernels, however I cannot find the names in what was loaded.

For now, I don't need Clang/LLVM to produce the binary without Mesa/Clover, I am fine with Mesa/Clover producing the binary.

Tom, from what you wrote below, it sounds like the clBuildProgram() implementation may only be expecting IR code and not a binary input?

Since getting this to function is related to my current assignment at work, I do have a lot of time I can spend on this task.

Thanks!
-Al

-----Original Message-----
From: Francisco Jerez [mailto:currojerez at riseup.net] 
Sent: Tuesday, January 14, 2014 4:06 AM
To: Tom Stellard; Dorrington, Albert
Cc: mesa-dev at lists.freedesktop.org
Subject: Re: EXTERNAL: Re: [Mesa-dev] OpenCL Clang/Clover Offline Compilation issue

Tom Stellard <tom at stellard.net> writes:

> On Mon, Jan 13, 2014 at 06:44:15PM +0000, Dorrington, Albert wrote:
>> Tom,
>> 
>> Thanks for your response. I am very interested in implementing this, so any pointers you can provide would be greatly appreciated.
>
> I'm cc'ing Fransisco since he may also have some feedback.
>
> The first step is to build a clover::module object from the binary code.
> When we compile OpenCL C, we use the build_module_llvm() function in 
> llvm/invocation.cpp to do this.  This function takes LLVM IR as input 
> (stored in the LLVM:Module object) and produces a clover::module as 
> output.
>
> With clCreateProgramFromBinary() we build a clover::module by 
> deserializing the binary code using the module::deserialize function declared in module.cpp.
> This function expects the binary code to use a specific format, the 
> code that is output from Clang/LLVM is not in the expected format 
> which is probably why this is crashing for you.
>
> I don't think this format is documented anywhere, but you should be 
> able to deduce it by looking through the code in core/module.cpp.
> The challenge is to get Clang/LLVM to produce code in the correct format.
>
> I think the correct way to do this would be to add a new triple, 
> something like r600-clover-unknown, and then have the code emitter 
> produce clover formatted code when it is passed this triple.  However, 
> I would recommend not worrying about the triple for now and just 
> change the code emitter to emit clover's format.  Once this is 
> working, then we can go back and add the new triple.
>
> Once LLVM is producing the correct format, you will need to find a way 
> for clover to communicate to the drivers that the code being passed is 
> binary and not whatever its preferred IR is.  One way to do this is to 
> add the
>
> enum pipe_shader_ir ir_type;
>
> field to struct pipe_compute_state and use this to tell the drivers 
> what kind of IR it has.  You will also need to add the 
> PIPE_SHADER_IR_BINARY type to enum pipe_hsader_ir.
>
> Then you will need to implement support for PIPE_SHADER_IR_BINARY in r600g.
> The code for doing this is already their you will just need to add a 
> code path which skips over all of the LLVM compilation stages.
>
> Hopefully, this will help get you started.
>

Hi Tom,

I'm not sure if this makes sense to me.  Ideally programs created from source code and from binary would share the same driver path.  If there's a way to invoke clang that gets us a native GPU binary in the clover::module format we should always invoke it that way -- also when the program is specified using clCreateProgramWithSource().  If we want to stick to the two-stage compilation approach r600g uses now with a half-baked LLVM representation, we should expect the input of
clCreateProgramFromBinary() to be in the same format.  Mixing two program representations depending on the way the program is created sounds like a recipe for headache to me...

Albert, have you tried creating your binary using a CL client that calls
clBuildProgram() and then extracts the generated code with clGetProgramInfo(CL_PROGRAM_BINARIES)?  That's the only portable way to generate a binary for a given CL device, invoking clang manually is not
-- and as you've already realized it's not supposed to work right now.

Thanks.