[Mesa-dev] XOrg EVoC - OpenCL testing framework for piglit

[Tom Stellard]

On Tue, May 29, 2012 at 10:14:06AM +0200, Blaž Tomažič wrote:
> Hi,
> 
> I am applying for XOrg EVoC and have written a proposal for working on a
> OpenCL testing framework for piglit. I am including project details I've
> written in my proposal. I am looking for comments and suggestions.
> 
> Name:
>         OpenCL testing framework for Piglit
> 
> Summary:
>         For XOrg EVoC I would like to implement OpenCL testing support
>         for Piglit testing framework. I am interested in this project
>         because I like programming software that runs on GPUs and the
>         APIs that are used in the process. I have an extensive
>         programming knowledge and am a long time Linux user and
>         developer.
>         
>         The project aims on bringing OpenCL API functions testing and
>         support for arbitrary OpenCL kernel testing with predefined
>         input and output. A part of the project will also be fixing
>         CLover and Gallium3D drivers on detected invalid/missing OpenCL
>         implementation or reporting to proper bug tracker, depending on
>         the complexity of the problem.
> 
> Description:
>         I would start with studying Piglit code and implementing basic
>         framework for testing OpenCL conformance with Piglit. I don't
>         plan to create most of the framework at the beginning, instead
>         it would be extended as needed over the course of whole project.
>         For all tests it should be possible to select platform and
>         devices before running tests. The default action would be to
>         test all platforms with all supported devices.After basic
>         testing would be supported, so that it could run API tests, I
>         would implement tests for OpenCL API functions. First I would
>         create tests for functions that are context independent
>         (platform and device info calls, error and get functions,...).
>         Then I would continue with API functions for creating and
>         manipulating context and memory objects. After that I would
>         write tests for API functions involving kernels, memory objects
>         and queues. Here I would use very simple kernels because we
>         would be testing the API and not the kernels.Next step would be
>         to implement support for arbitrary OpenCL kernel testing. First
>         stage would be to implement kernels with hard-coded input (if
>         any) that are concentrated on detecting compiler conformance
>         (variable types, reserved words, different types of memory,
>         casts, vector/scalar operations,...). In the next kernel stage I
>         would add support for executing kernels with predefined
>         arbitrary input and comparing it to expected output. An optional
>         feature would be to add support for running kernels with random
>         input data and comparing the results to non-OpenCL
>         implementation (e.g. running on CPU).After most of the framework
>         functionality would be implemented I would integrate the tests
>         into Piglit's HTML result viewer.
>         
>         Over the course of the project (mostly in the later development)
>         I plan to implement/fix as much missing and invalid
>         functionality from CLover and Gallium3D drivers supporting
>         OpenCL as time will allow. For functionality that is to large or
>         complex to implement on my own I would report it to the proper
>         bug tracker.
>         
>         At the end of the project I would review all the code, improve
>         its readability, add comments where missing and create
>         documentation.
> 
> Benefits:
>         Community will have an open-source tool for OpenCL conformance
>         testing and regression tracking for any OpenCL platform.
> 
> Schedule:
>         On this project I plan to work full-time for 12-16 weeks. The
>         proposed 12 week plan is optimistic, but if some step will take
>         more then expected it can be extended to 16 weeks (or more if I
>         will have time). The proposed schedule is as follows:
> 
>       * Week 1: Study Piglit's design. Start implementing utilities to 
>         support OpenCL testing.
>
I've started writing a few utilities for testing OpenCL.  The code is
here: http://cgit.freedesktop.org/~tstellar/opencl-example/
You might want to look at that and see if there is anything you can
reuse.

>       * Week 2: Continue implementing utilities for the framework.
>         Implement tests for context independent API functions.
> 
>       * Week 3-4: Add support for choosing platform and devices.
>         Implement API tests for manipulating context, memory objects and
>         queues. Implement tests for API functions that depend on kernels
>         (with simple or empty kernel). Extend framework as needed.
> 
>       * Week 5: Extend framework to compile and execute an arbitrary
>         kernel.
> 
>       * Week 6: Write a lot of kernels to test compiler conformance.

The git repository mentioned above also has a few compiler conformance
tests you might be able to use.  See the run_tests.sh script.

> 
>       * Week 7: Extend framework to support predefined inputs and
>         outputs for kernels. Add support for defining how kernels are
>         executed (data-parallel/task-parallel, number of dimensions,
>         work-group size)
> 
>       * Week 8-9: Write a lot of kernel tests to test for all kinds of
>         possible errors.
> 
>       * Week 10-11: Write more kernel and API tests, fix simple bugs in
>         CLover and Gallium3D drivers, report complex bugs. Integrate
>         tests into Piglit's HTML result viewer.

You should try to integrating with the HTML result viewer from
the very beginning.  I think this is actually trivial to do.

> 
>       * Week 12: Clear up all the code, improve code readability,
>         improve documentation.
> 
> 

Overall, this proposal looks good to me.  As Francisco mentioned it would
also be a good idea to look at the previous OpenCL work done for piglit
to see what you can reuse.

-Tom

_______________________________________________
> mesa-dev mailing list
> mesa-dev at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/mesa-dev
>