[Mesa-dev] XOrg EVoC - OpenCL testing framework for piglit

[Blaž Tomažič]

On tor, 2012-05-29 at 15:07 -0400, Tom Stellard wrote:
> On Tue, May 29, 2012 at 10:14:06AM +0200, Blaž Tomažič wrote:
> > Hi,
> > 
> > I am applying for XOrg EVoC and have written a proposal for working on a
> > OpenCL testing framework for piglit. I am including project details I've
> > written in my proposal. I am looking for comments and suggestions.
> > 
> > Name:
> >         OpenCL testing framework for Piglit
> > 
> > Summary:
> >         For XOrg EVoC I would like to implement OpenCL testing support
> >         for Piglit testing framework. I am interested in this project
> >         because I like programming software that runs on GPUs and the
> >         APIs that are used in the process. I have an extensive
> >         programming knowledge and am a long time Linux user and
> >         developer.
> >         
> >         The project aims on bringing OpenCL API functions testing and
> >         support for arbitrary OpenCL kernel testing with predefined
> >         input and output. A part of the project will also be fixing
> >         CLover and Gallium3D drivers on detected invalid/missing OpenCL
> >         implementation or reporting to proper bug tracker, depending on
> >         the complexity of the problem.
> > 
> > Description:
> >         I would start with studying Piglit code and implementing basic
> >         framework for testing OpenCL conformance with Piglit. I don't
> >         plan to create most of the framework at the beginning, instead
> >         it would be extended as needed over the course of whole project.
> >         For all tests it should be possible to select platform and
> >         devices before running tests. The default action would be to
> >         test all platforms with all supported devices.After basic
> >         testing would be supported, so that it could run API tests, I
> >         would implement tests for OpenCL API functions. First I would
> >         create tests for functions that are context independent
> >         (platform and device info calls, error and get functions,...).
> >         Then I would continue with API functions for creating and
> >         manipulating context and memory objects. After that I would
> >         write tests for API functions involving kernels, memory objects
> >         and queues. Here I would use very simple kernels because we
> >         would be testing the API and not the kernels.Next step would be
> >         to implement support for arbitrary OpenCL kernel testing. First
> >         stage would be to implement kernels with hard-coded input (if
> >         any) that are concentrated on detecting compiler conformance
> >         (variable types, reserved words, different types of memory,
> >         casts, vector/scalar operations,...). In the next kernel stage I
> >         would add support for executing kernels with predefined
> >         arbitrary input and comparing it to expected output. An optional
> >         feature would be to add support for running kernels with random
> >         input data and comparing the results to non-OpenCL
> >         implementation (e.g. running on CPU).After most of the framework
> >         functionality would be implemented I would integrate the tests
> >         into Piglit's HTML result viewer.
> >         
> >         Over the course of the project (mostly in the later development)
> >         I plan to implement/fix as much missing and invalid
> >         functionality from CLover and Gallium3D drivers supporting
> >         OpenCL as time will allow. For functionality that is to large or
> >         complex to implement on my own I would report it to the proper
> >         bug tracker.
> >         
> >         At the end of the project I would review all the code, improve
> >         its readability, add comments where missing and create
> >         documentation.
> > 
> > Benefits:
> >         Community will have an open-source tool for OpenCL conformance
> >         testing and regression tracking for any OpenCL platform.
> > 
> > Schedule:
> >         On this project I plan to work full-time for 12-16 weeks. The
> >         proposed 12 week plan is optimistic, but if some step will take
> >         more then expected it can be extended to 16 weeks (or more if I
> >         will have time). The proposed schedule is as follows:
> > 
> >       * Week 1: Study Piglit's design. Start implementing utilities to 
> >         support OpenCL testing.
> >
> I've started writing a few utilities for testing OpenCL.  The code is
> here: http://cgit.freedesktop.org/~tstellar/opencl-example/
> You might want to look at that and see if there is anything you can
> reuse.
> 
> >       * Week 2: Continue implementing utilities for the framework.
> >         Implement tests for context independent API functions.
> > 
> >       * Week 3-4: Add support for choosing platform and devices.
> >         Implement API tests for manipulating context, memory objects and
> >         queues. Implement tests for API functions that depend on kernels
> >         (with simple or empty kernel). Extend framework as needed.
> > 
> >       * Week 5: Extend framework to compile and execute an arbitrary
> >         kernel.
> > 
> >       * Week 6: Write a lot of kernels to test compiler conformance.
> 
> The git repository mentioned above also has a few compiler conformance
> tests you might be able to use.  See the run_tests.sh script.
> 
> > 
> >       * Week 7: Extend framework to support predefined inputs and
> >         outputs for kernels. Add support for defining how kernels are
> >         executed (data-parallel/task-parallel, number of dimensions,
> >         work-group size)
> > 
> >       * Week 8-9: Write a lot of kernel tests to test for all kinds of
> >         possible errors.
> > 
> >       * Week 10-11: Write more kernel and API tests, fix simple bugs in
> >         CLover and Gallium3D drivers, report complex bugs. Integrate
> >         tests into Piglit's HTML result viewer.
> 
> You should try to integrating with the HTML result viewer from
> the very beginning.  I think this is actually trivial to do.
> 
> > 
> >       * Week 12: Clear up all the code, improve code readability,
> >         improve documentation.
> > 
> > 
> 
> Overall, this proposal looks good to me.  As Francisco mentioned it would
> also be a good idea to look at the previous OpenCL work done for piglit
> to see what you can reuse.

I will look up all the links you've given me and try to reuse as much as
is possible.

Blaž