[Mesa-dev] XOrg EVoC - OpenCL testing framework for piglit

[Francisco Jerez]

Blaž Tomažič <blaz.tomazic at gmail.com> writes:

> Hi,
>
> I am applying for XOrg EVoC and have written a proposal for working on a
> OpenCL testing framework for piglit. I am including project details I've
> written in my proposal. I am looking for comments and suggestions.
>
> Name:
>         OpenCL testing framework for Piglit
>
> Summary:
>         For XOrg EVoC I would like to implement OpenCL testing support
>         for Piglit testing framework. I am interested in this project
>         because I like programming software that runs on GPUs and the
>         APIs that are used in the process. I have an extensive
>         programming knowledge and am a long time Linux user and
>         developer.
>         
>         The project aims on bringing OpenCL API functions testing and
>         support for arbitrary OpenCL kernel testing with predefined
>         input and output. A part of the project will also be fixing
>         CLover and Gallium3D drivers on detected invalid/missing OpenCL
>         implementation or reporting to proper bug tracker, depending on
>         the complexity of the problem.
>
> Description:
>         I would start with studying Piglit code and implementing basic
>         framework for testing OpenCL conformance with Piglit. I don't
>         plan to create most of the framework at the beginning, instead
>         it would be extended as needed over the course of whole project.
>         For all tests it should be possible to select platform and
>         devices before running tests. The default action would be to
>         test all platforms with all supported devices.After basic
>         testing would be supported, so that it could run API tests, I
>         would implement tests for OpenCL API functions. First I would
>         create tests for functions that are context independent
>         (platform and device info calls, error and get functions,...).
>         Then I would continue with API functions for creating and
>         manipulating context and memory objects. After that I would
>         write tests for API functions involving kernels, memory objects
>         and queues. Here I would use very simple kernels because we
>         would be testing the API and not the kernels.Next step would be
>         to implement support for arbitrary OpenCL kernel testing. First
>         stage would be to implement kernels with hard-coded input (if
>         any) that are concentrated on detecting compiler conformance
>         (variable types, reserved words, different types of memory,
>         casts, vector/scalar operations,...). In the next kernel stage I
>         would add support for executing kernels with predefined
>         arbitrary input and comparing it to expected output. An optional
>         feature would be to add support for running kernels with random
>         input data and comparing the results to non-OpenCL
>         implementation (e.g. running on CPU).After most of the framework
>         functionality would be implemented I would integrate the tests
>         into Piglit's HTML result viewer.
>         
>         Over the course of the project (mostly in the later development)
>         I plan to implement/fix as much missing and invalid
>         functionality from CLover and Gallium3D drivers supporting
>         OpenCL as time will allow. For functionality that is to large or
>         complex to implement on my own I would report it to the proper
>         bug tracker.
>         
>         At the end of the project I would review all the code, improve
>         its readability, add comments where missing and create
>         documentation.
>
> Benefits:
>         Community will have an open-source tool for OpenCL conformance
>         testing and regression tracking for any OpenCL platform.
>
> Schedule:
>         On this project I plan to work full-time for 12-16 weeks. The
>         proposed 12 week plan is optimistic, but if some step will take
>         more then expected it can be extended to 16 weeks (or more if I
>         will have time). The proposed schedule is as follows:
>
>       * Week 1: Study Piglit's design. Start implementing utilities to 
>         support OpenCL testing.
>
>       * Week 2: Continue implementing utilities for the framework.
>         Implement tests for context independent API functions.
>
>       * Week 3-4: Add support for choosing platform and devices.
>         Implement API tests for manipulating context, memory objects and
>         queues. Implement tests for API functions that depend on kernels
>         (with simple or empty kernel). Extend framework as needed.
>
>       * Week 5: Extend framework to compile and execute an arbitrary
>         kernel.
>
>       * Week 6: Write a lot of kernels to test compiler conformance.
>
>       * Week 7: Extend framework to support predefined inputs and
>         outputs for kernels. Add support for defining how kernels are
>         executed (data-parallel/task-parallel, number of dimensions,
>         work-group size)
>
>       * Week 8-9: Write a lot of kernel tests to test for all kinds of
>         possible errors.
>
>       * Week 10-11: Write more kernel and API tests, fix simple bugs in
>         CLover and Gallium3D drivers, report complex bugs. Integrate
>         tests into Piglit's HTML result viewer.
>
>       * Week 12: Clear up all the code, improve code readability,
>         improve documentation.
>

This looks good.  Though I think someone already started to make a CL
test framework out of piglit [1], it might be worth to have a look at
what has already been done, I'm CC'ing the author for the case he can
tell you more.

You should also have a look at the gallium unit tests for the compute
interface [2], I've been using them until now to test the nv50 compute
implementation.  It probably makes sense on its own to have a batch of
unit tests at the gallium API level, but I think some of them could be
moved one layer up and translated from TGSI to OpenCL so they can run on
r600, that doesn't understand TGSI compute programs.

[1] https://github.com/rakadam/piglit
[2] http://cgit.freedesktop.org/mesa/mesa/tree/src/gallium/tests/trivial/compute.c
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 229 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/mesa-dev/attachments/20120529/e5a43370/attachment.pgp>