[Mesa-dev] XOrg EVoC - OpenCL testing framework for piglit

[Blaž Tomažič]

Hi,

I am applying for XOrg EVoC and have written a proposal for working on a
OpenCL testing framework for Piglit. I include in this e-mail project
details I have written in my proposal. I am looking for comments and
suggestions.

------------------------------------------------------------------------
Name:
        OpenCL testing framework for Piglit

Summary:
        For XOrg EVoC I would like to implement OpenCL testing support
        for Piglit testing framework. I am interested in this project
        because I like programming software that runs on GPUs and the
        APIs that are used in the process. I have an extensive
        programming knowledge and am a long time Linux user and
        developer.
        
        The project aims on bringing OpenCL API functions testing and
        support for arbitrary OpenCL kernel testing with predefined
        input and output. A part of the project will also be fixing
        CLover and Gallium3D drivers on detected invalid/missing OpenCL
        implementation or reporting to proper bug tracker, depending on
        the complexity of the problem.

Description:
        I would start with studying Piglit code and implementing basic
        framework for testing OpenCL conformance with Piglit. I don't
        plan to create most of the framework at the beginning, instead
        it would be extended as needed over the course of whole project.
        For all tests it should be possible to select platform and
        devices before running tests. The default action would be to
        test all platforms with all supported devices.
        
        After basic testing would be supported, so that it could run API
        tests, I would implement tests for OpenCL API functions. First I
        would create tests for functions that are context independent
        (platform and device info calls, error and get functions,...).
        Then I would continue with API functions for creating and
        manipulating context and memory objects. After that I would
        write tests for API functions involving kernels, memory objects
        and queues. Here I would use very simple kernels because we
        would be testing the API and not the kernels.
        
        Next step would be to implement support for arbitrary OpenCL
        kernel testing. First stage would be to implement kernels with
        hard-coded input (if any) that are concentrated on detecting
        compiler conformance (variable types, reserved words, different
        types of memory, casts, vector/scalar operations,...). In the
        next kernel stage I would add support for executing kernels with
        predefined arbitrary input and comparing it to expected output.
        An optional feature would be to add support for running kernels
        with random input data and comparing the results to non-OpenCL
        implementation (e.g. running on CPU).
        
        After most of the framework functionality would be implemented I
        would integrate the tests into Piglit's HTML result viewer.
        
        Over the course of the project (mostly in the later development)
        I plan to implement/fix as much missing and invalid
        functionality from CLover and Gallium3D drivers supporting
        OpenCL as time will allow. For functionality that is to large or
        complex to implement on my own I would report it to the proper
        bug tracker.
        
        At the end of the project I would review all the code, improve
        its readability, add comments where missing and create
        documentation.

Benefits:
        Community will have an open-source tool for OpenCL conformance
        testing and regression tracking for any OpenCL platform.

Schedule:
        On this project I plan to work full-time for 12-16 weeks. The
        proposed 12 week plan is optimistic, but if some step will take
        more then expected it can be extended to 16 weeks (or more if I
        will have time). The proposed schedule is as follows:

              * Week 1: Study Piglit's design. Start implementing
                utilities to 
                support OpenCL testing.

              * Week 2: Continue implementing utilities for the
                framework. Implement tests for context independent API
                functions.

              * Week 3-4: Add support for choosing platform and devices.
                Implement API tests for manipulating context, memory
                objects and queues. Implement tests for API functions
                that depend on kernels (with simple or empty kernel).
                Extend framework as needed.

              * Week 5: Extend framework to compile and execute an
                arbitrary kernel.

              * Week 6: Write a lot of kernels to test compiler
                conformance.

              * Week 7: Extend framework to support predefined inputs
                and outputs for kernels. Add support for defining how
                kernels are executed (data-parallel/task-parallel,
                number of dimensions, work-group size)

              * Week 8-9: Write a lot of kernel tests to test for all
                kinds of possible errors.

              * Week 10-11: Write more kernel and API tests, fix simple
                bugs in CLover and Gallium3D drivers, report complex
                bugs. Integrate tests into Piglit's HTML result viewer.

              * Week 12: Clear up all the code, improve code
                readability, improve documentation.
------------------------------------------------------------------------

Blaž Tomažič