[Mesa-dev] XOrg EVoC - OpenCL testing framework for piglit

[Blaž Tomažič]

Hi,

I am applying for XOrg EVoC and have written a proposal for working on a
OpenCL testing framework for piglit. I am including project details I've
written in my proposal. I am looking for comments and suggestions.

Name:
        OpenCL testing framework for Piglit

Summary:
        For XOrg EVoC I would like to implement OpenCL testing support
        for Piglit testing framework. I am interested in this project
        because I like programming software that runs on GPUs and the
        APIs that are used in the process. I have an extensive
        programming knowledge and am a long time Linux user and
        developer.
        
        The project aims on bringing OpenCL API functions testing and
        support for arbitrary OpenCL kernel testing with predefined
        input and output. A part of the project will also be fixing
        CLover and Gallium3D drivers on detected invalid/missing OpenCL
        implementation or reporting to proper bug tracker, depending on
        the complexity of the problem.

Description:
        I would start with studying Piglit code and implementing basic
        framework for testing OpenCL conformance with Piglit. I don't
        plan to create most of the framework at the beginning, instead
        it would be extended as needed over the course of whole project.
        For all tests it should be possible to select platform and
        devices before running tests. The default action would be to
        test all platforms with all supported devices.After basic
        testing would be supported, so that it could run API tests, I
        would implement tests for OpenCL API functions. First I would
        create tests for functions that are context independent
        (platform and device info calls, error and get functions,...).
        Then I would continue with API functions for creating and
        manipulating context and memory objects. After that I would
        write tests for API functions involving kernels, memory objects
        and queues. Here I would use very simple kernels because we
        would be testing the API and not the kernels.Next step would be
        to implement support for arbitrary OpenCL kernel testing. First
        stage would be to implement kernels with hard-coded input (if
        any) that are concentrated on detecting compiler conformance
        (variable types, reserved words, different types of memory,
        casts, vector/scalar operations,...). In the next kernel stage I
        would add support for executing kernels with predefined
        arbitrary input and comparing it to expected output. An optional
        feature would be to add support for running kernels with random
        input data and comparing the results to non-OpenCL
        implementation (e.g. running on CPU).After most of the framework
        functionality would be implemented I would integrate the tests
        into Piglit's HTML result viewer.
        
        Over the course of the project (mostly in the later development)
        I plan to implement/fix as much missing and invalid
        functionality from CLover and Gallium3D drivers supporting
        OpenCL as time will allow. For functionality that is to large or
        complex to implement on my own I would report it to the proper
        bug tracker.
        
        At the end of the project I would review all the code, improve
        its readability, add comments where missing and create
        documentation.

Benefits:
        Community will have an open-source tool for OpenCL conformance
        testing and regression tracking for any OpenCL platform.

Schedule:
        On this project I plan to work full-time for 12-16 weeks. The
        proposed 12 week plan is optimistic, but if some step will take
        more then expected it can be extended to 16 weeks (or more if I
        will have time). The proposed schedule is as follows:

      * Week 1: Study Piglit's design. Start implementing utilities to 
        support OpenCL testing.

      * Week 2: Continue implementing utilities for the framework.
        Implement tests for context independent API functions.

      * Week 3-4: Add support for choosing platform and devices.
        Implement API tests for manipulating context, memory objects and
        queues. Implement tests for API functions that depend on kernels
        (with simple or empty kernel). Extend framework as needed.

      * Week 5: Extend framework to compile and execute an arbitrary
        kernel.

      * Week 6: Write a lot of kernels to test compiler conformance.

      * Week 7: Extend framework to support predefined inputs and
        outputs for kernels. Add support for defining how kernels are
        executed (data-parallel/task-parallel, number of dimensions,
        work-group size)

      * Week 8-9: Write a lot of kernel tests to test for all kinds of
        possible errors.

      * Week 10-11: Write more kernel and API tests, fix simple bugs in
        CLover and Gallium3D drivers, report complex bugs. Integrate
        tests into Piglit's HTML result viewer.

      * Week 12: Clear up all the code, improve code readability,
        improve documentation.