[RFC] Mechanism for high priority scheduling in amdgpu

Fri Dec 23 16:20:03 UTC 2016

One question I just remembered - the amdgpu driver includes some scheduler logic which maintains per-process queues and therefore avoids loading up the primary ring with a ton of work.

Has there been any experimentation with injecting priorities at that level rather than jumping straight to HW-level changes ?

________________________________
From: amd-gfx <amd-gfx-bounces at lists.freedesktop.org> on behalf of Andres Rodriguez <andresx7 at gmail.com>
Sent: December 23, 2016 11:13 AM
To: Koenig, Christian
Cc: Zhou, David(ChunMing); Huan, Alvin; Mao, David; Sagalovitch, Serguei; amd-gfx at lists.freedesktop.org; Andres Rodriguez; Pierre-Loup A. Griffais; Zhang, Hawking
Subject: Re: [RFC] Mechanism for high priority scheduling in amdgpu

Hey Christian,

But yes, in general you don't want another compositor in the way, so we'll be acquiring the HMD display directly, separate from any desktop or display server.

Assuming that the the HMD is attached to the rendering device in some way you have the X server and the Compositor which both try to be DRM master at the same time.

Please correct me if that was fixed in the meantime, but that sounds like it will simply not work. Or is this what Andres mention below Dave is working on ?.

You are correct on both statements. We can't have two DRM_MASTERs, so the current DRM+X does not support this use case. And this what Dave and Pierre-Loup are currently working on.

Additional to that a compositor in combination with X is a bit counter productive when you want to keep the latency low.

One thing I'd like to correct is that our main goal is to get latency _predictable_, secondary goal is to make it low.

The high priority queue feature addresses our main source of unpredictability: the scheduling latency when the hardware is already full of work from the game engine.

The DirectMode feature addresses one of the latency sources: multiple (unnecessary) context switches to submit a surface to the DRM driver.

Targeting something like Wayland and when you need X compatibility XWayland sounds like the much better idea.

We are pretty enthusiastic about Wayland (and really glad to see Fedora 25 use Wayland by default). Once we have everything working nicely under X (where most of the users are currently), I'm sure Pierre-Loup will be pushing us to get everything optimized under Wayland as well (which should be a lot simpler!).

Ever since working with SurfaceFlinger on Android with explicit fencing I've been waiting for the day I can finally ditch X altogether :)

Regards,
Andres

On Fri, Dec 23, 2016 at 5:54 AM, Christian König <christian.koenig at amd.com<mailto:christian.koenig at amd.com>> wrote:
But yes, in general you don't want another compositor in the way, so we'll be acquiring the HMD display directly, separate from any desktop or display server.
Assuming that the the HMD is attached to the rendering device in some way you have the X server and the Compositor which both try to be DRM master at the same time.

Please correct me if that was fixed in the meantime, but that sounds like it will simply not work. Or is this what Andres mention below Dave is working on ?.

Additional to that a compositor in combination with X is a bit counter productive when you want to keep the latency low.

E.g. the "normal" flow of a GL or Vulkan surface filled with rendered data to be displayed is from the Application -> X server -> compositor -> X server.

The extra step between X server and compositor just means extra latency and for this use case you probably don't want that.

Targeting something like Wayland and when you need X compatibility XWayland sounds like the much better idea.

Regards,
Christian.

Am 22.12.2016 um 20:54 schrieb Pierre-Loup A. Griffais:
Display concerns are a separate issue, and as Andres said we have other plans to address. But yes, in general you don't want another compositor in the way, so we'll be acquiring the HMD display directly, separate from any desktop or display server. Same with security, we can have a separate conversation about that when the time comes.

On 12/22/2016 08:41 AM, Serguei Sagalovitch wrote:
Andres,

Did you measure  latency, etc. impact of __any__ compositor?

My understanding is that VR has pretty strict requirements related to QoS.

Sincerely yours,
Serguei Sagalovitch

On 2016-12-22 11:35 AM, Andres Rodriguez wrote:
Hey Christian,

We are currently interested in X, but with some distros switching to
other compositors by default, we also need to consider those.

We agree, running the full vrcompositor in root isn't something that
we want to do. Too many security concerns. Having a small root helper
that does the privilege escalation for us is the initial idea.

For a long term approach, Pierre-Loup and Dave are working on dealing
with the "two compositors" scenario a little better in DRM+X.
Fullscreen isn't really a sufficient approach, since we don't want the
HMD to be used as part of the Desktop environment when a VR app is not
in use (this is extremely annoying).

When the above is settled, we should have an auth mechanism besides
DRM_MASTER or DRM_AUTH that allows the vrcompositor to take over the
HMD permanently away from X. Re-using that auth method to gate this
IOCTL is probably going to be the final solution.

I propose to start with ROOT_ONLY since it should allow us to respect
kernel IOCTL compatibility guidelines with the most flexibility. Going
from a restrictive to a more flexible permission model would be
inclusive, but going from a general to a restrictive model may exclude
some apps that used to work.

Regards,
Andres

On 12/22/2016 6:42 AM, Christian König wrote:
Hi Andres,

well using root might cause stability and security problems as well.
We worked quite hard to avoid exactly this for X.

We could make this feature depend on the compositor being DRM master,
but for example with X the X server is master (and e.g. can change
resolutions etc..) and not the compositor.

So another question is also what windowing system (if any) are you
planning to use? X, Wayland, Flinger or something completely different ?

Regards,
Christian.

Am 20.12.2016 um 16:51 schrieb Andres Rodriguez:
Hi Christian,

That is definitely a concern. What we are currently thinking is to
make the high priority queues accessible to root only.

Therefore is a non-root user attempts to set the high priority flag
on context allocation, we would fail the call and return ENOPERM.

Regards,
Andres

On 12/20/2016 7:56 AM, Christian König wrote:
BTW: If there is  non-VR application which will use high-priority
h/w queue then VR application will suffer.  Any ideas how
to solve it?
Yeah, that problem came to my mind as well.

Basically we need to restrict those high priority submissions to
the VR compositor or otherwise any malfunctioning application could
use it.

Just think about some WebGL suddenly taking all our rendering away
and we won't get anything drawn any more.

Alex or Michel any ideas on that?

Regards,
Christian.

Am 19.12.2016 um 15:48 schrieb Serguei Sagalovitch:
> If compute queue is occupied only by you, the efficiency
> is equal with setting job queue to high priority I think.
The only risk is the situation when graphics will take all
needed CUs. But in any case it should be very good test.

Andres/Pierre-Loup,

Did you try to do it or it is a lot of work for you?

BTW: If there is  non-VR application which will use high-priority
h/w queue then VR application will suffer.  Any ideas how
to solve it?

Sincerely yours,
Serguei Sagalovitch

On 2016-12-19 12:50 AM, zhoucm1 wrote:
Do you encounter the priority issue for compute queue with
current driver?

If compute queue is occupied only by you, the efficiency is equal
with setting job queue to high priority I think.

Regards,
David Zhou

On 2016年12月19日 13:29, Andres Rodriguez wrote:
Yes, vulkan is available on all-open through the mesa radv UMD.

I'm not sure if I'm asking for too much, but if we can
coordinate a similar interface in radv and amdgpu-pro at the
vulkan level that would be great.

I'm not sure what that's going to be yet.

- Andres

On 12/19/2016 12:11 AM, zhoucm1 wrote:

On 2016年12月19日 11:33, Pierre-Loup A. Griffais wrote:
We're currently working with the open stack; I assume that a
mechanism could be exposed by both open and Pro Vulkan
userspace drivers and that the amdgpu kernel interface
improvements we would pursue following this discussion would
let both drivers take advantage of the feature, correct?
Of course.
Does open stack have Vulkan support?

Regards,
David Zhou

On 12/18/2016 07:26 PM, zhoucm1 wrote:
By the way, are you using all-open driver or amdgpu-pro driver?

+David Mao, who is working on our Vulkan driver.

Regards,
David Zhou

On 2016年12月18日 06:05, Pierre-Loup A. Griffais wrote:
Hi Serguei,

I'm also working on the bringing up our VR runtime on top of
amgpu;
see replies inline.

On 12/16/2016 09:05 PM, Sagalovitch, Serguei wrote:
Andres,

 For current VR workloads we have 3 separate processes
running
actually:
So we could have potential memory overcommit case or do you do
partitioning
on your own?  I would think that there is need to avoid
overcomit in
VR case to
prevent any BO migration.

You're entirely correct; currently the VR runtime is setting up
prioritized CPU scheduling for its VR compositor, we're
working on
prioritized GPU scheduling and pre-emption (eg. this
thread), and in
the future it will make sense to do work in order to make
sure that
its memory allocations do not get evicted, to prevent any
unwelcome
additional latency in the event of needing to perform
just-in-time
reprojection.

BTW: Do you mean __real__ processes or threads?
Based on my understanding sharing BOs between different
processes
could introduce additional synchronization constrains. btw:
I am not
sure
if we are able to share Vulkan sync. object cross-process
boundary.

They are different processes; it is important for the
compositor that
is responsible for quality-of-service features such as
consistently
presenting distorted frames with the right latency,
reprojection, etc,
to be separate from the main application.

Currently we are using unreleased cross-process memory and
semaphore
extensions to fetch updated eye images from the client
application,
but the just-in-time reprojection discussed here does not
actually
have any direct interactions with cross-process resource
sharing,
since it's achieved by using whatever is the latest, most
up-to-date
eye images that have already been sent by the client
application,
which are already available to use without additional
synchronization.

   3) System compositor (we are looking at approaches to
remove this
overhead)
Yes,  IMHO the best is to run in  "full screen mode".

Yes, we are working on mechanisms to present directly to the
headset
display without any intermediaries as a separate effort.

 The latency is our main concern,
I would assume that this is the known problem (at least for
compute
usage).
It looks like that amdgpu / kernel submission is rather CPU
intensive
(at least
in the default configuration).

As long as it's a consistent cost, it shouldn't an issue.
However, if
there's high degrees of variance then that would be
troublesome and we
would need to account for the worst case.

Hopefully the requirements and approach we described make
sense, we're
looking forward to your feedback and suggestions.

Thanks!
 - Pierre-Loup

Sincerely yours,
Serguei Sagalovitch

From: Andres Rodriguez <andresr at valvesoftware.com<mailto:andresr at valvesoftware.com>>
Sent: December 16, 2016 10:00 PM
To: Sagalovitch, Serguei; amd-gfx at lists.freedesktop.org<mailto:amd-gfx at lists.freedesktop.org>
Subject: RE: [RFC] Mechanism for high priority scheduling
in amdgpu

Hey Serguei,

[Serguei] No. I mean pipe :-) as MEC define it.  As far as I
understand (by simplifying)
some scheduling is per pipe.  I know about the current
allocation
scheme but I do not think
that it is  ideal.  I would assume that we need to switch to
dynamical partition
of resources  based on the workload otherwise we will have
resource
conflict
between Vulkan compute and  OpenCL.

I agree the partitioning isn't ideal. I'm hoping we can
start with a
solution that assumes that
only pipe0 has any work and the other pipes are idle (no
HSA/ROCm
running on the system).

This should be more or less the use case we expect from VR
users.

I agree the split is currently not ideal, but I'd like to
consider
that a separate task, because
making it dynamic is not straight forward :P

[Serguei] Vulkan works via amdgpu (kernel submissions) so
amdkfd
will be not
involved.  I would assume that in the case of VR we will
have one main
application ("console" mode(?)) so we could temporally
"ignore"
OpenCL/ROCm needs when VR is running.

Correct, this is why we want to enable the high priority
compute
queue through
libdrm-amdgpu, so that we can expose it through Vulkan later.

For current VR workloads we have 3 separate processes
running actually:
    1) Game process
    2) VR Compositor (this is the process that will require
high
priority queue)
    3) System compositor (we are looking at approaches to
remove this
overhead)

For now I think it is okay to assume no OpenCL/ROCm running
simultaneously, but
I would also like to be able to address this case in the
future
(cross-pipe priorities).

[Serguei]  The problem with pre-emption of graphics task:
(a) it
may take time so
latency may suffer

The latency is our main concern, we want something that is
predictable. A good
illustration of what the reprojection scheduling looks like
can be
found here:
https://community.amd.com/servlet/JiveServlet/showImage/38-1310-104754/pastedImage_3.png

(b) to preempt we need to have different "context" - we want
to guarantee that submissions from the same context will
be executed
in order.

This is okay, as the reprojection work doesn't have
dependencies on
the game context, and it
even happens in a separate process.

BTW: (a) Do you want "preempt" and later resume or do you
want
"preempt" and
"cancel/abort"

Preempt the game with the compositor task and then resume it.

(b) Vulkan is generic API and could be used for graphics
as well as
for plain compute tasks (VK_QUEUE_COMPUTE_BIT).

Yeah, the plan is to use vulkan compute. But if you figure
out a way
for us to get
a guaranteed execution time using vulkan graphics, then
I'll take you
out for a beer :)

Regards,
Andres
________________________________________
From: Sagalovitch, Serguei [Serguei.Sagalovitch at amd.com<mailto:Serguei.Sagalovitch at amd.com>]
Sent: Friday, December 16, 2016 9:13 PM
To: Andres Rodriguez; amd-gfx at lists.freedesktop.org<mailto:amd-gfx at lists.freedesktop.org>
Subject: Re: [RFC] Mechanism for high priority scheduling
in amdgpu

Hi Andres,

Please see inline (as [Serguei])

Sincerely yours,
Serguei Sagalovitch

From: Andres Rodriguez <andresr at valvesoftware.com<mailto:andresr at valvesoftware.com>>
Sent: December 16, 2016 8:29 PM
To: Sagalovitch, Serguei; amd-gfx at lists.freedesktop.org<mailto:amd-gfx at lists.freedesktop.org>
Subject: RE: [RFC] Mechanism for high priority scheduling
in amdgpu

Hi Serguei,

Thanks for the feedback. Answers inline as [AR].

Regards,
Andres

________________________________________
From: Sagalovitch, Serguei [Serguei.Sagalovitch at amd.com<mailto:Serguei.Sagalovitch at amd.com>]
Sent: Friday, December 16, 2016 8:15 PM
To: Andres Rodriguez; amd-gfx at lists.freedesktop.org<mailto:amd-gfx at lists.freedesktop.org>
Subject: Re: [RFC] Mechanism for high priority scheduling
in amdgpu

Andres,

Quick comments:

1) To minimize "bubbles", etc. we need to "force" CU
assignments/binding
to high-priority queue  when it will be in use and "free"
them later
(we  do not want forever take CUs from e.g. graphic task to
degrade
graphics
performance).

Otherwise we could have scenario when long graphics task (or
low-priority
compute) will took all (extra) CUs and high--priority will
wait for
needed resources.
It will not be visible on "NOP " but only when you submit
"real"
compute task
so I would recommend  not to use "NOP" packets at all for
testing.

It (CU assignment) could be relatively easy done when
everything is
going via kernel
(e.g. as part of frame submission) but I must admit that I
am not sure
about the best way for user level submissions (amdkfd).

[AR] I wasn't aware of this part of the programming
sequence. Thanks
for the heads up!
Is this similar to the CU masking programming?
[Serguei] Yes. To simplify: the problem is that "scheduler"
when
deciding which
queue to  run will check if there is enough resources and
if not then
it will begin
to check other queues with lower priority.

2) I would recommend to dedicate the whole pipe to
high-priority
queue and have
nothing their except it.

[AR] I'm guessing in this context you mean pipe = queue?
(as opposed
to the MEC definition
of pipe, which is a grouping of queues). I say this because
amdgpu
only has access to 1 pipe,
and the rest are statically partitioned for amdkfd usage.

[Serguei] No. I mean pipe :-)  as MEC define it. As far as I
understand (by simplifying)
some scheduling is per pipe.  I know about the current
allocation
scheme but I do not think
that it is  ideal.  I would assume that we need to switch to
dynamical partition
of resources  based on the workload otherwise we will have
resource
conflict
between Vulkan compute and  OpenCL.

BTW: Which user level API do you want to use for compute:
Vulkan or
OpenCL?

[AR] Vulkan

[Serguei] Vulkan works via amdgpu (kernel submissions) so
amdkfd will
be not
involved.  I would assume that in the case of VR we will
have one main
application ("console" mode(?)) so we could temporally
"ignore"
OpenCL/ROCm needs when VR is running.

 we will not be able to provide a solution compatible with
GFX
worloads.
I assume that you are talking about graphics? Am I right?

[AR] Yeah, my understanding is that pre-empting the
currently running
graphics job and scheduling in
something else using mid-buffer pre-emption has some cases
where it
doesn't work well. But if with
polaris10 it starts working well, it might be a better
solution for
us (because the whole reprojection
work uses the vulkan graphics stack at the moment, and
porting it to
compute is not trivial).

[Serguei]  The problem with pre-emption of graphics task:
(a) it may
take time so
latency may suffer (b) to preempt we need to have different
"context"
- we want
to guarantee that submissions from the same context will be
executed
in order.
BTW: (a) Do you want  "preempt" and later resume or do you
want
"preempt" and
"cancel/abort"?  (b) Vulkan is generic API and could be used
for graphics as well as for plain compute tasks
(VK_QUEUE_COMPUTE_BIT).

Sincerely yours,
Serguei Sagalovitch

From: amd-gfx <amd-gfx-bounces at lists.freedesktop.org<mailto:amd-gfx-bounces at lists.freedesktop.org>> on
behalf of
Andres Rodriguez <andresr at valvesoftware.com<mailto:andresr at valvesoftware.com>>
Sent: December 16, 2016 6:15 PM
To: amd-gfx at lists.freedesktop.org<mailto:amd-gfx at lists.freedesktop.org>
Subject: [RFC] Mechanism for high priority scheduling in
amdgpu

Hi Everyone,

This RFC is also available as a gist here:
https://gist.github.com/lostgoat/7000432cd6864265dbc2c3ab93204249

[RFC] Mechanism for high priority scheduling in amdgpu
gist.github.com<http://gist.github.com>
[RFC] Mechanism for high priority scheduling in amdgpu

[RFC] Mechanism for high priority scheduling in amdgpu
gist.github.com<http://gist.github.com>
[RFC] Mechanism for high priority scheduling in amdgpu

[RFC] Mechanism for high priority scheduling in amdgpu
gist.github.com<http://gist.github.com>
[RFC] Mechanism for high priority scheduling in amdgpu

We are interested in feedback for a mechanism to
effectively schedule
high
priority VR reprojection tasks (also referred to as
time-warping) for
Polaris10
running on the amdgpu kernel driver.

Brief context:
--------------

The main objective of reprojection is to avoid motion
sickness for VR
users in
scenarios where the game or application would fail to finish
rendering a new
frame in time for the next VBLANK. When this happens, the
user's head
movements
are not reflected on the Head Mounted Display (HMD) for the
duration
of an
extra frame. This extended mismatch between the inner ear
and the
eyes may
cause the user to experience motion sickness.

The VR compositor deals with this problem by fabricating a
new frame
using the
user's updated head position in combination with the
previous frames.
This
avoids a prolonged mismatch between the HMD output and the
inner ear.

Because of the adverse effects on the user, we require high
confidence that the
reprojection task will complete before the VBLANK interval.
Even if
the GFX pipe
is currently full of work from the game/application (which
is most
likely the case).

For more details and illustrations, please refer to the
following
document:
https://community.amd.com/community/gaming/blog/2016/03/28/asynchronous-shaders-evolved

Gaming: Asynchronous Shaders Evolved | Community
community.amd.com<http://community.amd.com>
One of the most exciting new developments in GPU technology
over the
past year has been the adoption of asynchronous shaders,
which can
make more efficient use of ...

Requirements:
-------------

The mechanism must expose the following functionaility:

    * Job round trip time must be predictable, from
submission to
fence signal

    * The mechanism must support compute workloads.

Goals:
------

    * The mechanism should provide low submission latencies

Test: submitting a NOP packet through the mechanism on busy
hardware
should
be equivalent to submitting a NOP on idle hardware.

Nice to have:
-------------

    * The mechanism should also support GFX workloads.

My understanding is that with the current hardware
capabilities in
Polaris10 we
will not be able to provide a solution compatible with GFX
worloads.

But I would love to hear otherwise. So if anyone has an idea,
approach or
suggestion that will also be compatible with the GFX ring,
please let
us know
about it.

    * The above guarantees should also be respected by
amdkfd workloads

Would be good to have for consistency, but not strictly
necessary as
users running
games are not traditionally running HPC workloads in the
background.

Proposed approach:
------------------

Similar to the windows driver, we could expose a high priority
compute queue to
userspace.

Submissions to this compute queue will be scheduled with high
priority, and may
acquire hardware resources previously in use by other queues.

This can be achieved by taking advantage of the 'priority'
field in
the HQDs
and could be programmed by amdgpu or the amdgpu scheduler.
The relevant
register fields are:
        * mmCP_HQD_PIPE_PRIORITY
        * mmCP_HQD_QUEUE_PRIORITY

Implementation approach 1 - static partitioning:
------------------------------------------------

The amdgpu driver currently controls 8 compute queues from
pipe0. We can
statically partition these as follows:
        * 7x regular
        * 1x high priority

The relevant priorities can be set so that submissions to
the high
priority
ring will starve the other compute rings and the GFX ring.

The amdgpu scheduler will only place jobs into the high
priority
rings if the
context is marked as high priority. And a corresponding
priority
should be
added to keep track of this information:
     * AMD_SCHED_PRIORITY_KERNEL
     * -> AMD_SCHED_PRIORITY_HIGH
     * AMD_SCHED_PRIORITY_NORMAL

The user will request a high priority context by setting an
appropriate flag
in drm_amdgpu_ctx_in (AMDGPU_CTX_HIGH_PRIORITY or similar):
https://github.com/torvalds/linux/blob/master/include/uapi/drm/amdgpu_drm.h#L163

The setting is in a per context level so that we can:
    * Maintain a consistent FIFO ordering of all
submissions to a
context
    * Create high priority and non-high priority contexts
in the same
process

Implementation approach 2 - dynamic priority programming:
---------------------------------------------------------

Similar to the above, but instead of programming the
priorities and
amdgpu_init() time, the SW scheduler will reprogram the
queue priorities
dynamically when scheduling a task.

This would involve having a hardware specific callback from
the
scheduler to
set the appropriate queue priority: set_priority(int ring,
int index,
int priority)

During this callback we would have to grab the SRBM mutex
to perform
the appropriate
HW programming, and I'm not really sure if that is
something we
should be doing from
the scheduler.

On the positive side, this approach would allow us to
program a range of
priorities for jobs instead of a single "high priority"
value",
achieving
something similar to the niceness API available for CPU
scheduling.

I'm not sure if this flexibility is something that we would
need for
our use
case, but it might be useful in other scenarios (multiple
users
sharing compute
time on a server).

This approach would require a new int field in
drm_amdgpu_ctx_in, or
repurposing
of the flags field.

Known current obstacles:
------------------------

The SQ is currently programmed to disregard the HQD
priorities, and
instead it picks
jobs at random. Settings from the shader itself are also
disregarded
as this is
considered a privileged field.

Effectively we can get our compute wavefront launched ASAP,
but we
might not get the
time we need on the SQ.

The current programming would have to be changed to allow
priority
propagation
from the HQD into the SQ.

Generic approach for all HW IPs:
--------------------------------

For consistency purposes, the high priority context can be
enabled
for all HW IPs
with support of the SW scheduler. This will function
similarly to the
current
AMD_SCHED_PRIORITY_KERNEL priority, where the job can jump
ahead of
anything not
commited to the HW queue.

The benefits of requesting a high priority context for a
non-compute
queue will
be lesser (e.g. up to 10s of wait time if a GFX command is
stuck in
front of
you), but having the API in place will allow us to easily
improve the
implementation
in the future as new features become available in new
hardware.

Future steps:
-------------

Once we have an approach settled, I can take care of the
implementation.

Also, once the interface is mostly decided, we can start
thinking about
exposing the high priority queue through radv.

Request for feedback:
---------------------

We aren't married to any of the approaches outlined above.
Our goal
is to
obtain a mechanism that will allow us to complete the
reprojection
job within a
predictable amount of time. So if anyone anyone has any
suggestions for
improvements or alternative strategies we are more than
happy to hear
them.

If any of the technical information above is also
incorrect, feel
free to point
out my misunderstandings.

Looking forward to hearing from you.

Regards,
Andres

_______________________________________________
amd-gfx mailing list
amd-gfx at lists.freedesktop.org<mailto:amd-gfx at lists.freedesktop.org>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

amd-gfx Info Page - lists.freedesktop.org<http://lists.freedesktop.org>
lists.freedesktop.org<http://lists.freedesktop.org>
To see the collection of prior postings to the list, visit the
amd-gfx Archives. Using amd-gfx: To post a message to all
the list
members, send email ...

_______________________________________________
amd-gfx mailing list
amd-gfx at lists.freedesktop.org<mailto:amd-gfx at lists.freedesktop.org>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
amd-gfx at lists.freedesktop.org<mailto:amd-gfx at lists.freedesktop.org>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
amd-gfx at lists.freedesktop.org<mailto:amd-gfx at lists.freedesktop.org>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Sincerely yours,
Serguei Sagalovitch

_______________________________________________
amd-gfx mailing list
amd-gfx at lists.freedesktop.org<mailto:amd-gfx at lists.freedesktop.org>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Sincerely yours,
Serguei Sagalovitch

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/amd-gfx/attachments/20161223/13744fe7/attachment-0001.html>