[PATCH v2 2/2] drm/i915/gvt: Refactor GVT vblank emulator for vGPU virtual display
Zhenyu Wang
zhenyuw at linux.intel.com
Mon Mar 1 03:28:27 UTC 2021
On 2021.02.26 12:46:30 +0800, Colin Xu wrote:
> Current vblank emulator uses single hrtimer at 16ms period for all vGPUs,
> which introduces three major issues:
>
> - 16ms matches the refresh rate at 62.5Hz (instead of 60Hz) which
> doesn't follow standard timing. This leads to some frame drop or glitch
> issue during video playback. SW expects a vsync interval of 16.667ms or
> higher precision for an accurate 60Hz refresh rate. However current
> vblank emulator only works at 16ms.
>
> - Doesn't respect the fact that with current virtual EDID timing set,
> not all resolutions are running at 60Hz. For example, current virtual
> EDID also supports refresh rate at 56Hz, 59.97Hz, 60Hz, 75Hz, etc.
>
> - Current vblank emulator use single hrtimer for all vGPUs. Regardsless
> the possibility that different guests could run in different
> resolutions, all vsync interrupts are injected at 16ms interval with
> same hrtimer.
>
> Based on previous patch which decode guest expected refresh rate from
> vreg, the vblank emulator refactor patch makes following changes:
> - Change the vblank emulator hrtimer from gvt global to per-vGPU.
> By doing this, each vGPU display can operates at different refresh
> rates. Currently only one dislay is supported for each vGPU so per-vGPU
> hrtimer is enough. If multiple displays are supported per-vGPU in
> future, we can expand to per-PIPE further.
> - Change the fixed hrtimer period from 16ms to dynamic based on vreg.
> GVT is expected to emulate the HW as close as possible. So reflacting
> the accurate vsync interrupt interval is more correct than fixed 16ms.
> - Change the vblank timer period and start the timer on PIPECONF change.
> The initial period is updated to 16666667 based on 60Hz refresh rate.
> According to PRM, PIPECONF controls the timing generator of the
> connected display on this pipe, so it's safe to stop hrtimer on
> PIPECONF disabling, and re-start hrtimer at new period on enabling.
>
> Other changes including:
> - Move vblank_timer_fn from irq.c into display.c.
> - Clean per-vGPU vblank timer at clean_display instead of clean_irq.
>
> To run quick test, launch a web browser and goto URL: www.displayhz.com
>
> The actual refresh rate from guest can now always match guest settings.
>
> V2:
> Rebase to 5.11.
> Remove unused intel_gvt_clean_irq().
> Simplify enable logic in update_vblank_emulation(). (zhenyu)
> Loop all vGPU by idr when check all vblank timer. (zhenyu)
>
> Signed-off-by: Colin Xu <colin.xu at intel.com>
> ---
Thanks a lot to refresh this!
After quick review just one concern is that looks you don't handle
possible active vblank timer cleanup in vGPU destroy?
> drivers/gpu/drm/i915/gvt/display.c | 105 +++++++++++++++------------
> drivers/gpu/drm/i915/gvt/display.h | 11 ++-
> drivers/gpu/drm/i915/gvt/gvt.c | 25 +++++--
> drivers/gpu/drm/i915/gvt/gvt.h | 12 ++-
> drivers/gpu/drm/i915/gvt/handlers.c | 7 +-
> drivers/gpu/drm/i915/gvt/interrupt.c | 37 ----------
> drivers/gpu/drm/i915/gvt/interrupt.h | 7 --
> drivers/gpu/drm/i915/gvt/vgpu.c | 2 -
> 8 files changed, 94 insertions(+), 112 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gvt/display.c b/drivers/gpu/drm/i915/gvt/display.c
> index 1cae92e3752c..034c060f89d4 100644
> --- a/drivers/gpu/drm/i915/gvt/display.c
> +++ b/drivers/gpu/drm/i915/gvt/display.c
> @@ -516,11 +516,27 @@ static void clean_virtual_dp_monitor(struct intel_vgpu *vgpu, int port_num)
> port->dpcd = NULL;
> }
>
> +static enum hrtimer_restart vblank_timer_fn(struct hrtimer *data)
> +{
> + struct intel_vgpu_vblank_timer *vblank_timer;
> + struct intel_vgpu *vgpu;
> +
> + vblank_timer = container_of(data, struct intel_vgpu_vblank_timer, timer);
> + vgpu = container_of(vblank_timer, struct intel_vgpu, vblank_timer);
> +
> + /* Set vblank emulation request per-vGPU bit */
> + intel_gvt_request_service(vgpu->gvt,
> + INTEL_GVT_REQUEST_EMULATE_VBLANK + vgpu->id);
> + hrtimer_add_expires_ns(&vblank_timer->timer, vblank_timer->period);
> + return HRTIMER_RESTART;
> +}
> +
> static int setup_virtual_dp_monitor(struct intel_vgpu *vgpu, int port_num,
> int type, unsigned int resolution)
> {
> struct drm_i915_private *i915 = vgpu->gvt->gt->i915;
> struct intel_vgpu_port *port = intel_vgpu_port(vgpu, port_num);
> + struct intel_vgpu_vblank_timer *vblank_timer = &vgpu->vblank_timer;
>
> if (drm_WARN_ON(&i915->drm, resolution >= GVT_EDID_NUM))
> return -EINVAL;
> @@ -547,47 +563,56 @@ static int setup_virtual_dp_monitor(struct intel_vgpu *vgpu, int port_num,
> port->vrefresh_k = GVT_DEFAULT_REFRESH_RATE * MSEC_PER_SEC;
> vgpu->display.port_num = port_num;
>
> + /* Init hrtimer based on default refresh rate */
> + hrtimer_init(&vblank_timer->timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
> + vblank_timer->timer.function = vblank_timer_fn;
> + vblank_timer->vrefresh_k = port->vrefresh_k;
> + vblank_timer->period = DIV64_U64_ROUND_CLOSEST(NSEC_PER_SEC * MSEC_PER_SEC, vblank_timer->vrefresh_k);
> +
> emulate_monitor_status_change(vgpu);
>
> return 0;
> }
>
> /**
> - * intel_gvt_check_vblank_emulation - check if vblank emulation timer should
> - * be turned on/off when a virtual pipe is enabled/disabled.
> - * @gvt: a GVT device
> + * vgpu_update_vblank_emulation - Update per-vGPU vblank_timer
> + * @vgpu: vGPU operated
> + * @turnon: Turn ON/OFF vblank_timer
> *
> - * This function is used to turn on/off vblank timer according to currently
> - * enabled/disabled virtual pipes.
> + * This function is used to turn on/off or update the per-vGPU vblank_timer
> + * when PIPECONF is enabled or disabled. vblank_timer period is also updated
> + * if guest changed the refresh rate.
> *
> */
> -void intel_gvt_check_vblank_emulation(struct intel_gvt *gvt)
> +void vgpu_update_vblank_emulation(struct intel_vgpu *vgpu, bool turnon)
> {
> - struct intel_gvt_irq *irq = &gvt->irq;
> - struct intel_vgpu *vgpu;
> - int pipe, id;
> - int found = false;
> -
> - mutex_lock(&gvt->lock);
> - for_each_active_vgpu(gvt, vgpu, id) {
> - for (pipe = 0; pipe < I915_MAX_PIPES; pipe++) {
> - if (pipe_is_enabled(vgpu, pipe)) {
> - found = true;
> - break;
> - }
> + struct intel_vgpu_vblank_timer *vblank_timer = &vgpu->vblank_timer;
> + struct intel_vgpu_port *port =
> + intel_vgpu_port(vgpu, vgpu->display.port_num);
> +
> + if (turnon) {
> + /*
> + * Skip the re-enable if already active and vrefresh unchanged.
> + * Otherwise, stop timer if already active and restart with new
> + * period.
> + */
> + if (vblank_timer->vrefresh_k != port->vrefresh_k ||
> + !hrtimer_active(&vblank_timer->timer)) {
> + /* Stop timer before start with new period if active */
> + if (hrtimer_active(&vblank_timer->timer))
> + hrtimer_cancel(&vblank_timer->timer);
> +
> + /* Make sure new refresh rate updated to timer period */
> + vblank_timer->vrefresh_k = port->vrefresh_k;
> + vblank_timer->period = DIV64_U64_ROUND_CLOSEST(NSEC_PER_SEC * MSEC_PER_SEC, vblank_timer->vrefresh_k);
> + hrtimer_start(&vblank_timer->timer,
> + ktime_add_ns(ktime_get(), vblank_timer->period),
> + HRTIMER_MODE_ABS);
> }
> - if (found)
> - break;
> + } else {
> + /* Caller request to stop vblank */
> + hrtimer_cancel(&vblank_timer->timer);
> }
> -
> - /* all the pipes are disabled */
> - if (!found)
> - hrtimer_cancel(&irq->vblank_timer.timer);
> - else
> - hrtimer_start(&irq->vblank_timer.timer,
> - ktime_add_ns(ktime_get(), irq->vblank_timer.period),
> - HRTIMER_MODE_ABS);
> - mutex_unlock(&gvt->lock);
> }
>
> static void emulate_vblank_on_pipe(struct intel_vgpu *vgpu, int pipe)
> @@ -619,7 +644,7 @@ static void emulate_vblank_on_pipe(struct intel_vgpu *vgpu, int pipe)
> }
> }
>
> -static void emulate_vblank(struct intel_vgpu *vgpu)
> +void intel_vgpu_emulate_vblank(struct intel_vgpu *vgpu)
> {
> int pipe;
>
> @@ -629,24 +654,6 @@ static void emulate_vblank(struct intel_vgpu *vgpu)
> mutex_unlock(&vgpu->vgpu_lock);
> }
>
> -/**
> - * intel_gvt_emulate_vblank - trigger vblank events for vGPUs on GVT device
> - * @gvt: a GVT device
> - *
> - * This function is used to trigger vblank interrupts for vGPUs on GVT device
> - *
> - */
> -void intel_gvt_emulate_vblank(struct intel_gvt *gvt)
> -{
> - struct intel_vgpu *vgpu;
> - int id;
> -
> - mutex_lock(&gvt->lock);
> - for_each_active_vgpu(gvt, vgpu, id)
> - emulate_vblank(vgpu);
> - mutex_unlock(&gvt->lock);
> -}
> -
> /**
> * intel_vgpu_emulate_hotplug - trigger hotplug event for vGPU
> * @vgpu: a vGPU
> @@ -755,6 +762,8 @@ void intel_vgpu_clean_display(struct intel_vgpu *vgpu)
> clean_virtual_dp_monitor(vgpu, PORT_D);
> else
> clean_virtual_dp_monitor(vgpu, PORT_B);
> +
> + vgpu_update_vblank_emulation(vgpu, false);
> }
>
> /**
> diff --git a/drivers/gpu/drm/i915/gvt/display.h b/drivers/gpu/drm/i915/gvt/display.h
> index 2481a2ae1f68..f5616f99ef2f 100644
> --- a/drivers/gpu/drm/i915/gvt/display.h
> +++ b/drivers/gpu/drm/i915/gvt/display.h
> @@ -36,6 +36,7 @@
> #define _GVT_DISPLAY_H_
>
> #include <linux/types.h>
> +#include <linux/hrtimer.h>
>
> struct intel_gvt;
> struct intel_vgpu;
> @@ -169,6 +170,12 @@ struct intel_vgpu_port {
> u32 vrefresh_k;
> };
>
> +struct intel_vgpu_vblank_timer {
> + struct hrtimer timer;
> + u32 vrefresh_k;
> + u64 period;
> +};
> +
> static inline char *vgpu_edid_str(enum intel_vgpu_edid id)
> {
> switch (id) {
> @@ -205,8 +212,8 @@ static inline unsigned int vgpu_edid_yres(enum intel_vgpu_edid id)
> }
> }
>
> -void intel_gvt_emulate_vblank(struct intel_gvt *gvt);
> -void intel_gvt_check_vblank_emulation(struct intel_gvt *gvt);
> +void intel_vgpu_emulate_vblank(struct intel_vgpu *vgpu);
> +void vgpu_update_vblank_emulation(struct intel_vgpu *vgpu, bool turnon);
>
> int intel_vgpu_init_display(struct intel_vgpu *vgpu, u64 resolution);
> void intel_vgpu_reset_display(struct intel_vgpu *vgpu);
> diff --git a/drivers/gpu/drm/i915/gvt/gvt.c b/drivers/gpu/drm/i915/gvt/gvt.c
> index aa7fc0dd1db5..2ecb8534930b 100644
> --- a/drivers/gpu/drm/i915/gvt/gvt.c
> +++ b/drivers/gpu/drm/i915/gvt/gvt.c
> @@ -203,6 +203,22 @@ static void init_device_info(struct intel_gvt *gvt)
> info->msi_cap_offset = pdev->msi_cap;
> }
>
> +static void intel_gvt_test_and_emulate_vblank(struct intel_gvt *gvt)
> +{
> + struct intel_vgpu *vgpu;
> + int id;
> +
> + mutex_lock(&gvt->lock);
> + idr_for_each_entry((&(gvt)->vgpu_idr), (vgpu), (id)) {
> + if (test_and_clear_bit(INTEL_GVT_REQUEST_EMULATE_VBLANK + id,
> + (void *)&gvt->service_request)) {
> + if (vgpu->active)
> + intel_vgpu_emulate_vblank(vgpu);
> + }
> + }
> + mutex_unlock(&gvt->lock);
> +}
> +
> static int gvt_service_thread(void *data)
> {
> struct intel_gvt *gvt = (struct intel_gvt *)data;
> @@ -220,9 +236,7 @@ static int gvt_service_thread(void *data)
> if (WARN_ONCE(ret, "service thread is waken up by signal.\n"))
> continue;
>
> - if (test_and_clear_bit(INTEL_GVT_REQUEST_EMULATE_VBLANK,
> - (void *)&gvt->service_request))
> - intel_gvt_emulate_vblank(gvt);
> + intel_gvt_test_and_emulate_vblank(gvt);
>
> if (test_bit(INTEL_GVT_REQUEST_SCHED,
> (void *)&gvt->service_request) ||
> @@ -278,7 +292,6 @@ void intel_gvt_clean_device(struct drm_i915_private *i915)
> intel_gvt_clean_sched_policy(gvt);
> intel_gvt_clean_workload_scheduler(gvt);
> intel_gvt_clean_gtt(gvt);
> - intel_gvt_clean_irq(gvt);
> intel_gvt_free_firmware(gvt);
> intel_gvt_clean_mmio_info(gvt);
> idr_destroy(&gvt->vgpu_idr);
> @@ -337,7 +350,7 @@ int intel_gvt_init_device(struct drm_i915_private *i915)
>
> ret = intel_gvt_init_gtt(gvt);
> if (ret)
> - goto out_clean_irq;
> + goto out_free_firmware;
>
> ret = intel_gvt_init_workload_scheduler(gvt);
> if (ret)
> @@ -392,8 +405,6 @@ int intel_gvt_init_device(struct drm_i915_private *i915)
> intel_gvt_clean_workload_scheduler(gvt);
> out_clean_gtt:
> intel_gvt_clean_gtt(gvt);
> -out_clean_irq:
> - intel_gvt_clean_irq(gvt);
> out_free_firmware:
> intel_gvt_free_firmware(gvt);
> out_clean_mmio_info:
> diff --git a/drivers/gpu/drm/i915/gvt/gvt.h b/drivers/gpu/drm/i915/gvt/gvt.h
> index da8bda7f7bd2..8dc8170ba00f 100644
> --- a/drivers/gpu/drm/i915/gvt/gvt.h
> +++ b/drivers/gpu/drm/i915/gvt/gvt.h
> @@ -215,6 +215,7 @@ struct intel_vgpu {
> struct list_head dmabuf_obj_list_head;
> struct mutex dmabuf_lock;
> struct idr object_idr;
> + struct intel_vgpu_vblank_timer vblank_timer;
>
> u32 scan_nonprivbb;
> };
> @@ -347,13 +348,16 @@ static inline struct intel_gvt *to_gvt(struct drm_i915_private *i915)
> }
>
> enum {
> - INTEL_GVT_REQUEST_EMULATE_VBLANK = 0,
> -
> /* Scheduling trigger by timer */
> - INTEL_GVT_REQUEST_SCHED = 1,
> + INTEL_GVT_REQUEST_SCHED = 0,
>
> /* Scheduling trigger by event */
> - INTEL_GVT_REQUEST_EVENT_SCHED = 2,
> + INTEL_GVT_REQUEST_EVENT_SCHED = 1,
> +
> + /* per-vGPU vblank emulation request */
> + INTEL_GVT_REQUEST_EMULATE_VBLANK = 2,
> + INTEL_GVT_REQUEST_EMULATE_VBLANK_MAX = INTEL_GVT_REQUEST_EMULATE_VBLANK
> + + GVT_MAX_VGPU,
> };
>
> static inline void intel_gvt_request_service(struct intel_gvt *gvt,
> diff --git a/drivers/gpu/drm/i915/gvt/handlers.c b/drivers/gpu/drm/i915/gvt/handlers.c
> index 321480209b89..477badfcb258 100644
> --- a/drivers/gpu/drm/i915/gvt/handlers.c
> +++ b/drivers/gpu/drm/i915/gvt/handlers.c
> @@ -703,14 +703,11 @@ static int pipeconf_mmio_write(struct intel_vgpu *vgpu, unsigned int offset,
> if (data & PIPECONF_ENABLE) {
> vgpu_vreg(vgpu, offset) |= I965_PIPECONF_ACTIVE;
> vgpu_update_refresh_rate(vgpu);
> -
> + vgpu_update_vblank_emulation(vgpu, true);
> } else {
> vgpu_vreg(vgpu, offset) &= ~I965_PIPECONF_ACTIVE;
> + vgpu_update_vblank_emulation(vgpu, false);
> }
> - /* vgpu_lock already hold by emulate mmio r/w */
> - mutex_unlock(&vgpu->vgpu_lock);
> - intel_gvt_check_vblank_emulation(vgpu->gvt);
> - mutex_lock(&vgpu->vgpu_lock);
> return 0;
> }
>
> diff --git a/drivers/gpu/drm/i915/gvt/interrupt.c b/drivers/gpu/drm/i915/gvt/interrupt.c
> index 7498878e6289..497d28ce47df 100644
> --- a/drivers/gpu/drm/i915/gvt/interrupt.c
> +++ b/drivers/gpu/drm/i915/gvt/interrupt.c
> @@ -647,38 +647,6 @@ static void init_events(
> }
> }
>
> -static enum hrtimer_restart vblank_timer_fn(struct hrtimer *data)
> -{
> - struct intel_gvt_vblank_timer *vblank_timer;
> - struct intel_gvt_irq *irq;
> - struct intel_gvt *gvt;
> -
> - vblank_timer = container_of(data, struct intel_gvt_vblank_timer, timer);
> - irq = container_of(vblank_timer, struct intel_gvt_irq, vblank_timer);
> - gvt = container_of(irq, struct intel_gvt, irq);
> -
> - intel_gvt_request_service(gvt, INTEL_GVT_REQUEST_EMULATE_VBLANK);
> - hrtimer_add_expires_ns(&vblank_timer->timer, vblank_timer->period);
> - return HRTIMER_RESTART;
> -}
> -
> -/**
> - * intel_gvt_clean_irq - clean up GVT-g IRQ emulation subsystem
> - * @gvt: a GVT device
> - *
> - * This function is called at driver unloading stage, to clean up GVT-g IRQ
> - * emulation subsystem.
> - *
> - */
> -void intel_gvt_clean_irq(struct intel_gvt *gvt)
> -{
> - struct intel_gvt_irq *irq = &gvt->irq;
> -
> - hrtimer_cancel(&irq->vblank_timer.timer);
> -}
> -
> -#define VBLANK_TIMER_PERIOD 16000000
> -
> /**
> * intel_gvt_init_irq - initialize GVT-g IRQ emulation subsystem
> * @gvt: a GVT device
> @@ -692,7 +660,6 @@ void intel_gvt_clean_irq(struct intel_gvt *gvt)
> int intel_gvt_init_irq(struct intel_gvt *gvt)
> {
> struct intel_gvt_irq *irq = &gvt->irq;
> - struct intel_gvt_vblank_timer *vblank_timer = &irq->vblank_timer;
>
> gvt_dbg_core("init irq framework\n");
>
> @@ -707,9 +674,5 @@ int intel_gvt_init_irq(struct intel_gvt *gvt)
>
> init_irq_map(irq);
>
> - hrtimer_init(&vblank_timer->timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
> - vblank_timer->timer.function = vblank_timer_fn;
> - vblank_timer->period = VBLANK_TIMER_PERIOD;
> -
> return 0;
> }
> diff --git a/drivers/gpu/drm/i915/gvt/interrupt.h b/drivers/gpu/drm/i915/gvt/interrupt.h
> index 287cd142629e..6c47d3e33161 100644
> --- a/drivers/gpu/drm/i915/gvt/interrupt.h
> +++ b/drivers/gpu/drm/i915/gvt/interrupt.h
> @@ -201,11 +201,6 @@ struct intel_gvt_irq_map {
> u32 down_irq_bitmask;
> };
>
> -struct intel_gvt_vblank_timer {
> - struct hrtimer timer;
> - u64 period;
> -};
> -
> /* structure containing device specific IRQ state */
> struct intel_gvt_irq {
> struct intel_gvt_irq_ops *ops;
> @@ -214,11 +209,9 @@ struct intel_gvt_irq {
> struct intel_gvt_event_info events[INTEL_GVT_EVENT_MAX];
> DECLARE_BITMAP(pending_events, INTEL_GVT_EVENT_MAX);
> struct intel_gvt_irq_map *irq_map;
> - struct intel_gvt_vblank_timer vblank_timer;
> };
>
> int intel_gvt_init_irq(struct intel_gvt *gvt);
> -void intel_gvt_clean_irq(struct intel_gvt *gvt);
>
> void intel_vgpu_trigger_virtual_event(struct intel_vgpu *vgpu,
> enum intel_gvt_event_type event);
> diff --git a/drivers/gpu/drm/i915/gvt/vgpu.c b/drivers/gpu/drm/i915/gvt/vgpu.c
> index 6a16d0ca7cda..9039787f123a 100644
> --- a/drivers/gpu/drm/i915/gvt/vgpu.c
> +++ b/drivers/gpu/drm/i915/gvt/vgpu.c
> @@ -300,8 +300,6 @@ void intel_gvt_destroy_vgpu(struct intel_vgpu *vgpu)
> mutex_unlock(&vgpu->vgpu_lock);
>
> mutex_lock(&gvt->lock);
> - if (idr_is_empty(&gvt->vgpu_idr))
> - intel_gvt_clean_irq(gvt);
> intel_gvt_update_vgpu_types(gvt);
> mutex_unlock(&gvt->lock);
>
> --
> 2.30.1
>
> _______________________________________________
> intel-gvt-dev mailing list
> intel-gvt-dev at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/intel-gvt-dev/attachments/20210301/d5c3825f/attachment.sig>
More information about the intel-gvt-dev
mailing list