I just sent out patch below yesterday.  swapping unpopulated bo is useless indeed.

[RFC PATCH 2/2] drm/ttm: skip swapout when ttm has no backend page.