X compression techniques (was Re: VNC server based on kdrive using damage extension?)

[Gian Filippo Pinzari]

Jim Gettys wrote:
> I gather the NX folks observe that by zeroing out the unused
> fields in the X protocol stream one can significantly improve
> compression over what our naive ssh based tests show, and they
> may have other tricks well worth understanding.

Zeroing the padding bytes provides little gain on ZLIB based
compression (no more than 10% in our tests), anyway as I tried
to explain in the past, it makes a real difference for nxproxy.
X messages are checksummed in NX and messages that are found
equal are differentially encoded. This works only if equal
messages are bitwise equal. Not many messages need to be zeroed.
For most messages NX proxy performs the zeroing internally. In
the case of images and RENDER messages, we found convenient to
do that in the Xlib and Xrender libraries. The modifications
can be easily imported in the fd.o code base.

> You ask: how can I do remote access: the answer for the moment
> is ssh, with or without compression enabled (ssh -X -C).

NX uses SSH to provide TLS. We plan to support HTTPS in
future. For the extent of security and authentication, any
valid method should be OK. In the long run security will be
built at the IP layer and X will be able to leverage this
as well.

 > Going through external proxies is almost certainly a performance
 > problem.

NX compression provides much better performance than plain
ZLIB compression. NX X channels are compressed using specific
algorythms. NX provides better compression and lower CPU usage
on X protocol the same way as a MPEG4 compressor yields better
results on video compared to compressing the same stream
with ZLIB.

/Gian Filippo.