X compression techniques (was Re: VNC server based on kdrive using damage extension?)

[Jim Gettys]

On Fri, 2004-02-20 at 12:59, Gian Filippo Pinzari wrote:
> Jim Gettys wrote:
> > I gather the NX folks observe that by zeroing out the unused
> > fields in the X protocol stream one can significantly improve
> > compression over what our naive ssh based tests show, and they
> > may have other tricks well worth understanding.
> 
> Zeroing the padding bytes provides little gain on ZLIB based
> compression (no more than 10% in our tests)

10% for little work is well worth doing.

> 
> , anyway as I tried
> to explain in the past, it makes a real difference for nxproxy.
> X messages are checksummed in NX and messages that are found
> equal are differentially encoded. This works only if equal
> messages are bitwise equal. Not many messages need to be zeroed.
> For most messages NX proxy performs the zeroing internally. In
> the case of images and RENDER messages, we found convenient to
> do that in the Xlib and Xrender libraries. The modifications
> can be easily imported in the fd.o code base.

Patches will be greatfully accepted.

Ah, so you are playing MD5 sorts of message tricks.  Goodness...


> 
> > You ask: how can I do remote access: the answer for the moment
> > is ssh, with or without compression enabled (ssh -X -C).
> 
> NX uses SSH to provide TLS. We plan to support HTTPS in
> future. For the extent of security and authentication, any
> valid method should be OK. In the long run security will be
> built at the IP layer and X will be able to leverage this
> as well.

Getting a decent authentication and authorization model is also
needed; it isn't clear that exchanging SSH keys is really viable for
a lot of uses, nor is it clear that IP level security will provide
all we need for authentication.  The other issue with IP level
security is that deployment is then limited to the deployment
rate of the kernel support, which inhibits rapid deployment
greatly.

In the IETF, the point of IP level security was to finally
get to the eventual nirvana of everything becoming secure,
even without touching 10 year old applications; this does not mean
that applications level security may not have other characteristics.

For example, given opportunistic encryption that Linux now
provides, some X traffic is already at least getting encrypted
without having lifted a finger.  But it is well understood that
applications level security may be more desirable in many
circumstances.


> 
>  > Going through external proxies is almost certainly a performance
>  > problem.
> 
> NX compression provides much better performance than plain
> ZLIB compression. NX X channels are compressed using specific
> algorythms. NX provides better compression and lower CPU usage
> on X protocol the same way as a MPEG4 compressor yields better
> results on video compared to compressing the same stream
> with ZLIB.

I not at all surprised, given that images in common
use may be amenable to compression tricks that might be missed
in a general purpose compressor. And hashing tricks are also
lots of fun.
                                    - Jim

-- 
Jim Gettys <Jim.Gettys@hp.com>
HP Labs, Cambridge Research Laboratory