X compression techniques (was Re: VNC server based on kdrive using damage extension?)

[Jim Gettys]

This is a topic we need to address.

Our work for Usenix last spring showed both:
	1) the gains possible even with simple minded compression
	(gzip/deflate). It makes a huge difference on current
	applications, so if you are at all bandwidth constrained,
	is an obvious performance win.
	2) that LBX is a waste these days, and can be deprecated.
(available at http://keithp.com/~keithp/talks/usenix2003/)

I gather the NX folks observe that by zeroing out the unused
fields in the X protocol stream one can significantly improve
compression over what our naive ssh based tests show, and they
may have other tricks well worth understanding.  I've honestly
been to busy the last six months to investigate their work
to the extent it deserves, and patching Xlib to zero the unused
fields is trivial (and since the writes should be hidden by the
memory traffic, should not cost any performance on current systems
I believe).

Due to the general security situation, and the very widespread
installation of wire sniffers in the real world internet
(even 5 years ago, Jeffrey Schiller who runs MIT's network and
was area director for security in the IETF estimated 40% of the
subnets of MIT had packet sniffers installed by bad guys),
using X unencrypted and without strong authentication is very
iffy indeed.  These sniffers are the first thing installed
by black hats when breaking corporate networks as well.

Right now, I highly recommend everyone ship X with any remote
access disabled (and throw away LBX :-)).

You ask: how can I do remote access: the answer for the moment
is ssh, with or without compression enabled (ssh -X -C).

Also note that since ssh has IPV6 support, that this reduces
the urgency of native X IPV6 support.  I honestly don't
care if what is currently present in the source pool ever sees
the light of day, since I believe it should be disabled
for the time being in almost all environments and is therefore
moot. IPv6 support without also handling security and authentication
is nearly useless, due to the state of the bad guys on the
network.

In the reasonably near term (within the next year), however, I
believe we have to face up to better integrated authentication,
security and compression.  Going through external proxies
is almost certainly a performance problem.
LTSP in particular needs this (for scaling's sake at a minimum),
as does any vision of migration of application's display.
And this needs to be done definitely with IPV6 in mind.
                         - Jim

On Fri, 2004-02-20 at 05:23, Egbert Eich wrote:
> Mike A. Harris writes:
>  > 
>  > Should the software become relicensed to be MIT/X11 compatible 
>  > license, then it would be much more valuable from the perspective 
>  > of an X developer looking to improve the X server, etc.  If 
>  > they're willing to relicense the code, then that is great news.
>  > 
> I'd think it should work the other way around.
> If we find parts of the code interesting to be integrated in
> X itself we should discuss if it can be relicensed.
> If we find that this element of the X wire communication
> best lives inside a proxy there is not need to relicense
> it as in this case the license is irrelevant.
> 
>  > 
>  > I'm not hostile towards NX itself, nor it's developers, and if it 
>  > has appeared that way to anyone, then I hope my intentions are 
>  > clarified by this message.
> 
> OK.
> 
>  > 
>  > However, on several occasions both in email and IRC I have had 
>  > non-developer end users come into develmental discussions asking 
>  > why we don't just use NX, and that irritates me, because we are 
>  > developers trying to improve X.  We don't improve X by just using 
>  > some other pre-existing software, however non-developer types 
>  > just don't seem to "get" what our goals are.  Afterall, we could 
>  > just disable the TCP transport entirely in X and just use VNC, or 
>  > some other solution.  The fact we're not doing so indicates that 
>  > we all believe that there are other options available, and that 
>  > there are other paths we can take to work on a better solution in 
>  > X in the future.
> 
> We often disable the TCP transport in X as for most situations we
> tunnel it thru ssh. X's network capability doesn't exist because
> it has a TCP layer but because there is a protocol and certain
> features inside the protocol that make it (more or less) feasable
> across a wire. If there was just an ABI it wouldn't be network
> transparent. Most solutions (also NX) take advantage of this 
> 'network transparency'. It just uses a more efficient transport
> methods across some parts of the wire.
> In cases where a network transparency doesn't exist you have to
> implement this into a 'driver' on a very low level.
> 
>  > 
>  > NX seems like wonderful technology, and so if it could be
>  > relicensed to be compatible with the MIT license then it would be 
>  > something useable inside X potentially, and any of my voiced 
>  > opinions about it from an X11 developmental viewpoint more or 
>  > less become null and void.
> 
> Right.
>  > 
>  > Apologies if I have seemed overly hostile in the discussion, as I 
>  > do often get a bit short with end users who try to fit into 
>  > developmental discussions without understanding things from a 
>  > developer viewpoint.
>  > 
>  Thank you.
> 
> Egbert.
> 
> _______________________________________________
> xserver mailing list
> xserver@freedesktop.org
> http://freedesktop.org/mailman/listinfo/xserver
-- 
Jim Gettys <Jim.Gettys@hp.com>
HP Labs, Cambridge Research Laboratory