[Xcb] Re: [E-devel] XCB/XCL

[Bart Massey]

Greetings!  Sorry it's taken me so long to get back to you; 
I've got a job-critical paper due tomorrow, and it's been
sucking all of the time (pun intended).

In message <20030424143221.073153d4.raster@rasterman.com> you wrote:
> I wrote:
> > The other stuff you get is mostly bonus here.  XCB is tiny
> > (about 35KB built from about 5K lines of C and M4), but
> > that's probably not so interesting to you.  If you switch to
> > a threaded model, you should need absolutely no changes to
> > your app for XCB---no locking, no magic variable decls, etc.
> 
> We are really interested in XCB, not XCL though :) but hey... :)

My bad: I was unclear here.  What I tried and failed to say
is that you interact identically with XCB/single-threaded or
XCB/multithreaded: it's all the same API.  This gives
opportunities to use pthreads to make your life easier
without a bunch of magic machinery.

> Now to the meat. XCB has pretty much what we
> need... EXCEPT Xshm. I did a quick google for xshm
> protocol specs.. got nothing within the first 50 or so
> entries google gave me. for now i really don't want to dig
> thru x source to get the docs.

Jamey just checked in Xshm bits today.  Took him and Keith
about 1/2 hour to write and compile.  It's completely
untested, but simple enough that we're naive and think it
will work anyway :-).  Let us know how we're wrong.

> last week or so i removed all Xlib data types from ecore_x
> and now its opaque as to what sits under it. this lets us
> 1. continue development with xlib, planning a bottom-up
> move to xcb under the covers.

Very cool.

> basically here was an example of one of the bottlenecks
> i was hoping to fix:
> 
> + we get propertynotify event.
> + ecore_x now gets event and posts it.
> + event callbacks ends up being called.
> + if callback is interested in the actual property data
> the app calls a property_fetch (this will be the async
> version).
> + some time later the reply comes back. this gets
> converted into a synthetic event---property_notify_data---
> and gets posted to all the callbacks for it.
>
> this way when wm clients change title or properties -
> especially a lot of them at once, the wm does not have to
> block and wait for each fetch. they get pipelined as
> needed.
> 
> xcl ALMOST lets us do this...
(meant XCB here and elsewhere, no?)
> EXCEPT that when we want to get the property get reply -
> it will BLOCK. we basically have to add a call to poll if
> a certain cookie has been fetched already (we can run this
> after the fd has become active and we've fetched all the
> protocol).
>
> ok - ecore's design works like this:
> 
> big select loop waiting on fd's and select exiting due to
> signal, fd active or a timeout (for a timer). when it
> exist it examines what caused it to exit and reads an fd
> if its active, etc. (well calls he callback to read). this
> is where currently xlib hooks in. when the x fd goes
> active it processes events that are in the queue.
> 
> xcb needs to be able to fit in here - ie xcb fd went
> active.  ecore_x callback is called and that calls xcb's
> fd data processor. ecore_x maintains a list of pending
> replies we're interested in and if one is in the incoming
> data buffer we post a synthetic reply event to the event
> queue (locally). if tis an event we post the x event to
> the queue. now when everything has been posted the event
> queue is processed and callbacks set for the listed events
> are called per event in the queue. it's pretty efficient
> at what it does - it's the blocking xlib calls such as
> XGetWindowProperty etc. that are bad - and we need to be
> able to wake up on an fd. xcl is almost there, but not
> quite

You should be able to select on the XCB fd and have
something sensible happen.  See below.

As you say elsewhere, the question is what to do about
marshalling X replies---XCB has a queue for matching replies
with cookies: it seems like this should be useful somehow
here :-).  Given that I understand your architecture (and
I've been confused before) here's three suggestions in
decreasing order of radicalness/work on your part.

(1) Use threads.  The whole reason that XCB has transparent
    thread support is to avoid callback hell.  Basically,
    allocate one or more threads, and let them block forcing
    their cookie.  This probably requires some major
    rearchitecting of your stuff, though.

(2) (My favorite.)  Maybe I'm misunderstanding, but the
    single-threaded XCB model is designed to encourage the
    following behavior: don't ever force a cookie until you
    need the data!  IOW, instead of generating a synthetic
    event when a reply comes in and then digging around to
    process it, change things a bit to work as follows:

    When a PropertyNotify event appears, a property fetch
    request is sent to the server for any property that
    might matter, and the cookie is stashed somewhere.  (At
    this point, if there's an existing cookie outstanding
    for fetching this property, ask XCB to throw it away:
    you want the new value, not some stale one.)  Now (and
    here's the tricky part) never force the cookie until
    you're sure you actually need the reply data for
    something.  At that point, you know you might block, but
    by gosh you need to block, since the app can't go on
    without the data.

    Of course, you'll still want to select on the X fd and
    process events: XCB will let you do this fine by calling
    XCBPollEvent when select returns true.

    Can this model work with your app?  It sounded like it
    from what you said above, but it's quite possible I'm
    missing something.

(3) Do the simple thing you were proposing.  As noted above,
    you should be able to select on the X server fd using
    XCB.  When you've done that, you somehow ask XCB to
    process the connection and let you know what replies it
    found.  IIRC there's no mechanism to do this currently,
    but it should be as simple as digging the "seqnum" field
    out of the XCBConnection and checking to see if it has
    become >= the sequence number of some cookie you're
    holding.  Some API documentation on XCB would help
    :-)---apparently we never documented it much, as far as
    I can tell now :-(.

> :) for now we're still on xlib - but i've made things
> ready to use xcb and hope people will help move to it as
> it can give considerable speedups as very few people
> realise one of the worst performance bottlenecks in x is
> actually the round-trips you have to do.

You bet.

> > If there's stuff we can do to help, we will.  As someone
> > noted on the list, we think that most toolkits and apps
> > using Xlib can move to XCB pretty easily and get a lot out
> > of it.  We're excited to help it move forward.
> 
> but how can they get something out of it if they still
> just use xlib calls? you dont save on the round
> trips... :) unless u start using xcb calls :)

Yeah, XCL is kind of a side gig.  It's not really targeted
at currently-supported X apps: they should be able to just
switch to XCB pretty easily.  The biggest uses of XCL are
(1) since you can mix XCB and XCL calls, you can migrate an
app incrementally, which can be a bit easier, and (2)
XCL+XCB is tiny: handhelds and the like can avoid Xlib
altogether even for legacy Xlib apps (and without even
recompiling, since XCL is binary-compatible).


Hope this helps!  Let us know what option of (1)..(3) above
appeals to you: if it's (3) we'll look at getting the
necessary evil hacked into XCB.

	Bart Massey
	bart@cs.pdx.edu