[Xcb] Help with external socket handling and xcb

[Carlo Wood]

> Well, there is no public API for this. Internally, _xcb_in_read()
> reads from the socket into a buffer and then reads individual packets
> from this via read_packet(). However, inside a packet, the remainder
> of a packet is again read blockingly (see call to
> _xcb_in_read_block()). The expectation here is that the X11 server
> will write the whole packet in one go.
> 
> This is necessary because X11 packets can be quite large, certainly
> larger than the 4 KiB that is the default read buffer size.
> 
> Also: Your approach would mean that the data would have to be copied
> again. In some cases (the _xcb_in_read_block() call I mention above),
> libxcb can read the data directly into its final place; avoiding an
> extra copy.
> 
> > Because I assume that xcb internally will keep reading the socket
> > until that returns that there is no data left to read. And in order
> > for that not to be blocking, the fd has to be non-blocking (which
> > is obviously what I normally want).  
> 
> xcb actually calls poll()/select() before doing anything with the
> socket to make sure that following call will not block "too badly".
> Anything else would again interfere with thread-safety. However,
> functions like xcb_wait_for_reply() still call this code in a loop
> until they are done with what they want to do.
> 
> I don't really see how this could be done with an external "loop
> driver". Every function returning some kind of EAGAIN? I wouldn't want
> to try to use such an API.

It kinda depends on the protocol (not every socket fd can be treated
equally) I guess.

In this case, I am willing to go a long way to provide the lowest
possible latency for user input; probably even as far as making
a thread block when we already know that a reply from the X server
is about to happen and that this normally doesn't happen (because
the packet fits in the buffer and the server did send everyone
at once, plus we're talking about a UNIX socket here; so the delay
should be extremely short).

So, I think I'm fine with the existing API where a function that
has to return a reply from the server will block until it receives
that reply (for now anyway).

My main problem is with xcb_poll_for_event and how to get that
to work without having to call it 100 times per second even when
there are no events. I really want to make that event driven:
only call something when there is something to read on the socket.

But yeah - I understand that if at the same time some other thread
calls a function like xcb_get_input_focus() then my fd will become
readable - and I'd still call xcb_poll_for_event for nothing (or
whatever other function I should be calling), but that is at least
better than having to call it nonstop no matter what.

I hope that if I do that before the xcb internal poll returns
that MY call will cause the fd to be read, otherwise I'd be wasting
100% CPU on the fact that the socket keeps being readable (then
I'll have to add a timer to stop that from happening). I also hope
(assume) that you have a mutex around the actual call to recv,
so no two threads will be reading the socket at the same time :P.

> > However, doesn't there need to be data written to the socket too?
> > When does that happen?  
> 
> Whenever it needs to be. There is a buffer for outgoing data. When
> that is full, the current call blocks until the buffered data was
> written. Also, xcb_flush() will, well, flush the buffer.

So much blocking :(. My library just writes outgoing buffers to
fd's whenever those fds are writable. You can however write to a
buffer without that that is seen as data being in the buffer;
That is, you need to flush the stream for the data to become visible
and writing will begin.

Aka (this is C++, so I use ostreams):

device << x << ", " << y << " = " << z << std::endl;

would first write the whole line to the buffer and only at the
flush of the endl start to monitor to outgoing fd for writability
(and then write the whole line at once). Or you can use "<< '\n'"
and buffer multiple lines before you send anything off.

There is no blocking however. Doing a "flush" makes the data
visible to the code that does the writing to the socket, but it
doesn't block.

> 
> So basically: Any function that sends a request can block waiting for
> some data to be written.
> 
> > With the current set up I can't monitor the
> > socket to see if it writable because it is ALWAYS writable. I can
> > only do this when xcb tells me (through a callback) that it NEEDS
> > me to monitor the socket because it has something to write (and
> > also tells me when I can stop monitoring the fd again).  
> 
> What would you want a function like xcb_get_input_focus() to return
> when it needs to write some data, but the buffer is currently full?

When receiving data, I just stop monitoring the socket for more input
until the buffer drops below a certain watermark:

https://github.com/CarloWood/evio/blob/master/TLSSocket.cxx#L198

When writing output, there is nothing else to do imho then to keep
increasing the size of the buffer until you run out of memory.
Obviously, my output buffers have a method that allow you to test
if it runs too full and if that is the case, you should slow down;
but at the very moment that you are actually trying to write, it is
pretty unacceptable (and hard to recover) when that fails. Ie, when
it fails to execute:

socket << my_message;

That doesn't even have a return value ;). So, it would throw an
exception. Since I use the std::streambuf interface that boils
down to returning EOF in the appropriate places, ie

https://github.com/CarloWood/evio/blob/master/StreamBuf.cxx#L162
https://github.com/CarloWood/evio/blob/master/StreamBuf.cxx#L622

> 
> > So, since it doesn't do that (as far as I know), it will have to
> > make sure that every time it has something to write, it writes
> > ALL of it before returning to my code... and that sounds very
> > blocking. How can this ever work reliable with a non-blocking
> > fd? Or is xcb just calling write in a loop until it wrote everything
> > it has to write (also when write returns EAGAIN)?  
> 
> Basically yes. _xcb_conn_wait() calls write_vec() to write some data
> whenever poll()/select() say that the FD is writeable and there is
> some pending data to send. Here is the loop that then calls
> _xcb_conn_wait() until everything was written:
> https://github.com/freedesktop/xcb-libxcb/blob/ee9dfc9a7658e7fe75d27483bb5ed1ba4d1e2c86/src/xcb_out.c#L457-L458

Just to be clear; so this means that in order for xcb_poll_for_event
to work while only calling it when I see that the fd is readable, I only
ever have to monitor that the fd is readable, and never have to worry
about monitoring if the fd is writable, because all writing is done
blocking, even though the fd itself is non-blocking?

> 
> > If anyone can give me pointers, or maybe tell me about a part of
> > the API that I missed that is suitable for this, that would be
> > great!  
> 
> For non-blockingly waiting for a reply to a request: Do you know about
> xcb_poll_for_reply()? There is an example on how to use that to get a
> callback-based "your reply arrived"-API here:
> https://gitlab.freedesktop.org/xorg/lib/libxcb/-/issues/52#note_712068
> (Hm, it seems like this code does not handle sequence number
> overflows...)

Hmm. This is still polling in nature and hard to wrap my head around
without knowing more about the protocol that is being sent to and from
the X server.

Really non-polling would be when a callback for arrived replies is done
whenever the function that reads incoming data (called by my
application when there is data to read on the fd) sees that such
replies did arrive.

I could call "poll_for_reply" whenever the socket becomes readable, but
that makes no sense because at that moment the socket hasn't been read
yet. Or I can first call xcb_poll_for_event which will read data from
the fd and also decode any replies, correct? And then call
xcb_poll_for_reply to see if any replies just have been decoded are
now queued up?

Regards,
Carlo