[Xcb] Help with external socket handling and xcb

[Carlo Wood]

Ping 

Was it something I said?

On Thu, 9 Sep 2021 12:48:58 +0200
Carlo Wood <carlo at alinoe.com> wrote:

> > Well, there is no public API for this. Internally, _xcb_in_read()
> > reads from the socket into a buffer and then reads individual
> > packets from this via read_packet(). However, inside a packet, the
> > remainder of a packet is again read blockingly (see call to
> > _xcb_in_read_block()). The expectation here is that the X11 server
> > will write the whole packet in one go.
> > 
> > This is necessary because X11 packets can be quite large, certainly
> > larger than the 4 KiB that is the default read buffer size.
> > 
> > Also: Your approach would mean that the data would have to be copied
> > again. In some cases (the _xcb_in_read_block() call I mention
> > above), libxcb can read the data directly into its final place;
> > avoiding an extra copy.
> >   
> > > Because I assume that xcb internally will keep reading the socket
> > > until that returns that there is no data left to read. And in
> > > order for that not to be blocking, the fd has to be non-blocking
> > > (which is obviously what I normally want).    
> > 
> > xcb actually calls poll()/select() before doing anything with the
> > socket to make sure that following call will not block "too badly".
> > Anything else would again interfere with thread-safety. However,
> > functions like xcb_wait_for_reply() still call this code in a loop
> > until they are done with what they want to do.
> > 
> > I don't really see how this could be done with an external "loop
> > driver". Every function returning some kind of EAGAIN? I wouldn't
> > want to try to use such an API.  
> 
> It kinda depends on the protocol (not every socket fd can be treated
> equally) I guess.
> 
> In this case, I am willing to go a long way to provide the lowest
> possible latency for user input; probably even as far as making
> a thread block when we already know that a reply from the X server
> is about to happen and that this normally doesn't happen (because
> the packet fits in the buffer and the server did send everyone
> at once, plus we're talking about a UNIX socket here; so the delay
> should be extremely short).
> 
> So, I think I'm fine with the existing API where a function that
> has to return a reply from the server will block until it receives
> that reply (for now anyway).
> 
> My main problem is with xcb_poll_for_event and how to get that
> to work without having to call it 100 times per second even when
> there are no events. I really want to make that event driven:
> only call something when there is something to read on the socket.
> 
> But yeah - I understand that if at the same time some other thread
> calls a function like xcb_get_input_focus() then my fd will become
> readable - and I'd still call xcb_poll_for_event for nothing (or
> whatever other function I should be calling), but that is at least
> better than having to call it nonstop no matter what.
> 
> I hope that if I do that before the xcb internal poll returns
> that MY call will cause the fd to be read, otherwise I'd be wasting
> 100% CPU on the fact that the socket keeps being readable (then
> I'll have to add a timer to stop that from happening). I also hope
> (assume) that you have a mutex around the actual call to recv,
> so no two threads will be reading the socket at the same time :P.
> 
> > > However, doesn't there need to be data written to the socket too?
> > > When does that happen?    
> > 
> > Whenever it needs to be. There is a buffer for outgoing data. When
> > that is full, the current call blocks until the buffered data was
> > written. Also, xcb_flush() will, well, flush the buffer.  
> 
> So much blocking :(. My library just writes outgoing buffers to
> fd's whenever those fds are writable. You can however write to a
> buffer without that that is seen as data being in the buffer;
> That is, you need to flush the stream for the data to become visible
> and writing will begin.
> 
> Aka (this is C++, so I use ostreams):
> 
> device << x << ", " << y << " = " << z << std::endl;
> 
> would first write the whole line to the buffer and only at the
> flush of the endl start to monitor to outgoing fd for writability
> (and then write the whole line at once). Or you can use "<< '\n'"
> and buffer multiple lines before you send anything off.
> 
> There is no blocking however. Doing a "flush" makes the data
> visible to the code that does the writing to the socket, but it
> doesn't block.
> 
> > 
> > So basically: Any function that sends a request can block waiting
> > for some data to be written.
> >   
> > > With the current set up I can't monitor the
> > > socket to see if it writable because it is ALWAYS writable. I can
> > > only do this when xcb tells me (through a callback) that it NEEDS
> > > me to monitor the socket because it has something to write (and
> > > also tells me when I can stop monitoring the fd again).    
> > 
> > What would you want a function like xcb_get_input_focus() to return
> > when it needs to write some data, but the buffer is currently full?
> >  
> 
> When receiving data, I just stop monitoring the socket for more input
> until the buffer drops below a certain watermark:
> 
> https://github.com/CarloWood/evio/blob/master/TLSSocket.cxx#L198
> 
> When writing output, there is nothing else to do imho then to keep
> increasing the size of the buffer until you run out of memory.
> Obviously, my output buffers have a method that allow you to test
> if it runs too full and if that is the case, you should slow down;
> but at the very moment that you are actually trying to write, it is
> pretty unacceptable (and hard to recover) when that fails. Ie, when
> it fails to execute:
> 
> socket << my_message;
> 
> That doesn't even have a return value ;). So, it would throw an
> exception. Since I use the std::streambuf interface that boils
> down to returning EOF in the appropriate places, ie
> 
> https://github.com/CarloWood/evio/blob/master/StreamBuf.cxx#L162
> https://github.com/CarloWood/evio/blob/master/StreamBuf.cxx#L622
> 
> >   
> > > So, since it doesn't do that (as far as I know), it will have to
> > > make sure that every time it has something to write, it writes
> > > ALL of it before returning to my code... and that sounds very
> > > blocking. How can this ever work reliable with a non-blocking
> > > fd? Or is xcb just calling write in a loop until it wrote
> > > everything it has to write (also when write returns EAGAIN)?    
> > 
> > Basically yes. _xcb_conn_wait() calls write_vec() to write some data
> > whenever poll()/select() say that the FD is writeable and there is
> > some pending data to send. Here is the loop that then calls
> > _xcb_conn_wait() until everything was written:
> > https://github.com/freedesktop/xcb-libxcb/blob/ee9dfc9a7658e7fe75d27483bb5ed1ba4d1e2c86/src/xcb_out.c#L457-L458
> >  
> 
> Just to be clear; so this means that in order for xcb_poll_for_event
> to work while only calling it when I see that the fd is readable, I
> only ever have to monitor that the fd is readable, and never have to
> worry about monitoring if the fd is writable, because all writing is
> done blocking, even though the fd itself is non-blocking?
> 
> >   
> > > If anyone can give me pointers, or maybe tell me about a part of
> > > the API that I missed that is suitable for this, that would be
> > > great!    
> > 
> > For non-blockingly waiting for a reply to a request: Do you know
> > about xcb_poll_for_reply()? There is an example on how to use that
> > to get a callback-based "your reply arrived"-API here:
> > https://gitlab.freedesktop.org/xorg/lib/libxcb/-/issues/52#note_712068
> > (Hm, it seems like this code does not handle sequence number
> > overflows...)  
> 
> Hmm. This is still polling in nature and hard to wrap my head around
> without knowing more about the protocol that is being sent to and from
> the X server.
> 
> Really non-polling would be when a callback for arrived replies is
> done whenever the function that reads incoming data (called by my
> application when there is data to read on the fd) sees that such
> replies did arrive.
> 
> I could call "poll_for_reply" whenever the socket becomes readable,
> but that makes no sense because at that moment the socket hasn't been
> read yet. Or I can first call xcb_poll_for_event which will read data
> from the fd and also decode any replies, correct? And then call
> xcb_poll_for_reply to see if any replies just have been decoded are
> now queued up?
> 
> Regards,
> Carlo
>