Re: Comm API notes

On Thu, 2008-09-04 at 23:27 -0600, Alex Rousskov wrote:
> Hello,
>
> As a part of Squid3 cleanup and planning, I promised to assemble
> Comm API-related notes from a few squid-dev threads and bring them in
> sync with the current implementation. The first draft of that
> documentation is below, FYI.
>
> This is a work-in-progress and no formal review is requested at this
> time. Moreover, please keep in mind that portions of the text refer to
> fixes being implemented in the current trunk cleanup effort. Those
> portions may not be in sync with the code in the trunk (I will begin
> posting those fixes soon).
>
> If something jumps at you as totally wrong with respect to How Things
> Should Be Done, please yell.

Here is an updated version, including I/O cancellation notes.

---
* Basic Comm API principles.
  Comm API supports four major kinds of socket-related operations:
  accept, connect, read, and write. Sockets are identified by their
  descriptors.
  A Comm user requests its interest in operations by calling an
  appropriate API function (e.g., comm_read) and optionally providing
  a notification "callback" (a.k.a., handler). For a given descriptor,
  there can be no more than one registered user per operation.
  In addition to the four operations listed above, a user can register
  to be notified when a given descriptor is closed.
  When a Comm operation completes, all users that registered the
  corresponding handler are notified. When a descriptor is closed, all
  users that registered any callback for the descriptor are notified
  (this will change though, see "Anticipated changes" below).
  All Comm notifications are asynchronous, performed using the
  AsyncCall API.
  Notifications for four operations listed above are scheduled in the
  order of the corresponding operations (i.e., in the order the I/O
  loop discovers that the operation can proceed). Notifications
  related to closing of a descriptor are documented in the next
  section.
* I/O cancellation.
  To cancel an interest in a read operation, call comm_read_cancel()
  with an AsyncCall object. This call guarantees that the passed Call
  will be canceled (see the AsyncCall API for call cancellation
  definitions and details). Naturally, the code has to store the
  original read callback Call pointer to use this interface. This call
  does not guarantee that the read operation has not already happen.
  This call guarantees that the read operation will not happen.
  You cannot reliably cancel an interest in read operation using the old
  comm_read_cancel call that uses a function pointer. The handler may
  get even called after old comm_read_cancel was called. This old API
  will be removed.
  It is OK to call comm_read_cancel (both old and new) at any time as
  long as the descriptor has not been closed and there is either no read
  interest registered or the passed parameters match the registered
  ones. If the descriptor has been closed, the behavior is undefined.
  Otherwise, if parameters do not match, you get an assertion.
  To cancel other operations, close the descriptor with comm_close.
* Descriptor closing with comm_close.
  Comm_close does not close the descriptor but initiates the following
  closing sequence:
      1) The descriptor is placed in a "closing" state.
      2) The registered read, write, and accept callbacks (if any) are
         scheduled (in an unspecified order).
      3) The close callbacks are scheduled (in an unspecified order).
      4) A call to the internal descriptor closing handler is
         scheduled.
  The "unspecified" order above means that the user should not rely on
  any specific or deterministic order because the currently hard-coded
  order may change.
  The read, write, and accept notifications (scheduled in step #2
  above) carry the COMM_ERR_CLOSING error flag. When handling
  COMM_ERR_CLOSING event, the user code should limit
  descriptor-related processing, especially Comm calls, because
  supported Comm functionality is very limited when the descriptor is
  closing. New code should use the close handlers instead (scheduled
  in step #3).
  The internal closing handler (scheduled in step #4 above) closes the
  descriptor. When the descriptor is closed, all operations on the
  descriptor are prohibited and may cause bugs and asserts. Currently,
  the same descriptor will eventually be reused for another socket and
  Comm may not notice that a buggy code is still using a stale
  descriptor, but that may change.
  Since all notifications are asynchronous, it is possible for a read
  or write notification that was scheduled before comm_close was
  called to arrive at its destination after comm_close was called.
  Such notification will arrive with COMM_ERR_CLOSING flag even though
  that flag was not set at the time of the I/O (and the I/O may have
  been successful). This behavior may change.
* Anticipated changes and preparation recommendations
  This section lists anticipated Comm API changes and provides
  recommendations for developers writing new (or rewriting old) Comm
  user code. The changes are listed in a rough order from more likely
  to less certain and from near-feature to long-term.
  The old comm_read_cancel function that uses a function pointer will be
  removed as unreliable. Use the AsyncCall-based comm_read_cancel
  instead.
  COMM_ERR_CLOSING interface will be removed. The read, write, and
  accept notifications will not be scheduled after comm_close is
  called.  New user code should register close handlers instead.
  When COMM_ERR_CLOSING interface is removed, pending notifications
  (if any) will be canceled after comm_close is called. However, the
  cancellation may be removed later if Comm is modified to provide safe
  access to closing descriptors and their fragile state. New user code
  should continue to assume that it is safe to access Comm in a read,
  write, and accept handlers.
  Comm users may be required to become children of Comm-provided
  classes, to eliminate the need for a complicated (albeit hidden)
  hierarchy of Comm callback dialers (see CommCalls.h) and to provide
  default implementation for common I/O handling cases.  New user code
  should use methods of AsyncJob-derived classes to handle Comm
  notifications instead of using stand-alone functions.
  The comm_close API will be split into "stop I/O" and "kill me" APIs.
  New user code should not use comm_close for the purpose of ending a
  job (and should not assume it does). In most cases, the job has more
  tasks to do even if the descriptor is closed. AsyncJob API provides
  job-ending tools.
  Raw socket descriptors may be replaced with unique IDs or small
  objects that help detect stale descriptor/socket usage bugs and
  encapsulate access to socket-specific information. New user code
  should treat descriptor integers as opaque objects.
---
Cheers,
Alex.