Comm API notes

Hello,

    As a part of Squid3 cleanup and planning, I promised to assemble
Comm API-related notes from a few squid-dev threads and bring them in
sync with the current implementation. The first draft of that
documentation is below, FYI.

This is a work-in-progress and no formal review is requested at this
time. Moreover, please keep in mind that portions of the text refer to
fixes being implemented in the current trunk cleanup effort. Those
portions may not be in sync with the code in the trunk (I will begin
posting those fixes soon).

If something jumps at you as totally wrong with respect to How Things
Should Be Done, please yell.

--------

* Basic Comm API principles.

  Comm API supports four major kinds of socket-related operations:
  accept, connect, read, and write. Sockets are identified by their
  descriptors.

  A Comm user requests its interest in operations by calling an
  appropriate API function (e.g., comm_read) and optionally providing
  a notification "callback" (a.k.a., handler). For a given descriptor,
  there can be no more than one registered user per operation.

  In addition to the four operations listed above, a user can register
  to be notified when a given descriptor is closed.

  When a Comm operation completes, all users that registered the
  corresponding handler are notified. When a descriptor is closed, all
  users that registered any callback for the descriptor are notified
  (this will change though, see "Anticipated changes" below).

  All Comm notifications are asynchronous, performed using the
  AsyncCall API.

  Notifications for four operations listed above are scheduled in the
  order of the corresponding operations (i.e., in the order the I/O
  loop discovers that the operation can proceed). Notifications
  related to closing of a descriptor are documented in the next
  section.

* Descriptor closing with comm_close.

  Comm_close does not close the descriptor but initiates the following
  closing sequence:

      1) The descriptor is placed in a "closing" state.

      2) The registered read, write, and accept callbacks (if any) are
         scheduled (in an unspecified order).

      3) The close callbacks are scheduled (in an unspecified order).

      4) A call to the internal descriptor closing handler is
         scheduled.

  The "unspecified" order above means that the user should not rely on
  any specific or deterministic order because the currently hard-coded
  order may change.

  The read, write, and accept notifications (scheduled in step #2
  above) carry the COMM_ERR_CLOSING error flag. When handling
  COMM_ERR_CLOSING event, the user code should limit
  descriptor-related processing, especially Comm calls, because
  supported Comm functionality is very limited when the descriptor is
  closing. New code should use the close handlers instead (scheduled
  in step #3).

  The internal closing handler (scheduled in step #4 above) closes the
  descriptor. When the descriptor is closed, all operations on the
  descriptor are prohibited and may cause bugs and asserts. Currently,
  the same descriptor will eventually be reused for another socket and
  Comm may not notice that a buggy code is still using a stale
  descriptor, but that may change.

  Since all notifications are asynchronous, it is possible for a read
  or write notification that was scheduled before comm_close was
  called to arrive at its destination after comm_close was called.
  Such notification will arrive with COMM_ERR_CLOSING flag even though
  that flag was not set at the time of the I/O (and the I/O may have
  been successful). This behavior may change.

* Anticipated changes and preparation recommendations

  This section lists anticipated Comm API changes and provides
  recommendations for developers writing new (or rewriting old) Comm
  user code. The changes are listed in a rough order from more likely
  to less certain and from near-feature to long-term.

  COMM_ERR_CLOSING interface will be removed. The read, write, and
  accept notifications will not be scheduled after comm_close is
  called. New user code should register close handlers instead.

  When COMM_ERR_CLOSING interface is removed, pending notifications
  (if any) will be canceled after comm_close is called. However, the
  cancellation may be removed later if Comm is modified to provide safe
  access to closing descriptors and their fragile state. New user code
  should continue to assume that it is safe to access Comm in a read,
  write, and accept handlers.

  Comm users may be required to become children of Comm-provided
  classes, to eliminate the need for a complicated (albeit hidden)
  hierarchy of Comm callback dialers (see CommCalls.h) and to provide
  default implementation for common I/O handling cases. New user code
  should use methods of AsyncJob-derived classes to handle Comm
  notifications instead of using stand-alone functions.

  Raw socket descriptors may be replaced with unique IDs or small
  objects that help detect stale descriptor/socket usage bugs and
  encapsulate access to socket-specific information. New user code
  should treat descriptor integers as opaque objects.

-------

Thank you,

Alex.
P.S. The draft focuses on topics discussed in recent Comm- and
AsyncCall- related threads on squid-dev. It is not meant to provide
comprehensive documentation of all Comm API aspects.
Received on Fri Sep 05 2008 - 05:27:42 MDT