CA over TCP

Jeff Hill and Ralph Lange, Codeathon 2008

(This refers to the CA over TCP topic of Andrew's project list.)

Goal

Provide a way to run CA on TCP circuits. Intended use: Run CA clients remotely that use an ssh tunnel to safely and securely connect to the control system. For security reasons these tunneled connections should connect to a CA Gateway.

Approach

TCP connections can easily be tunneled through ssh. To allow CA connections through a TCP tunnel, the use of UDP (currently for name resolution and server beacons) must be avoided.

Beacons

We are just ignoring the beacon topic for the time being. With the current EPICS release, CA clients do not raise the search interval above a certain limit. Even without beacon anomalies being detected, all unresolved channels will connect sooner or later. That should be good enough for a start.

Name Resolution

A new environment variable EPICS_CA_NAME_SERVERS will be used to configure TCP name resolution. This variable takes a list of IP addresses. The client will open regular TCP connections (default port 5064) to those servers at init time and issue name resolution requests over TCP in addition to issuing the UDP requests to addresses in EPICS_CA_ADDR_LIST.

The CA server will be changed to forward the name resolution requests coming from TCP directly to the server tool.

Servers that are not capable of handling name requests over TCP (e.g. older rsrv or CAS version) will be ignored, i.e. the client will not see the requested channels through TCP and not connect unless a different (name) server answers the request.

List of Necessary Changes

CA Client

• tcpiiu changes
  • add bool _nameService private member data flag
  • pass bool nameService arg to tcpiiu constructor (setting private bool flag)
  • if _nameService, constructor registers member SearchDest with cac
  • if _nameService, destructor unregisters member SearchDest with cac
  • if _nameService flag is set send thread sleeps and then tries to connect again if connect fails
  • if _nameService flag set the receive thread, upon receiving disconnect notification from the socket
    • command send thread to exit
    • waits for send thread exit
    • (clear all buffers)
    • sleep
    • restarts the send thread
    • exits
  • class deriving from SearchDest called SearchDestTCP
  • private member of type SearchDestTCP called _searchDest
  • implement code calling Callback :: notify () when resp to search request arrives
  • implement SearchDest :: searchRequest () that issues a tcp search request if server is capable, else directly calls the Callback :: notify ()

• udpiiu changes
  • private member class deriving from SearchDest called SearchDestUDP
    • instances on the search list
  • private member class deriving from SearchDest :: Callback called SearchRespCallback
    • implement SearchRespCallback  :: notify ()
  • implement code calling Callback :: notify () when resp to search request arrives
  • build udp search dest objects
    • remove "dest" member data
    • add private tsDLLList < SearchDest > list called _searchDestList
    • build address list ELLLIST using iocinf.c code parsing addr list and traversing NICs
    • for each member of this list create private SearchDest derived objects
    • install the SearchDest derived objects in _searchDestList
    • destroy ELLLIST members
  • add public register/unregister SearchDest to udpiiu (called by cac)

• cac changes
  • code parsing EPICS_CA_NAME_SERVERS (in constructor)
  • creates SearchDestTCP and adds it to the search list
  • creates tcpiiu specifying the SeachDestTCP to its constructor
  • add public register/unregister SearchDest to cac (called by tcpiiu const and dest)
  • in destructor guarantee that udpiiu is shutdown prior to deleting nameService tcpiiu

CAS

• casStrmClient changes
  • add private client addr member (init'd through constructor arg)
  • add casStrmClient::searchAction that does roughly the same as the casDGClient::searchAction (probably this should be moved to separate name resolver class to avoid code repetition)
  • add casStrmClient::asyncSearchResponse and casStrmClient::searchResponse (again: same as casDGClient methods, should be moved to a resolver than both can use)
• casStreamIO changes
  • add private client address member
  • add public methods to get client host name and address

rsrv

• camessage.c changes
  • rename search_reply() to search_reply_udp()
  • add search_reply_tcp() with similar functionality
  • change jump table accordingly

Issues

Fixed: TCP Nameserver Connections Do Not Reconnect

When a TCP nameserver connection goes away (e.g. if the nameserver dies), the client does not reconnect when the connection comes back (e.g. the nameserver is restarted).

Fix (2010-04-09): The private member SearchDestTCP::_addr must be an osiSockAddr, not a reference to it. The constructor cac::cac() frees the list of destinations after creating the SearchDestTCP instances. The SearchDestTCP couldn't reconnect, because it had lost the target address.

Fixed: TCP Nameserver Connection Down Blocks All Callbacks

When a TCP nameserver connection goes away (e.g. if the nameserver dies), the client's TCP receive thread tries to reconnect in an endless loop, doing a epicsThreadSleep(cac.connectionTimeout) between attempts. While the receive thread is sleeping, no other CA callbacks are executed.

Fix: (2010-04-15) tcpRecvThread::connect() holds the cac mutex lock. That lock must be released during the epicsThreadSleep().