EPICS networking

From Beam Line Controls
Revision as of 21:58, 26 February 2024 by Tguruswamy (talk | contribs) (→‎Potential issues: fix techtalk link)
Jump to navigation Jump to search

EPICS networking

For basic terminology, see EPICS 101.

An EPICS IOC is an application which serves EPICS clients (command-line programs like caget; applications using p4p, pyepics, pvapy, or other libraries; GUI display managers like MEDM or caQtDM) process variables over the network. They do so using either the Channel Access (CA) or PVAccess (PVA) network protocols.

By default, an EPICS IOC with CA and/or PVA support enabled will listen for clients on all available network interfaces​.

Connection Flow

  • A client emits a request for particular PV names, e.g. 13SIM1:cam1:Image​ to a specific "search list" of IP addresses (which may be unicast or broadcast addresses).
  • Any IOC which receives the request and provides any PV name requested will respond to the client​.
  • The client and Server now know each other's IP addresses​.
  • A one-to-one communications channel between client and server is established​.
  • The opened Channel is used for either a one-off READ or WRITE, or a SUBSCRIPTION (monitor) to receive an update every time the PV value changes​.

This flow is more or less the same for the CA or PVA protocols​, and involves a mix of TCP/IP and UDP/IP transport with varying port numbers. Many client libraries (e.g. pyepics) will cache connections, or convert repeated READs into SUBSCRIPTIONS behind-the-scenes to save resources.

This system ensures that a newly-started IOC is immediately accessible to all clients without needing to register with a central server, and the only information you need to connect to a PV is its full name.

The search request

EPICS clients follow a simple two-step process to construct the destination of a PV search request. This is performed separately for CA and PVA protocols (substitute either CA or PVA into the names of the environment variables described below).

  1. Start with an empty list.
  2. If the environment variable EPICS_{CA,PVA}_AUTO_ADDR_LIST=yes, then add all broadcast IP addresses for all available network interfaces to the list.
    • "yes" is the default value if unset.
    • If EPICS_{CA,PVA}_AUTO_ADDR_LIST=no, skip this step.
  3. Add the contents of the environment variable EPICS_{CA,PVA}_ADDR_LIST to the list.
    • This should be a space-separated list of IP addresses or hostnames.
  4. Send the PV search request to every IP or host in the list.

Environment variables should be set prior to starting the client program to be effective, and may need to be "exported" to persist (depending on your shell). Example in bash: 

   export EPICS_CA_AUTO_ADDR_LIST=no EPICS_CA_ADDR_LIST=s1pvgate.xray.aps.anl.gov
   caget 1BM:ExampleVariable

Potential issues

Crossing subnet boundaries

Since by default only broadcast addresses are in the search request destination list, and broadcast packets are not routed between subnets, without extra configuration you will only be able to find PVs on the same subnet as your client.

Solution 1: add the IP address of the IOC server directly to EPICS_{CA,PVA}_ADDR_LIST.
This works across subnet boundaries as long as the required ports are available (no firewalls), but doesn't scale well if there are many IOCs you want to connect to.
Solution 2: set up a PV Gateway server in the IOC subnet, and add the PV gateway IP address
A PV gateway is a dedicated server program which relays incoming requests on to the rest of the subnet. It can also apply security rules to the traffic it is forwarding, e.g. it can force a PV to be read-only. The PV gateway hostname or IP address must be added to the relevant ADDR_LIST for all clients.
Solution 3: set up a PV nameserver.
Similar to DNS, the nameserver is a server program that takes over responding to all client name search requests. In this setup, the nameserver must be added to the ADDR_LIST for all clients.

Multiple paths between client and server

An IOC will reply to all requests for PVs it servers, even multiple ones from the same client.

Therefore if there are multiple network paths from client to server, for example because the client and/or server have multiple network interfaces, the server may see multiple requests from the client and the client will get multiple responses, each with a different IP.

The same situation can occur if multiple IOCs are serving a PV with the exact same name. The client will usually pick the first response it receives and print a warning to the user, for example: 

   12-12-2023 01:36:01 Channel Access Exception Channel: "bdpSim0:cam1:Model_RBV", Connecting to: beryl-gpm.xray.aps.anl.gov:5064, Ignored: 10.0.0.1:5064 on ? (op=5 data_type=DBR_invalid count=0)

Which server IP is selected is essentially random each time a search request completes and a connection is created, and there is no guarantee it is the optimal or fastest path from client to server.

Solution: limit EPICS traffic to only one interface and one network path.
On the client, set EPICS_{CA,PVA}_AUTO_ADDR_LIST=no and manually set EPICS_{CA,PVA}_ADDR_LIST to the single broadcast address on the network interface you want to use.

It is also possible to set EPICS_{CA,PVA}S_INTF_ADDR_LIST on the server to limit the addresses the IOC listens on, but this must be done before starting the IOC.

The most common variant of this situation is when the client and server are on the same computer. In this case, all the available network interfaces (including the loopback interface lo) present a valid path from client to server.

To limit all traffic to only travel over the loopback interface (which will be the fastest), before running your client set: 

  EPICS_CA_ADDR_LIST=127.255.255.255​
  EPICS_PVA_ADDR_LIST=127.255.255.255​
  EPICS_CA_AUTO_ADDR_LIST=no​
  EPICS_PVA_AUTO_ADDR_LIST=no

The difference between using only the loopback address (127.0.0.1) and using the loopback broadcast address (127.255.255.255) is that the broadcast address supports communicating with multiple IOCs on the same host on Linux. See the EPICS documentation for more details and alternative solutions.

Note: Limiting traffic to 127.0.0.1/127.255.255.255 may not work for the EPICSv7 built-in pvaServer due to an issue in the implementation, in which case try a different local IP. See TechTalk: [1] [2]

References

Retrieved from "https://wiki-ext.aps.anl.gov/blc/index.php?title=EPICS_networking&oldid=759"