Troubleshooting EPICS IOCs

Understanding the structure of a deployed EPICS IOC and environment, determining configuration, and parsing error messages.

Linux-specific

Examine the environment with env

EPICS reads several "environment variables" during operation. You can examine the current list of environment variables in a terminal (either bash or csh) using env.

To get the value of a particular environment variable, try env | grep VARIABLE or echo $VARIABLE .

Particularly relevant ones are:

PATH

list of folders which are searched for executable scripts and binaries. If you are seeing "command not found"-type errors, consider if $PATH is set correctly.

EPICS_CA_ADDR_LIST

IP addresses that are contacted when searching for a CA (channel-access) variable. Important for any program which is accessing CA variables provided by a different program, possibly on a different host.

EPICS_CA_AUTO_ADDR_LIST

(yes | no). Whether to also include the broadcast address for all configured network interfaces in the CA_ADDR_LIST. This automatically searches all computers on the same subnet for variables.

EPICS_PVA_ADDR_LIST

As above, but for PVaccess variables.

EPICS_PVA_AUTO_ADDR_LIST

As above, but for automatic use of broadcast addresses for PVaccess variables.

For example, when examining a GUI which should only connect to the channel access variables of a local instance of an IOC, you might expect to see a configuration like:

EPICS_CA_AUTO_ADDR_LIST=no
EPICS_CA_ADDR_LIST=127.0.0.255

Find library dependencies with ldd

Sometimes it is necessary to determine which versions of a shared library are linked into the IOC binary. On other occasions, an IOC binary will not run (typically, an immediate segfault) because a necessary linked library is missing. ldd is a Linux tool which will list all linked dynamic (shared) libraries of a binary, to answer these questions. Execute ldd -r against the IOC binary (not startup script) and examine its output.

Example:

$ cd ~/epics/synApps_6_2_1/ioc/IocSpinnaker/
$ ldd -r bin/rhel8-x86_64/SpinnakerApp
      linux-vdso.so.1 (0x00007ffffb195000)
      libADSpinnaker.so => /net/s100dserv/APSshare/epics/synApps_6_2_1/support/areaDetector-R3-12-1/ADSpinnaker/lib/rhel8-x86_64/libADSpinnaker.so (0x00007f149e6f9000)
      libADGenICam.so => /net/s100dserv/APSshare/epics/synApps_6_2_1/support/areaDetector-R3-12-1/ADGenICam/lib/rhel8-x86_64/libADGenICam.so (0x00007f149e4e6000)
      ...

The location of all found libraries will be listed in full, while any missing libraries will be noted:

      libfoo.so => not found

View the full IOC output when running in screen

By default, IOCs built from the XXX template are controlled via a startup shell script:

ExampleIOC.sh (start|stop|status)

When you start an IOC this way, no output is immediately visible, and control returns to your terminal. It is possible to connect to the IOC console via

ExampleIOC.sh console

which opens a connection to screen, the Linux terminal multiplexer. Type exit at the IOC prompt to exit, or use Ctrl-a Ctrl-d (detach) to leave the IOC running in the background and return to the terminal. Ctrl-a [ will change screen to scrollback-mode, where you will be able to examine the stored history of IOC output. Use / to search forward through the text, and ? to search backwards, for example to find errors.