How to Set Up a Soft IOC Framework on Linux

This evolved from my notes when doing this for the soft IOCs at BESSY. Please add or correct things as you find them wrong or out-of-date.

The following instructions are based on our Debian Linux machines. (Which version? I don't remember. Too stable, I guess.) Other distributions (or other Unixes) might have different commands and different places for things. This is especially true for the Debian /etc/init.d script I'm attaching to this page. If you create a different script for a different distribution, please add it to this page. Others will be able to use it. The general steps will be the same on all distributions, though.

Knowledge of general system administration tasks (creating user accounts etc.) is assumed.

Introduction

Why are we doing this?

When using soft IOCs in production, they should be treated as important system services:

• Soft IOCs should be started and stopped by the system.
• There should be a fallback system you can easily switch over to in case of hardware failures.

Other objectives were: In the same way as for VME IOCs, the application developer should be able to reset the soft IOC without needing root access to the host.

• The IOC application developer should be able to start and stop IOCs manually.

When multiple soft IOCs share the same host (and the same IP address), Channel Access can not tell them apart. Access Security will not be able to distiguish between CA connections coming from different soft IOCs. When debugging CA clients, CA will not be able to tell you which of the soft IOCs a connection goes to.

• Channel Access should be able to distinguish between different soft IOCs, even if they are hosted on the same machine.

I was considering using a virtualization layer (based on VMware) to allow running soft IOCs in an encapsulated environment. I found the effort too high, the layer too thick, and the expected performance hit too hard - only to get a separate IP address for each soft IOC.

When debugging and/or trying to look what is happening on an IOC, the developer does not necessarily know if the database is running on a VME based or on a host based soft IOC.

• Console access (and logging console output) should be uniform: working the same way for soft as for VME IOCs.

The setup necessary to achieve this is described in the document How to Set Up Console Access and Logging for VME and Soft IOCs.

Concept

To allow Access Security telling the soft IOCs apart, they are run under separate user names.

The screen utility will be used as an environment that allows to start soft IOCs in the background and connect to their consoles later, much like the serial consoles of VME IOCs.

Attaching to a soft IOC console will be done through ssh, using a special console access key. Ssh is set up with the matching screen commands that reattach to the soft IOCs. Opening an ssh connection using the console access key to the user ioc123 on the soft IOC host will immediately attach to the console of the soft IOC named ioc123 (that is running as user ioc123).

Setting up Your Machine

Create User Accounts and ssh Access

Soft IOC Administrator Account

Create a generic user account that application developers will use to start/stop soft IOCs. (We call it iocadm.)

Put the public ssh keys of the application developers into ~/.ssh/authorized_keys of iocadm.

ssh Key Pairs

As iocadm, create one key pair for this user, and another key pair for console access.

Soft IOCs

Create one user account for each soft IOC you intend to host. User name should be the IOC name, the group is not really important. (Maybe create a group iocs that you put all of them into?)

Into each of the ~/.ssh/authorized_keys files, put two public keys:

1. The public ssh key of iocadm.
2. The public ssh key for console access.

In front of the console access key, put the screen command to reattach to the soft IOC console. For a user/IOC ioc123, the line should look like this:

command="screen -r ioc123" ssh-rsa AAAAB3NzaC1yc2EAAAA.....

Configure the sudo Facility

Allow the iocadm User to Start and Stop Soft IOCs

On the soft IOC host, allow iocadm to use sudo to execute commands as any of the soft IOC users. /etc/sudoers should have a line like:

iocadm ALL = (ioc123, ioc124, ioc125) NOPASSWD: ALL

Setup the Start/Stop script

Create the /etc/init.d script

I'm attaching the script that we're using as /etc/init.d/softIOC. It got quite huge and complex - sorry!. It has been modelled after Debian's skeleton scripts, you should probably adapt it to match the standards that your distribution implies.

It contains the local settings for where to find things, routines to read in the configuration file, the code necessary to start/stop a soft IOC as a different user within a named screen session, and the usual init.d script stuff that checks command line arguments and calls the other routines.

If you have a script working for a different distribution, please add it to this page, as it could make life easier for others!

Create the Configuration File

The configuration file contains a section for each of the soft IOCs. A section starts with the IOC name followed by a colon, and ends with an empty line.

Within a section you can set special variables used by the softIOC script as well as environment variables that will be set for the soft IOC.

The special section global: contains settings that will be applied to all soft IOCs (may be overridden by the IOC section).

The special line auto: contains the names of the soft IOCs that should be started when the script is run as part of the system boot-up process.

Section and IOC names are not case sensitive.

So a minimal configuration file could look like this (remember the empty line that is required after each section):

AUTO:   ioc123

GLOBAL:

ioc123:

ioc124:

ioc125:

Distribute the Required Stuff to the Soft IOC Host

EPICS Base

Soft IOCS will need libraries from EPICS base. Make sure these are existing and can be found.

Code and Databases

Add the soft IOC host to the code deployment scheme you are using. The soft IOC binaries, databases, and start up scripts must be available for the soft IOCs to be started.

Start Your Soft IOCs

Start the IOCs using the startup script

Starting and stopping the soft IOCs should work now! Ssh to the soft IOC host as iocadm and try calling the startup script:

/etc/init.d/softIOC start ioc123

Watch them run

Ssh to the soft IOC host using the console access key and see if you can get access to the IOC console:

ssh -i ~iocadm/.ssh/console_access -t ioc123@iochost

You should be directly connected to your IOC's console.

Check if Starting IOCs at reboot works

If you made entries to the auto: section, reboot the machine to check that starting IOCs at boot time works.

Good luck!

Ralph Lange (BESSY)