Beam Line Controls - User contributions [en]

SciStream at APS

2025-04-14T18:29:31Z

Tguruswamy: /* Outbound Request ("Client side": 192.168.150.2) */

= SciStream Tunnel Setup Guide =

This guide provides step-by-step instructions on how to tunnel a connection through SciStream using <code>s2uc</code> running from a container. This setup involves two endpoints, each running on separate hosts, to establish a secure tunnel(TLS) for forwarding packets.

Scistream's API TLS certificates are on the test servers at <code>/local/scistream</code>. Use the folder corresponding to the server -- i.e. on server1, always use /server1; on server2, use /server2.

== How data flows through the tunnel ==

The tunnel consists of parts that work together:

'''Server side (Inbound Request):''' Destination that receives data. TCP server listens for connections.
'''Client side (Outbound Request):''' Source that sends data. TCP client that actively creates new connections

** Your client application connects to the local port (5100) on the client machine
** The SciStream outbound request forwards this connection through the secure tunnel
** The SciStream inbound request receives the forwarded connection
** Data is delivered to your server application listening on the receiver port (5300)

== Request Configuration ==

=== Inbound Request ("Server side": 192.168.150.1) ===
'''Purpose:''' Sets up the receiving end of the tunnel. This establishes a listener on the specified SciStream endpoint that waits for incoming connections from the outbound request (other SciStream endpoint) and it forwards this connection to the <remote_ip>:<receiver_ports>.

<pre>podman run --rm --net=host -v ./server1:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 inbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.1:5000 --receiver_ports 5300 --num_conn 1 --server_cert=/scistream/server.crt
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The address the user application is listening on.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: The port the user application is listening on (i.e. the user application is bound to {remote_ip:receiver_port}).
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.

'''Note the UID:'''

After running the command, note the UID and listener port provided in the output. Example:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5000 INVALID_TOKEN PROD
sending client request message
started client request
waiting for hello message
sending for hello message
Hello message sent successfully
Listeners: ['192.168.150.1:5100']
</pre>

=== Outbound Request ("Client side": 192.168.150.2) ===

Purpose: Sets up the sending end of the tunnel. This establishes a listener on the specified SciStream endpoint that waits for incoming connections from the client application and forwards that data to the other SciStream endpoint.

<pre>podman run --rm --net=host -v ./server2:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 outbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.2:5000 --receiver_ports 5100 --num_conn 1 --server_cert=/scistream/server.crt bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5100
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The eventual destination IP address we want data to be forwarded to.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: Local port which the client application will connect to (i.e. client should connect to {s2cs_ip:receiver_port}).
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.
** <code>UID</code>: Use the UID noted from the inbound request.
** <code>IP:PORT</code>: Use the IP and port noted from the inbound request.

Example output:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.2:5000 INVALID_TOKEN CONS
started client request
waiting for hello message
Hello message sent successfully
Listeners: ['192.168.150.2:5100']
</pre>

== Verification ==

* '''Check Listening Ports:'''

On both hosts, verify that the <code>stunnel</code> processes are listening on the expected ports using:

<pre>ss -tlpn
</pre>

* '''Test the Tunnel:'''

Use <code>iperf3</code> to test the tunnel:

* '''Client Side (Host 192.168.150.2):'''

<pre>iperf3 -c 192.168.150.2 -p 5100
</pre>

* '''Server Side (Host 192.168.150.1):'''

<pre>iperf3 -s -p 5300
</pre>

== Closing the Connection ==

To close a connection request, use the <code>release</code> option with <code>s2uc</code>:

<pre>s2uc release <uid>
</pre>

SciStream at APS

2025-04-14T18:29:20Z

Tguruswamy: /* Inbound Request ("Server side": 192.168.150.1) */

= SciStream Tunnel Setup Guide =

This guide provides step-by-step instructions on how to tunnel a connection through SciStream using <code>s2uc</code> running from a container. This setup involves two endpoints, each running on separate hosts, to establish a secure tunnel(TLS) for forwarding packets.

Scistream's API TLS certificates are on the test servers at <code>/local/scistream</code>. Use the folder corresponding to the server -- i.e. on server1, always use /server1; on server2, use /server2.

== How data flows through the tunnel ==

The tunnel consists of parts that work together:

'''Server side (Inbound Request):''' Destination that receives data. TCP server listens for connections.
'''Client side (Outbound Request):''' Source that sends data. TCP client that actively creates new connections

** Your client application connects to the local port (5100) on the client machine
** The SciStream outbound request forwards this connection through the secure tunnel
** The SciStream inbound request receives the forwarded connection
** Data is delivered to your server application listening on the receiver port (5300)

== Request Configuration ==

=== Inbound Request ("Server side": 192.168.150.1) ===
'''Purpose:''' Sets up the receiving end of the tunnel. This establishes a listener on the specified SciStream endpoint that waits for incoming connections from the outbound request (other SciStream endpoint) and it forwards this connection to the <remote_ip>:<receiver_ports>.

<pre>podman run --rm --net=host -v ./server1:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 inbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.1:5000 --receiver_ports 5300 --num_conn 1 --server_cert=/scistream/server.crt
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The address the user application is listening on.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: The port the user application is listening on (i.e. the user application is bound to {remote_ip:receiver_port}).
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.

'''Note the UID:'''

After running the command, note the UID and listener port provided in the output. Example:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5000 INVALID_TOKEN PROD
sending client request message
started client request
waiting for hello message
sending for hello message
Hello message sent successfully
Listeners: ['192.168.150.1:5100']
</pre>

=== Outbound Request ("Client side": 192.168.150.2) ===

Purpose: Sets up the sending end of the tunnel. This establishes a listener on the local SciStream endpoint that waits for incoming connections from the client application and forwards that data to the other SciStream endpoint.

<pre>podman run --rm --net=host -v ./server2:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 outbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.2:5000 --receiver_ports 5100 --num_conn 1 --server_cert=/scistream/server.crt bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5100
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The eventual destination IP address we want data to be forwarded to.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: Local port which the client application will connect to (i.e. client should connect to {s2cs_ip:receiver_port}).
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.
** <code>UID</code>: Use the UID noted from the inbound request.
** <code>IP:PORT</code>: Use the IP and port noted from the inbound request.

Example output:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.2:5000 INVALID_TOKEN CONS
started client request
waiting for hello message
Hello message sent successfully
Listeners: ['192.168.150.2:5100']
</pre>

== Verification ==

* '''Check Listening Ports:'''

On both hosts, verify that the <code>stunnel</code> processes are listening on the expected ports using:

<pre>ss -tlpn
</pre>

* '''Test the Tunnel:'''

Use <code>iperf3</code> to test the tunnel:

* '''Client Side (Host 192.168.150.2):'''

<pre>iperf3 -c 192.168.150.2 -p 5100
</pre>

* '''Server Side (Host 192.168.150.1):'''

<pre>iperf3 -s -p 5300
</pre>

== Closing the Connection ==

To close a connection request, use the <code>release</code> option with <code>s2uc</code>:

<pre>s2uc release <uid>
</pre>

SciStream at APS

2025-04-14T18:28:57Z

Tguruswamy: /* Inbound Request ("Server side": 192.168.150.1) */

= SciStream Tunnel Setup Guide =

This guide provides step-by-step instructions on how to tunnel a connection through SciStream using <code>s2uc</code> running from a container. This setup involves two endpoints, each running on separate hosts, to establish a secure tunnel(TLS) for forwarding packets.

Scistream's API TLS certificates are on the test servers at <code>/local/scistream</code>. Use the folder corresponding to the server -- i.e. on server1, always use /server1; on server2, use /server2.

== How data flows through the tunnel ==

The tunnel consists of parts that work together:

'''Server side (Inbound Request):''' Destination that receives data. TCP server listens for connections.
'''Client side (Outbound Request):''' Source that sends data. TCP client that actively creates new connections

** Your client application connects to the local port (5100) on the client machine
** The SciStream outbound request forwards this connection through the secure tunnel
** The SciStream inbound request receives the forwarded connection
** Data is delivered to your server application listening on the receiver port (5300)

== Request Configuration ==

=== Inbound Request ("Server side": 192.168.150.1) ===
'''Purpose:''' Sets up the receiving end of the tunnel. This establishes a listener that waits for incoming connections from the outbound request (other SciStream endpoint) and it forwards this connection to the <remote_ip>:<receiver_ports>.

<pre>podman run --rm --net=host -v ./server1:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 inbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.1:5000 --receiver_ports 5300 --num_conn 1 --server_cert=/scistream/server.crt
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The address the user application is listening on.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: The port the user application is listening on (i.e. the user application is bound to {remote_ip:receiver_port}).
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.

'''Note the UID:'''

After running the command, note the UID and listener port provided in the output. Example:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5000 INVALID_TOKEN PROD
sending client request message
started client request
waiting for hello message
sending for hello message
Hello message sent successfully
Listeners: ['192.168.150.1:5100']
</pre>

=== Outbound Request ("Client side": 192.168.150.2) ===

Purpose: Sets up the sending end of the tunnel. This establishes a listener on the local SciStream endpoint that waits for incoming connections from the client application and forwards that data to the other SciStream endpoint.

<pre>podman run --rm --net=host -v ./server2:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 outbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.2:5000 --receiver_ports 5100 --num_conn 1 --server_cert=/scistream/server.crt bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5100
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The eventual destination IP address we want data to be forwarded to.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: Local port which the client application will connect to (i.e. client should connect to {s2cs_ip:receiver_port}).
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.
** <code>UID</code>: Use the UID noted from the inbound request.
** <code>IP:PORT</code>: Use the IP and port noted from the inbound request.

Example output:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.2:5000 INVALID_TOKEN CONS
started client request
waiting for hello message
Hello message sent successfully
Listeners: ['192.168.150.2:5100']
</pre>

== Verification ==

* '''Check Listening Ports:'''

On both hosts, verify that the <code>stunnel</code> processes are listening on the expected ports using:

<pre>ss -tlpn
</pre>

* '''Test the Tunnel:'''

Use <code>iperf3</code> to test the tunnel:

* '''Client Side (Host 192.168.150.2):'''

<pre>iperf3 -c 192.168.150.2 -p 5100
</pre>

* '''Server Side (Host 192.168.150.1):'''

<pre>iperf3 -s -p 5300
</pre>

== Closing the Connection ==

To close a connection request, use the <code>release</code> option with <code>s2uc</code>:

<pre>s2uc release <uid>
</pre>

SciStream at APS

2025-04-14T18:28:33Z

Tguruswamy: /* Outbound Request ("Client side": 192.168.150.2) */

= SciStream Tunnel Setup Guide =

This guide provides step-by-step instructions on how to tunnel a connection through SciStream using <code>s2uc</code> running from a container. This setup involves two endpoints, each running on separate hosts, to establish a secure tunnel(TLS) for forwarding packets.

Scistream's API TLS certificates are on the test servers at <code>/local/scistream</code>. Use the folder corresponding to the server -- i.e. on server1, always use /server1; on server2, use /server2.

== How data flows through the tunnel ==

The tunnel consists of parts that work together:

'''Server side (Inbound Request):''' Destination that receives data. TCP server listens for connections.
'''Client side (Outbound Request):''' Source that sends data. TCP client that actively creates new connections

** Your client application connects to the local port (5100) on the client machine
** The SciStream outbound request forwards this connection through the secure tunnel
** The SciStream inbound request receives the forwarded connection
** Data is delivered to your server application listening on the receiver port (5300)

== Request Configuration ==

=== Inbound Request ("Server side": 192.168.150.1) ===
'''Purpose:''' Sets up the receiving end of the tunnel. This establishes a listener that waits for incoming connections from the outbound request and it forwards this connection to the <remote_ip>:<receiver_ports>.

<pre>podman run --rm --net=host -v ./server1:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 inbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.1:5000 --receiver_ports 5300 --num_conn 1 --server_cert=/scistream/server.crt
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The address the user application is listening on.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: The port the user application is listening on (i.e. the user application is bound to {remote_ip:receiver_port}).
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.

'''Note the UID:'''

After running the command, note the UID and listener port provided in the output. Example:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5000 INVALID_TOKEN PROD
sending client request message
started client request
waiting for hello message
sending for hello message
Hello message sent successfully
Listeners: ['192.168.150.1:5100']
</pre>

=== Outbound Request ("Client side": 192.168.150.2) ===

Purpose: Sets up the sending end of the tunnel. This establishes a listener on the local SciStream endpoint that waits for incoming connections from the client application and forwards that data to the other SciStream endpoint.

<pre>podman run --rm --net=host -v ./server2:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 outbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.2:5000 --receiver_ports 5100 --num_conn 1 --server_cert=/scistream/server.crt bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5100
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The eventual destination IP address we want data to be forwarded to.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: Local port which the client application will connect to (i.e. client should connect to {s2cs_ip:receiver_port}).
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.
** <code>UID</code>: Use the UID noted from the inbound request.
** <code>IP:PORT</code>: Use the IP and port noted from the inbound request.

Example output:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.2:5000 INVALID_TOKEN CONS
started client request
waiting for hello message
Hello message sent successfully
Listeners: ['192.168.150.2:5100']
</pre>

== Verification ==

* '''Check Listening Ports:'''

On both hosts, verify that the <code>stunnel</code> processes are listening on the expected ports using:

<pre>ss -tlpn
</pre>

* '''Test the Tunnel:'''

Use <code>iperf3</code> to test the tunnel:

* '''Client Side (Host 192.168.150.2):'''

<pre>iperf3 -c 192.168.150.2 -p 5100
</pre>

* '''Server Side (Host 192.168.150.1):'''

<pre>iperf3 -s -p 5300
</pre>

== Closing the Connection ==

To close a connection request, use the <code>release</code> option with <code>s2uc</code>:

<pre>s2uc release <uid>
</pre>

SciStream at APS

2025-04-14T15:22:26Z

Tguruswamy: /* SciStream Tunnel Setup Guide */

= SciStream Tunnel Setup Guide =

This guide provides step-by-step instructions on how to tunnel a connection through SciStream using <code>s2uc</code> running from a container. This setup involves two endpoints, each running on separate hosts, to establish a secure tunnel for forwarding packets.

Scistream TLS certificates are on the test servers at <code>/local/scistream</code>. Use the folder corresponding to the server -- i.e. on server1, always use /server1; on server2, use /server2.

== Request Configuration ==

=== Inbound Request ("Server side": 192.168.150.1) ===

<pre>podman run --rm --net=host -v ./server1:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 inbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.1:5000 --receiver_ports 5300 --num_conn 1 --server_cert=/scistream/server.crt
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The address the user application is listening on.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: The port the user application is listening on (i.e. the user application is bound to {remote_ip:receiver_port}).
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.

'''Note the UID:'''

After running the command, note the UID and listener port provided in the output. Example:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5000 INVALID_TOKEN PROD
sending client request message
started client request
waiting for hello message
sending for hello message
Hello message sent successfully
Listeners: ['192.168.150.1:5100']
</pre>

=== Outbound Request ("Client side": 192.168.150.2) ===

<pre>podman run --rm --net=host -v ./server2:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 outbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.2:5000 --receiver_ports 5100 --num_conn 1 --server_cert=/scistream/server.crt bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5100
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The eventual destination IP address we want data to be forwarded to.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: Local port which the client application will connect to.
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.
** <code>UID</code>: Use the UID noted from the inbound request.
** <code>IP:PORT</code>: Use the IP and port noted from the inbound request.

Example output:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.2:5000 INVALID_TOKEN CONS
started client request
waiting for hello message
Hello message sent successfully
Listeners: ['192.168.150.2:5100']
</pre>

== Verification ==

* '''Check Listening Ports:'''

On both hosts, verify that the <code>stunnel</code> processes are listening on the expected ports using:

<pre>ss -tlpn
</pre>

* '''Test the Tunnel:'''

Use <code>iperf3</code> to test the tunnel:

* '''Client Side (Host 192.168.150.2):'''

<pre>iperf3 -c 192.168.150.2 -p 5100
</pre>

* '''Server Side (Host 192.168.150.1):'''

<pre>iperf3 -s -p 5300
</pre>

== Closing the Connection ==

To close a connection request, use the <code>release</code> option with <code>s2uc</code>:

<pre>s2uc release <uid>
</pre>

SciStream at APS

2025-04-14T15:21:59Z

Tguruswamy: remove server names

= SciStream Tunnel Setup Guide =

This guide provides step-by-step instructions on how to tunnel a connection through SciStream using <code>s2uc</code> running from a container. This setup involves two endpoints, each running on separate hosts, to establish a secure tunnel for forwarding packets.

Scistream test server certificates are on the test servers at <code>/local/scistream</code>. Use the folder corresponding to the server -- i.e. on server1, always use /server1; on server2, use /server2.

== Request Configuration ==

=== Inbound Request ("Server side": 192.168.150.1) ===

<pre>podman run --rm --net=host -v ./server1:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 inbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.1:5000 --receiver_ports 5300 --num_conn 1 --server_cert=/scistream/server.crt
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The address the user application is listening on.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: The port the user application is listening on (i.e. the user application is bound to {remote_ip:receiver_port}).
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.

'''Note the UID:'''

After running the command, note the UID and listener port provided in the output. Example:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5000 INVALID_TOKEN PROD
sending client request message
started client request
waiting for hello message
sending for hello message
Hello message sent successfully
Listeners: ['192.168.150.1:5100']
</pre>

=== Outbound Request ("Client side": 192.168.150.2) ===

<pre>podman run --rm --net=host -v ./server2:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 outbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.2:5000 --receiver_ports 5100 --num_conn 1 --server_cert=/scistream/server.crt bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5100
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The eventual destination IP address we want data to be forwarded to.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: Local port which the client application will connect to.
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.
** <code>UID</code>: Use the UID noted from the inbound request.
** <code>IP:PORT</code>: Use the IP and port noted from the inbound request.

Example output:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.2:5000 INVALID_TOKEN CONS
started client request
waiting for hello message
Hello message sent successfully
Listeners: ['192.168.150.2:5100']
</pre>

== Verification ==

* '''Check Listening Ports:'''

On both hosts, verify that the <code>stunnel</code> processes are listening on the expected ports using:

<pre>ss -tlpn
</pre>

* '''Test the Tunnel:'''

Use <code>iperf3</code> to test the tunnel:

* '''Client Side (Host 192.168.150.2):'''

<pre>iperf3 -c 192.168.150.2 -p 5100
</pre>

* '''Server Side (Host 192.168.150.1):'''

<pre>iperf3 -s -p 5300
</pre>

== Closing the Connection ==

To close a connection request, use the <code>release</code> option with <code>s2uc</code>:

<pre>s2uc release <uid>
</pre>

SciStream at APS

2025-04-09T19:22:09Z

Tguruswamy: add cert location

= SciStream Tunnel Setup Guide =

This guide provides step-by-step instructions on how to tunnel a connection through SciStream using <code>s2uc</code> running from a container. This setup involves two endpoints, each running on separate hosts, to establish a secure tunnel for forwarding packets.

Scistream test server certificates are on updraft1/2 at <code>/local/scistream</code>. Use the folder corresponding to the server -- i.e. on updraft1, always use /server1; on updraft2, use /server2.

== Request Configuration ==

=== Inbound Request ("Server side": 192.168.150.1) ===

<pre>podman run --rm --net=host -v ./server1:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 inbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.1:5000 --receiver_ports 5300 --num_conn 1 --server_cert=/scistream/server.crt
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The address the user application is listening on.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: The port the user application is listening on (i.e. the user application is bound to {remote_ip:receiver_port}).
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.

'''Note the UID:'''

After running the command, note the UID and listener port provided in the output. Example:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5000 INVALID_TOKEN PROD
sending client request message
started client request
waiting for hello message
sending for hello message
Hello message sent successfully
Listeners: ['192.168.150.1:5100']
</pre>

=== Outbound Request ("Client side": 192.168.150.2) ===

<pre>podman run --rm --net=host -v ./server2:/scistream --entrypoint=s2uc castroflaviojr/scistream:1.2.1 outbound-request --remote_ip 192.168.150.1 --s2cs 192.168.150.2:5000 --receiver_ports 5100 --num_conn 1 --server_cert=/scistream/server.crt bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.1:5100
</pre>

* '''Parameters:'''
** <code>--remote_ip</code>: The eventual destination IP address we want data to be forwarded to.
** <code>--s2cs</code>: The IP and port for the local s2cs scistream control server.
** <code>--receiver_ports</code>: Local port which the client application will connect to.
** <code>--num_conn</code>: Number of connections to establish.
** <code>--server_cert</code>: Path to the server certificate for TLS.
** <code>UID</code>: Use the UID noted from the inbound request.
** <code>IP:PORT</code>: Use the IP and port noted from the inbound request.

Example output:

<pre>
uid; s2cs; access_token; role
bd9f1a7e-04d7-11f0-b44c-946dae415862 192.168.150.2:5000 INVALID_TOKEN CONS
started client request
waiting for hello message
Hello message sent successfully
Listeners: ['192.168.150.2:5100']
</pre>

== Verification ==

* '''Check Listening Ports:'''

On both hosts, verify that the <code>stunnel</code> processes are listening on the expected ports using:

<pre>ss -tlpn
</pre>

* '''Test the Tunnel:'''

Use <code>iperf3</code> to test the tunnel:

* '''Client Side (Host 192.168.150.2):'''

<pre>iperf3 -c 192.168.150.2 -p 5100
</pre>

* '''Server Side (Host 192.168.150.1):'''

<pre>iperf3 -s -p 5300
</pre>

== Closing the Connection ==

To close a connection request, use the <code>release</code> option with <code>s2uc</code>:

<pre>s2uc release <uid>
</pre>

SciStream at APS

2025-04-09T19:17:45Z

Tguruswamy: from talks with flavio

EPICS networking

2024-02-27T17:42:03Z

Tguruswamy: /* Multiple paths between client and server */ adjust note about pva localhost broadcast

= EPICS networking =

For basic terminology, see [[EPICS 101]].

An EPICS IOC is an application which serves EPICS clients (command-line programs like caget; applications using p4p, pyepics, pvapy, or other libraries; GUI display managers like MEDM or caQtDM) process variables over the network. They do so using either the Channel Access (CA) or PVAccess (PVA) network protocols.

By default, an EPICS IOC with CA and/or PVA support enabled will listen for clients on '''all available network interfaces.'''

== Connection Flow ==

* A client emits a request for particular PV names, e.g. ''13SIM1:cam1:Image'' to a specific "search list" of IP addresses (which may be unicast or broadcast addresses).
* Any IOC which receives the request and provides any PV name requested will respond to the client.
* The client and Server now know each other's IP addresses.
* A one-to-one communications channel between client and server is established.
* The opened Channel is used for either a one-off READ or WRITE, or a SUBSCRIPTION (monitor) to receive an update every time the PV value changes.

This flow is more or less the same for the CA or PVA protocols, and involves a mix of TCP/IP and UDP/IP transport with varying port numbers. Many client libraries (e.g. pyepics) will cache connections, or convert repeated READs into SUBSCRIPTIONS behind-the-scenes to save resources.

This system ensures that a newly-started IOC is immediately accessible to all clients without needing to register with a central server, and the only information you need to connect to a PV is its full name.

== The search request ==

EPICS clients follow a simple two-step process to construct the destination of a PV search request. This is performed separately for CA and PVA protocols (substitute ''either'' <code>CA</code> or <code>PVA</code> into the names of the environment variables described below).

# Start with an empty list.
# If the environment variable <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=yes</code>, then add all broadcast IP addresses for all available network interfaces to the list.
#* "yes" is the default value if unset.
#* If <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code>, skip this step.
# Add the contents of the environment variable <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the list.
#* This should be a space-separated list of IP addresses or hostnames.
# Send the PV search request to every IP or host in the list.

Environment variables should be set prior to starting the client program to be effective, and may need to be "exported" to persist (depending on your shell). Example in bash:

export EPICS_CA_AUTO_ADDR_LIST=no EPICS_CA_ADDR_LIST=s1pvgate.xray.aps.anl.gov
caget 1BM:ExampleVariable

== Potential issues ==
=== Crossing subnet boundaries ===

Since by default only broadcast addresses are in the search request destination list, and broadcast packets are not routed between subnets, without extra configuration you will only be able to find PVs on the same subnet as your client.

;Solution 1: add the IP address of the IOC server directly to <code>EPICS_{CA,PVA}_ADDR_LIST</code>.
:This works across subnet boundaries as long as the required ports are available (no firewalls), but doesn't scale well if there are many IOCs you want to connect to.

;Solution 2: set up a PV Gateway server in the IOC subnet, and add the PV gateway IP address
:A [https://github.com/epics-extensions/ca-gateway PV gateway] is a dedicated server program which relays incoming requests on to the rest of the subnet. It can also apply security rules to the traffic it is forwarding, e.g. it can force a PV to be read-only. The PV gateway hostname or IP address must be added to the relevant <code>ADDR_LIST</code> for all clients.

; Solution 3: set up a PV nameserver.
:Similar to DNS, the [https://github.com/epics-extensions/ca-nameserver nameserver] is a server program that takes over responding to all client name search requests. In this setup, the nameserver must be added to the <code>ADDR_LIST</code> for all clients.

=== Multiple paths between client and server ===

An IOC will reply to '''all''' requests for PVs it servers, even multiple ones from the same client.

Therefore if there are multiple network paths from client to server, for example because the client and/or server have multiple network interfaces, the server may see multiple requests from the client and the client will get multiple responses, each with a different IP.

The same situation can occur if multiple IOCs are serving a PV with the exact same name. The client will usually pick the first response it receives and print a warning to the user, for example:

12-12-2023 01:36:01 Channel Access Exception Channel: "bdpSim0:cam1:Model_RBV", Connecting to: beryl-gpm.xray.aps.anl.gov:5064, Ignored: 10.0.0.1:5064 on ? (op=5 data_type=DBR_invalid count=0)

Which server IP is selected is essentially random each time a search request completes and a connection is created, and there is no guarantee it is the optimal or fastest path from client to server.

;Solution: limit EPICS traffic to only one interface and one network path.
:On the client, set <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code> and manually set <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the single broadcast address on the network interface you want to use.

It is also possible to set <code>EPICS_{CA,PVA}S_INTF_ADDR_LIST</code> on the server to limit the addresses the IOC listens on, but this must be done ''before'' starting the IOC.

The most common variant of this situation is when the client and server are on the same computer. In this case, all the available network interfaces (including the loopback interface ''lo'') present a valid path from client to server.

To limit all traffic to only travel over the loopback interface (which will be the fastest), before running your client set:

EPICS_CA_ADDR_LIST=127.255.255.255
EPICS_PVA_ADDR_LIST=127.0.0.1
EPICS_CA_AUTO_ADDR_LIST=no
EPICS_PVA_AUTO_ADDR_LIST=no

The difference between using only the loopback address (127.0.0.1) and using the loopback broadcast address (127.255.255.255) is that the broadcast address supports communicating with multiple IOCs on the same host on Linux. See the [https://docs.epics-controls.org/en/latest/sys-admin/channel-access-reach-multiple-soft-iocs-linux.html EPICS documentation] for more details and alternative solutions.

Note: PVA is different to CA in this example because using the broadcast address 127.255.255.255 does not work for the EPICSv7 built-in pvaServer due to an issue in the implementation. See TechTalk: [https://epics.anl.gov/tech-talk/2021/msg00806.php] [https://epics.anl.gov/tech-talk/2021/msg00807.php]

== References ==

* [https://epics.anl.gov/base/R3-14/12-docs/CAref.html EPICS Channel Access Reference Manual (v3.14)]
* [https://mdavidsaver.github.io/pvxs/netconfig.html PVXS (PVA) Network Configuration Information]

EPICS networking

2024-02-26T21:58:00Z

Tguruswamy: /* Potential issues */ fix techtalk link

= EPICS networking =

For basic terminology, see [[EPICS 101]].

An EPICS IOC is an application which serves EPICS clients (command-line programs like caget; applications using p4p, pyepics, pvapy, or other libraries; GUI display managers like MEDM or caQtDM) process variables over the network. They do so using either the Channel Access (CA) or PVAccess (PVA) network protocols.

By default, an EPICS IOC with CA and/or PVA support enabled will listen for clients on '''all available network interfaces.'''

== Connection Flow ==

* A client emits a request for particular PV names, e.g. ''13SIM1:cam1:Image'' to a specific "search list" of IP addresses (which may be unicast or broadcast addresses).
* Any IOC which receives the request and provides any PV name requested will respond to the client.
* The client and Server now know each other's IP addresses.
* A one-to-one communications channel between client and server is established.
* The opened Channel is used for either a one-off READ or WRITE, or a SUBSCRIPTION (monitor) to receive an update every time the PV value changes.

This flow is more or less the same for the CA or PVA protocols, and involves a mix of TCP/IP and UDP/IP transport with varying port numbers. Many client libraries (e.g. pyepics) will cache connections, or convert repeated READs into SUBSCRIPTIONS behind-the-scenes to save resources.

This system ensures that a newly-started IOC is immediately accessible to all clients without needing to register with a central server, and the only information you need to connect to a PV is its full name.

== The search request ==

EPICS clients follow a simple two-step process to construct the destination of a PV search request. This is performed separately for CA and PVA protocols (substitute ''either'' <code>CA</code> or <code>PVA</code> into the names of the environment variables described below).

# Start with an empty list.
# If the environment variable <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=yes</code>, then add all broadcast IP addresses for all available network interfaces to the list.
#* "yes" is the default value if unset.
#* If <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code>, skip this step.
# Add the contents of the environment variable <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the list.
#* This should be a space-separated list of IP addresses or hostnames.
# Send the PV search request to every IP or host in the list.

Environment variables should be set prior to starting the client program to be effective, and may need to be "exported" to persist (depending on your shell). Example in bash:

export EPICS_CA_AUTO_ADDR_LIST=no EPICS_CA_ADDR_LIST=s1pvgate.xray.aps.anl.gov
caget 1BM:ExampleVariable

== Potential issues ==
=== Crossing subnet boundaries ===

Since by default only broadcast addresses are in the search request destination list, and broadcast packets are not routed between subnets, without extra configuration you will only be able to find PVs on the same subnet as your client.

;Solution 1: add the IP address of the IOC server directly to <code>EPICS_{CA,PVA}_ADDR_LIST</code>.
:This works across subnet boundaries as long as the required ports are available (no firewalls), but doesn't scale well if there are many IOCs you want to connect to.

;Solution 2: set up a PV Gateway server in the IOC subnet, and add the PV gateway IP address
:A [https://github.com/epics-extensions/ca-gateway PV gateway] is a dedicated server program which relays incoming requests on to the rest of the subnet. It can also apply security rules to the traffic it is forwarding, e.g. it can force a PV to be read-only. The PV gateway hostname or IP address must be added to the relevant <code>ADDR_LIST</code> for all clients.

; Solution 3: set up a PV nameserver.
:Similar to DNS, the [https://github.com/epics-extensions/ca-nameserver nameserver] is a server program that takes over responding to all client name search requests. In this setup, the nameserver must be added to the <code>ADDR_LIST</code> for all clients.

=== Multiple paths between client and server ===

An IOC will reply to '''all''' requests for PVs it servers, even multiple ones from the same client.

Therefore if there are multiple network paths from client to server, for example because the client and/or server have multiple network interfaces, the server may see multiple requests from the client and the client will get multiple responses, each with a different IP.

The same situation can occur if multiple IOCs are serving a PV with the exact same name. The client will usually pick the first response it receives and print a warning to the user, for example:

12-12-2023 01:36:01 Channel Access Exception Channel: "bdpSim0:cam1:Model_RBV", Connecting to: beryl-gpm.xray.aps.anl.gov:5064, Ignored: 10.0.0.1:5064 on ? (op=5 data_type=DBR_invalid count=0)

Which server IP is selected is essentially random each time a search request completes and a connection is created, and there is no guarantee it is the optimal or fastest path from client to server.

;Solution: limit EPICS traffic to only one interface and one network path.
:On the client, set <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code> and manually set <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the single broadcast address on the network interface you want to use.

It is also possible to set <code>EPICS_{CA,PVA}S_INTF_ADDR_LIST</code> on the server to limit the addresses the IOC listens on, but this must be done ''before'' starting the IOC.

The most common variant of this situation is when the client and server are on the same computer. In this case, all the available network interfaces (including the loopback interface ''lo'') present a valid path from client to server.

To limit all traffic to only travel over the loopback interface (which will be the fastest), before running your client set:

EPICS_CA_ADDR_LIST=127.255.255.255
EPICS_PVA_ADDR_LIST=127.255.255.255
EPICS_CA_AUTO_ADDR_LIST=no
EPICS_PVA_AUTO_ADDR_LIST=no

The difference between using only the loopback address (127.0.0.1) and using the loopback broadcast address (127.255.255.255) is that the broadcast address supports communicating with multiple IOCs on the same host on Linux. See the [https://docs.epics-controls.org/en/latest/sys-admin/channel-access-reach-multiple-soft-iocs-linux.html EPICS documentation] for more details and alternative solutions.

Note: Limiting traffic to 127.0.0.1/127.255.255.255 may not work for the EPICSv7 built-in pvaServer due to an issue in the implementation, in which case try a different local IP. See TechTalk: [https://epics.anl.gov/tech-talk/2021/msg00806.php] [https://epics.anl.gov/tech-talk/2021/msg00807.php]

== References ==

* [https://epics.anl.gov/base/R3-14/12-docs/CAref.html EPICS Channel Access Reference Manual (v3.14)]
* [https://mdavidsaver.github.io/pvxs/netconfig.html PVXS (PVA) Network Configuration Information]

EPICS networking

2024-02-21T21:47:49Z

Tguruswamy: /* The search request */ reword

= EPICS networking =

For basic terminology, see [[EPICS 101]].

An EPICS IOC is an application which serves EPICS clients (command-line programs like caget; applications using p4p, pyepics, pvapy, or other libraries; GUI display managers like MEDM or caQtDM) process variables over the network. They do so using either the Channel Access (CA) or PVAccess (PVA) network protocols.

By default, an EPICS IOC with CA and/or PVA support enabled will listen for clients on '''all available network interfaces.'''

== Connection Flow ==

* A client emits a request for particular PV names, e.g. ''13SIM1:cam1:Image'' to a specific "search list" of IP addresses (which may be unicast or broadcast addresses).
* Any IOC which receives the request and provides any PV name requested will respond to the client.
* The client and Server now know each other's IP addresses.
* A one-to-one communications channel between client and server is established.
* The opened Channel is used for either a one-off READ or WRITE, or a SUBSCRIPTION (monitor) to receive an update every time the PV value changes.

This flow is more or less the same for the CA or PVA protocols, and involves a mix of TCP/IP and UDP/IP transport with varying port numbers. Many client libraries (e.g. pyepics) will cache connections, or convert repeated READs into SUBSCRIPTIONS behind-the-scenes to save resources.

This system ensures that a newly-started IOC is immediately accessible to all clients without needing to register with a central server, and the only information you need to connect to a PV is its full name.

== The search request ==

EPICS clients follow a simple two-step process to construct the destination of a PV search request. This is performed separately for CA and PVA protocols (substitute ''either'' <code>CA</code> or <code>PVA</code> into the names of the environment variables described below).

# Start with an empty list.
# If the environment variable <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=yes</code>, then add all broadcast IP addresses for all available network interfaces to the list.
#* "yes" is the default value if unset.
#* If <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code>, skip this step.
# Add the contents of the environment variable <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the list.
#* This should be a space-separated list of IP addresses or hostnames.
# Send the PV search request to every IP or host in the list.

Environment variables should be set prior to starting the client program to be effective, and may need to be "exported" to persist (depending on your shell). Example in bash:

export EPICS_CA_AUTO_ADDR_LIST=no EPICS_CA_ADDR_LIST=s1pvgate.xray.aps.anl.gov
caget 1BM:ExampleVariable

== Potential issues ==
=== Crossing subnet boundaries ===

Since by default only broadcast addresses are in the search request destination list, and broadcast packets are not routed between subnets, without extra configuration you will only be able to find PVs on the same subnet as your client.

;Solution 1: add the IP address of the IOC server directly to <code>EPICS_{CA,PVA}_ADDR_LIST</code>.
:This works across subnet boundaries as long as the required ports are available (no firewalls), but doesn't scale well if there are many IOCs you want to connect to.

;Solution 2: set up a PV Gateway server in the IOC subnet, and add the PV gateway IP address
:A [https://github.com/epics-extensions/ca-gateway PV gateway] is a dedicated server program which relays incoming requests on to the rest of the subnet. It can also apply security rules to the traffic it is forwarding, e.g. it can force a PV to be read-only. The PV gateway hostname or IP address must be added to the relevant <code>ADDR_LIST</code> for all clients.

; Solution 3: set up a PV nameserver.
:Similar to DNS, the [https://github.com/epics-extensions/ca-nameserver nameserver] is a server program that takes over responding to all client name search requests. In this setup, the nameserver must be added to the <code>ADDR_LIST</code> for all clients.

=== Multiple paths between client and server ===

An IOC will reply to '''all''' requests for PVs it servers, even multiple ones from the same client.

Therefore if there are multiple network paths from client to server, for example because the client and/or server have multiple network interfaces, the server may see multiple requests from the client and the client will get multiple responses, each with a different IP.

The same situation can occur if multiple IOCs are serving a PV with the exact same name. The client will usually pick the first response it receives and print a warning to the user, for example:

12-12-2023 01:36:01 Channel Access Exception Channel: "bdpSim0:cam1:Model_RBV", Connecting to: beryl-gpm.xray.aps.anl.gov:5064, Ignored: 10.0.0.1:5064 on ? (op=5 data_type=DBR_invalid count=0)

Which server IP is selected is essentially random each time a search request completes and a connection is created, and there is no guarantee it is the optimal or fastest path from client to server.

;Solution: limit EPICS traffic to only one interface and one network path.
:On the client, set <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code> and manually set <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the single broadcast address on the network interface you want to use.

It is also possible to set <code>EPICS_{CA,PVA}S_INTF_ADDR_LIST</code> on the server to limit the addresses the IOC listens on, but this must be done ''before'' starting the IOC.

The most common variant of this situation is when the client and server are on the same computer. In this case, all the available network interfaces (including the loopback interface ''lo'') present a valid path from client to server.

To limit all traffic to only travel over the loopback interface (which will be the fastest), before running your client set:

EPICS_CA_ADDR_LIST=127.255.255.255
EPICS_PVA_ADDR_LIST=127.255.255.255
EPICS_CA_AUTO_ADDR_LIST=no
EPICS_PVA_AUTO_ADDR_LIST=no

The difference between using only the loopback address (127.0.0.1) and using the loopback broadcast address (127.255.255.255) is that the broadcast address supports communicating with multiple IOCs on the same host on Linux. See the [https://docs.epics-controls.org/en/latest/sys-admin/channel-access-reach-multiple-soft-iocs-linux.html EPICS documentation] for more details and alternative solutions.

Note: Limiting traffic to 127.0.0.1/127.255.255.255 may not work for the EPICSv7 built-in pvaServer due to an issue in the implementation, in which case try a different local IP. See TechTalk: [https://epics.anl.gov/tech-talk/2021/msg00806.php] [https://epics.anl.gov/tech-talk/2021/msg00806.php]

== References ==

* [https://epics.anl.gov/base/R3-14/12-docs/CAref.html EPICS Channel Access Reference Manual (v3.14)]
* [https://mdavidsaver.github.io/pvxs/netconfig.html PVXS (PVA) Network Configuration Information]

EPICS networking

2024-02-21T21:37:12Z

Tguruswamy: /* Connection Flow */ mention ip search list

= EPICS networking =

For basic terminology, see [[EPICS 101]].

An EPICS IOC is an application which serves EPICS clients (command-line programs like caget; applications using p4p, pyepics, pvapy, or other libraries; GUI display managers like MEDM or caQtDM) process variables over the network. They do so using either the Channel Access (CA) or PVAccess (PVA) network protocols.

By default, an EPICS IOC with CA and/or PVA support enabled will listen for clients on '''all available network interfaces.'''

== Connection Flow ==

* A client emits a request for particular PV names, e.g. ''13SIM1:cam1:Image'' to a specific "search list" of IP addresses (which may be unicast or broadcast addresses).
* Any IOC which receives the request and provides any PV name requested will respond to the client.
* The client and Server now know each other's IP addresses.
* A one-to-one communications channel between client and server is established.
* The opened Channel is used for either a one-off READ or WRITE, or a SUBSCRIPTION (monitor) to receive an update every time the PV value changes.

This flow is more or less the same for the CA or PVA protocols, and involves a mix of TCP/IP and UDP/IP transport with varying port numbers. Many client libraries (e.g. pyepics) will cache connections, or convert repeated READs into SUBSCRIPTIONS behind-the-scenes to save resources.

This system ensures that a newly-started IOC is immediately accessible to all clients without needing to register with a central server, and the only information you need to connect to a PV is its full name.

== The search request ==

EPICS clients follow a simple two-step process to construct the destination of a PV search request. This is performed separately for CA and PVA protocols (substitute ''either'' <code>CA</code> or <code>PVA</code> into the names of the environment variables described below).

# Start with an empty list.
# If the environment variable <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=yes</code>, then add all broadcast IP addresses for all available network interfaces to the list.
#* "yes" is the default value if unset.
#* To skip this step, set <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code>.
# Add the contents of the environment variable <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the list.
#* This should be a space-separated list of IP addresses or hostnames.
# Send the PV search request to every IP or host in the list.

Environment variables should be set prior to starting the client program to be effective, and may need to be "exported" to persist (depending on your shell). Example in bash:

export EPICS_CA_AUTO_ADDR_LIST=no EPICS_CA_ADDR_LIST=s1pvgate.xray.aps.anl.gov
caget 1BM:ExampleVariable

== Potential issues ==
=== Crossing subnet boundaries ===

Since by default only broadcast addresses are in the search request destination list, and broadcast packets are not routed between subnets, without extra configuration you will only be able to find PVs on the same subnet as your client.

;Solution 1: add the IP address of the IOC server directly to <code>EPICS_{CA,PVA}_ADDR_LIST</code>.
:This works across subnet boundaries as long as the required ports are available (no firewalls), but doesn't scale well if there are many IOCs you want to connect to.

;Solution 2: set up a PV Gateway server in the IOC subnet, and add the PV gateway IP address
:A [https://github.com/epics-extensions/ca-gateway PV gateway] is a dedicated server program which relays incoming requests on to the rest of the subnet. It can also apply security rules to the traffic it is forwarding, e.g. it can force a PV to be read-only. The PV gateway hostname or IP address must be added to the relevant <code>ADDR_LIST</code> for all clients.

; Solution 3: set up a PV nameserver.
:Similar to DNS, the [https://github.com/epics-extensions/ca-nameserver nameserver] is a server program that takes over responding to all client name search requests. In this setup, the nameserver must be added to the <code>ADDR_LIST</code> for all clients.

=== Multiple paths between client and server ===

An IOC will reply to '''all''' requests for PVs it servers, even multiple ones from the same client.

Therefore if there are multiple network paths from client to server, for example because the client and/or server have multiple network interfaces, the server may see multiple requests from the client and the client will get multiple responses, each with a different IP.

The same situation can occur if multiple IOCs are serving a PV with the exact same name. The client will usually pick the first response it receives and print a warning to the user, for example:

12-12-2023 01:36:01 Channel Access Exception Channel: "bdpSim0:cam1:Model_RBV", Connecting to: beryl-gpm.xray.aps.anl.gov:5064, Ignored: 10.0.0.1:5064 on ? (op=5 data_type=DBR_invalid count=0)

Which server IP is selected is essentially random each time a search request completes and a connection is created, and there is no guarantee it is the optimal or fastest path from client to server.

;Solution: limit EPICS traffic to only one interface and one network path.
:On the client, set <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code> and manually set <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the single broadcast address on the network interface you want to use.

It is also possible to set <code>EPICS_{CA,PVA}S_INTF_ADDR_LIST</code> on the server to limit the addresses the IOC listens on, but this must be done ''before'' starting the IOC.

The most common variant of this situation is when the client and server are on the same computer. In this case, all the available network interfaces (including the loopback interface ''lo'') present a valid path from client to server.

To limit all traffic to only travel over the loopback interface (which will be the fastest), before running your client set:

EPICS_CA_ADDR_LIST=127.255.255.255
EPICS_PVA_ADDR_LIST=127.255.255.255
EPICS_CA_AUTO_ADDR_LIST=no
EPICS_PVA_AUTO_ADDR_LIST=no

The difference between using only the loopback address (127.0.0.1) and using the loopback broadcast address (127.255.255.255) is that the broadcast address supports communicating with multiple IOCs on the same host on Linux. See the [https://docs.epics-controls.org/en/latest/sys-admin/channel-access-reach-multiple-soft-iocs-linux.html EPICS documentation] for more details and alternative solutions.

Note: Limiting traffic to 127.0.0.1/127.255.255.255 may not work for the EPICSv7 built-in pvaServer due to an issue in the implementation, in which case try a different local IP. See TechTalk: [https://epics.anl.gov/tech-talk/2021/msg00806.php] [https://epics.anl.gov/tech-talk/2021/msg00806.php]

== References ==

* [https://epics.anl.gov/base/R3-14/12-docs/CAref.html EPICS Channel Access Reference Manual (v3.14)]
* [https://mdavidsaver.github.io/pvxs/netconfig.html PVXS (PVA) Network Configuration Information]

EPICS networking

2024-01-09T23:38:42Z

Tguruswamy: /* Potential issues */ add additional techtalk link

= EPICS networking =

For basic terminology, see [[EPICS 101]].

An EPICS IOC is an application which serves EPICS clients (command-line programs like caget; applications using p4p, pyepics, pvapy, or other libraries; GUI display managers like MEDM or caQtDM) process variables over the network. They do so using either the Channel Access (CA) or PVAccess (PVA) network protocols.

By default, an EPICS IOC with CA and/or PVA support enabled will listen for clients on '''all available network interfaces.'''

== Connection Flow ==

* A client emits a request for particular PV names, e.g. ''13SIM1:cam1:Image''.
* Any IOC which receives the request and provides any PV name requested will respond to the client.
* The client and Server now know each other's IP addresses.
* A one-to-one communications channel between client and server is established.
* The opened Channel is used for either a one-off READ or WRITE, or a SUBSCRIPTION (monitor) to receive an update every time the PV value changes.

This flow is more or less the same for the CA or PVA protocols, and involves a mix of TCP/IP and UDP/IP transport with varying port numbers. Many client libraries (e.g. pyepics) will cache connections, or convert repeated READs into SUBSCRIPTIONS behind-the-scenes to save resources.

This system ensures that a newly-started IOC is immediately accessible to all clients without needing to register with a central server, and the only information you need to connect to a PV is its full name.

== The search request ==

EPICS clients follow a simple two-step process to construct the destination of a PV search request. This is performed separately for CA and PVA protocols (substitute ''either'' <code>CA</code> or <code>PVA</code> into the names of the environment variables described below).

# Start with an empty list.
# If the environment variable <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=yes</code>, then add all broadcast IP addresses for all available network interfaces to the list.
#* "yes" is the default value if unset.
#* To skip this step, set <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code>.
# Add the contents of the environment variable <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the list.
#* This should be a space-separated list of IP addresses or hostnames.
# Send the PV search request to every IP or host in the list.

Environment variables should be set prior to starting the client program to be effective, and may need to be "exported" to persist (depending on your shell). Example in bash:

export EPICS_CA_AUTO_ADDR_LIST=no EPICS_CA_ADDR_LIST=s1pvgate.xray.aps.anl.gov
caget 1BM:ExampleVariable

== Potential issues ==
=== Crossing subnet boundaries ===

Since by default only broadcast addresses are in the search request destination list, and broadcast packets are not routed between subnets, without extra configuration you will only be able to find PVs on the same subnet as your client.

;Solution 1: add the IP address of the IOC server directly to <code>EPICS_{CA,PVA}_ADDR_LIST</code>.
:This works across subnet boundaries as long as the required ports are available (no firewalls), but doesn't scale well if there are many IOCs you want to connect to.

;Solution 2: set up a PV Gateway server in the IOC subnet, and add the PV gateway IP address
:A [https://github.com/epics-extensions/ca-gateway PV gateway] is a dedicated server program which relays incoming requests on to the rest of the subnet. It can also apply security rules to the traffic it is forwarding, e.g. it can force a PV to be read-only. The PV gateway hostname or IP address must be added to the relevant <code>ADDR_LIST</code> for all clients.

; Solution 3: set up a PV nameserver.
:Similar to DNS, the [https://github.com/epics-extensions/ca-nameserver nameserver] is a server program that takes over responding to all client name search requests. In this setup, the nameserver must be added to the <code>ADDR_LIST</code> for all clients.

=== Multiple paths between client and server ===

An IOC will reply to '''all''' requests for PVs it servers, even multiple ones from the same client.

Therefore if there are multiple network paths from client to server, for example because the client and/or server have multiple network interfaces, the server may see multiple requests from the client and the client will get multiple responses, each with a different IP.

The same situation can occur if multiple IOCs are serving a PV with the exact same name. The client will usually pick the first response it receives and print a warning to the user, for example:

12-12-2023 01:36:01 Channel Access Exception Channel: "bdpSim0:cam1:Model_RBV", Connecting to: beryl-gpm.xray.aps.anl.gov:5064, Ignored: 10.0.0.1:5064 on ? (op=5 data_type=DBR_invalid count=0)

Which server IP is selected is essentially random each time a search request completes and a connection is created, and there is no guarantee it is the optimal or fastest path from client to server.

;Solution: limit EPICS traffic to only one interface and one network path.
:On the client, set <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code> and manually set <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the single broadcast address on the network interface you want to use.

It is also possible to set <code>EPICS_{CA,PVA}S_INTF_ADDR_LIST</code> on the server to limit the addresses the IOC listens on, but this must be done ''before'' starting the IOC.

The most common variant of this situation is when the client and server are on the same computer. In this case, all the available network interfaces (including the loopback interface ''lo'') present a valid path from client to server.

To limit all traffic to only travel over the loopback interface (which will be the fastest), before running your client set:

EPICS_CA_ADDR_LIST=127.255.255.255
EPICS_PVA_ADDR_LIST=127.255.255.255
EPICS_CA_AUTO_ADDR_LIST=no
EPICS_PVA_AUTO_ADDR_LIST=no

The difference between using only the loopback address (127.0.0.1) and using the loopback broadcast address (127.255.255.255) is that the broadcast address supports communicating with multiple IOCs on the same host on Linux. See the [https://docs.epics-controls.org/en/latest/sys-admin/channel-access-reach-multiple-soft-iocs-linux.html EPICS documentation] for more details and alternative solutions.

Note: Limiting traffic to 127.0.0.1/127.255.255.255 may not work for the EPICSv7 built-in pvaServer due to an issue in the implementation, in which case try a different local IP. See TechTalk: [https://epics.anl.gov/tech-talk/2021/msg00806.php] [https://epics.anl.gov/tech-talk/2021/msg00806.php]

== References ==

* [https://epics.anl.gov/base/R3-14/12-docs/CAref.html EPICS Channel Access Reference Manual (v3.14)]
* [https://mdavidsaver.github.io/pvxs/netconfig.html PVXS (PVA) Network Configuration Information]

EPICS networking

2024-01-09T23:37:03Z

Tguruswamy: /* Potential issues */ issue with pvaServer and localhost

= EPICS networking =

For basic terminology, see [[EPICS 101]].

An EPICS IOC is an application which serves EPICS clients (command-line programs like caget; applications using p4p, pyepics, pvapy, or other libraries; GUI display managers like MEDM or caQtDM) process variables over the network. They do so using either the Channel Access (CA) or PVAccess (PVA) network protocols.

By default, an EPICS IOC with CA and/or PVA support enabled will listen for clients on '''all available network interfaces.'''

== Connection Flow ==

* A client emits a request for particular PV names, e.g. ''13SIM1:cam1:Image''.
* Any IOC which receives the request and provides any PV name requested will respond to the client.
* The client and Server now know each other's IP addresses.
* A one-to-one communications channel between client and server is established.
* The opened Channel is used for either a one-off READ or WRITE, or a SUBSCRIPTION (monitor) to receive an update every time the PV value changes.

This flow is more or less the same for the CA or PVA protocols, and involves a mix of TCP/IP and UDP/IP transport with varying port numbers. Many client libraries (e.g. pyepics) will cache connections, or convert repeated READs into SUBSCRIPTIONS behind-the-scenes to save resources.

This system ensures that a newly-started IOC is immediately accessible to all clients without needing to register with a central server, and the only information you need to connect to a PV is its full name.

== The search request ==

EPICS clients follow a simple two-step process to construct the destination of a PV search request. This is performed separately for CA and PVA protocols (substitute ''either'' <code>CA</code> or <code>PVA</code> into the names of the environment variables described below).

# Start with an empty list.
# If the environment variable <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=yes</code>, then add all broadcast IP addresses for all available network interfaces to the list.
#* "yes" is the default value if unset.
#* To skip this step, set <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code>.
# Add the contents of the environment variable <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the list.
#* This should be a space-separated list of IP addresses or hostnames.
# Send the PV search request to every IP or host in the list.

Environment variables should be set prior to starting the client program to be effective, and may need to be "exported" to persist (depending on your shell). Example in bash:

export EPICS_CA_AUTO_ADDR_LIST=no EPICS_CA_ADDR_LIST=s1pvgate.xray.aps.anl.gov
caget 1BM:ExampleVariable

== Potential issues ==
=== Crossing subnet boundaries ===

Since by default only broadcast addresses are in the search request destination list, and broadcast packets are not routed between subnets, without extra configuration you will only be able to find PVs on the same subnet as your client.

;Solution 1: add the IP address of the IOC server directly to <code>EPICS_{CA,PVA}_ADDR_LIST</code>.
:This works across subnet boundaries as long as the required ports are available (no firewalls), but doesn't scale well if there are many IOCs you want to connect to.

;Solution 2: set up a PV Gateway server in the IOC subnet, and add the PV gateway IP address
:A [https://github.com/epics-extensions/ca-gateway PV gateway] is a dedicated server program which relays incoming requests on to the rest of the subnet. It can also apply security rules to the traffic it is forwarding, e.g. it can force a PV to be read-only. The PV gateway hostname or IP address must be added to the relevant <code>ADDR_LIST</code> for all clients.

; Solution 3: set up a PV nameserver.
:Similar to DNS, the [https://github.com/epics-extensions/ca-nameserver nameserver] is a server program that takes over responding to all client name search requests. In this setup, the nameserver must be added to the <code>ADDR_LIST</code> for all clients.

=== Multiple paths between client and server ===

An IOC will reply to '''all''' requests for PVs it servers, even multiple ones from the same client.

Therefore if there are multiple network paths from client to server, for example because the client and/or server have multiple network interfaces, the server may see multiple requests from the client and the client will get multiple responses, each with a different IP.

The same situation can occur if multiple IOCs are serving a PV with the exact same name. The client will usually pick the first response it receives and print a warning to the user, for example:

12-12-2023 01:36:01 Channel Access Exception Channel: "bdpSim0:cam1:Model_RBV", Connecting to: beryl-gpm.xray.aps.anl.gov:5064, Ignored: 10.0.0.1:5064 on ? (op=5 data_type=DBR_invalid count=0)

Which server IP is selected is essentially random each time a search request completes and a connection is created, and there is no guarantee it is the optimal or fastest path from client to server.

;Solution: limit EPICS traffic to only one interface and one network path.
:On the client, set <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code> and manually set <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the single broadcast address on the network interface you want to use.

It is also possible to set <code>EPICS_{CA,PVA}S_INTF_ADDR_LIST</code> on the server to limit the addresses the IOC listens on, but this must be done ''before'' starting the IOC.

The most common variant of this situation is when the client and server are on the same computer. In this case, all the available network interfaces (including the loopback interface ''lo'') present a valid path from client to server.

To limit all traffic to only travel over the loopback interface (which will be the fastest), before running your client set:

EPICS_CA_ADDR_LIST=127.255.255.255
EPICS_PVA_ADDR_LIST=127.255.255.255
EPICS_CA_AUTO_ADDR_LIST=no
EPICS_PVA_AUTO_ADDR_LIST=no

The difference between using only the loopback address (127.0.0.1) and using the loopback broadcast address (127.255.255.255) is that the broadcast address supports communicating with multiple IOCs on the same host on Linux. See the [https://docs.epics-controls.org/en/latest/sys-admin/channel-access-reach-multiple-soft-iocs-linux.html EPICS documentation] for more details and alternative solutions.

Note: Limiting traffic to 127.0.0.1/127.255.255.255 may not work for the EPICSv7 built-in pvaServer due to an issue in the implementation, in which case try a different local IP. https://epics.anl.gov/tech-talk/2021/msg00806.php

== References ==

* [https://epics.anl.gov/base/R3-14/12-docs/CAref.html EPICS Channel Access Reference Manual (v3.14)]
* [https://mdavidsaver.github.io/pvxs/netconfig.html PVXS (PVA) Network Configuration Information]

Beamline Controls

2023-12-23T02:53:24Z

Tguruswamy: Fix databases -> database

WIKI for the APS Beamline Controls Community, maintained by the Beamline Controls Technical Working Group. 
This wiki exists to collect the output of the working group, including documentation on beamline controls software and hardware useful for APS beamline scientists and resident users.

----

===Beamline Controls Documentation===

* [[EPICS Cheat Sheet]]: Channel Access command-lines, noteworthy record fields & IOC shell commands.
* [[EPICS 101]]: definition of basic EPICS components ([[EPICS_101#What_is_an_IOC.3F|IOC]], [[EPICS 101#What_is_a_PV.3F|PV]], [[EPICS_101#What_is_a_record.3F|record]], [[EPICS 101#What_is_a_database.3F|database]]).
* [[IOC 101]]: learn how to create your own IOC.
* [[EPICS networking]]: basic overview of how to control the network paths used by EPICS IOCs and clients.
* [[Git_101|Git 101]]
* [[Controls Software Documentation]]
* [[Hardware and IT Configuration]]

===Practical Beamline Controls Training (2023)===

[[Practical_Beamline_Controls_Training#IOC_Basics|Session 1: IOC Basics]]

[[Practical_Beamline_Controls_Training#IOC_Deployment_.26_Troubleshooting|Session 2: IOC Deployment & Troubleshooting]]

===Beamline Controls Technical Working Group information===

The BC Technical Working (BCTWG) group meets monthly to review and gather feedback on possible changes in either the way BC delivers support or the technologies that BC supports. The BCTWG comprises the BC group plus interested stakeholders from XSD. In a nutshell, the main goal of the BCTWG is to improve communication between BC and its customers on topics of mutual interest.

* Box folder with slides and meeting notes (ANL only): [https://anl.box.com/s/chcn7xauil93inay8ipinbxtcckc9k9f]

* Current chairs:
** Kevin Peterson (XSD-BC)
** Tejas Guruswamy (XSD-DET)
** Andrew Chuang (XSD-MPE)

===Beamline Data Pipelines===

* https://git.aps.anl.gov/groups/bdp-public/-/wikis/home

EPICS 101

2023-12-23T02:52:51Z

Tguruswamy: Adjust link text

== What is EPICS? ==

EPICS (Experimental Physics and Industrial Control System) is a control system built around programs serving Process Variables (PVs) over a computer network. It is free, open source, and in active development and used in a number of large industrial and research facilities around the world.
EPICS is a '''distributed''' control system, which means that the IOC (software) can run on a different computer as the user interfaces.

This page is dedicated to the definition of basic EPICS components ([[EPICS_101#What_is_an_IOC.3F|IOC]], [[EPICS_101#What_is_a_PV.3F|PV]], [[EPICS_101#What_is_a_record.3F|record]], [[EPICS_101#What_is_a_database.3F|database]]). To build your own EPICS support, you can skip directly to [[IOC_101|How does one create an IOC]].

== What is an IOC? ==

An EPICS '''IOC''' (Input/Output Controller) is a program which processes and serves a '''database'''. A database consists of many '''records''' each with their own '''fields''' which may contain data or change the record behavior. Each of these elements (fields) is known as a '''process variable (PV)'''. Databases may be written from scratch for the IOC or composed from pre-written EPICS '''module''' databases and record types.

Any client, locally or on the network, can read and/or write a process variable using either '''channel access''' (CA) or '''PVaccess''' (PVA) network protocols. Clients might be another IOC, a GUI display manager (medm/caqtdm/CSS), or any other program with support for the EPICS protocols. Only the name of the process variable (in the form <code>'''prefix:record_name.field_name'''</code>) is required to read or write a PV.

Note:
* IOCs used to run on VMEs, but nowadays it runs on anything.
* IOCs usually talks to hardware, but that isn’t a requirement.

You interact directly with a running IOC via the EPICS shell ('''iocsh'''), or indirectly by reading and writing Process Variables. Starting an IOC on a general purpose computer (a '''softIOC''', as opposed to running on dedicated hardware like VME) involves running a binary along with a startup script of iocsh commands (usually called '''st.cmd''').
This script tells the IOC what database files to load and where those files are. When the IOC is told to load a database, it basically parses that database file, identifying the records that are contained within, and then generating the actual epics PVs that a user will interact with.

'''Records''' and '''databases''' (lists of records) are configuration files describing the initial setup for '''PVs'''; the '''IOC''' is the software that manages the behavior of PVs and communication to get/set their values.

== What is a PV? ==

PV (Process Variable) is a the field from a record:
'''PV = record_name + . + field_name'''

Most of the interactive parts of user displays have PVs associated with them.

== What is a record? ==

Records are the building blocks of EPICS:
* There is a lot of different types of record, and each type is designed to perform a specific function
* Records are customizable via fields that allow users to configure how they behave and monitor their state (see [[EPICS_Cheat_Sheet#Noteworthy_fields|Noteworthy Fields]] & [[#How to find more info about record fields]] to know how to learn about record fields)
* Records do nothing until they are processed (see [[#SCAN, PROC & FLNK fields]])
* Records vary significantly in complexity (see [[#More about records]])
* It is rare for new records to be created, now that [https://github.com/epics-modules/asyn''asyn''] device support exists).

=== Where do records come from? ===

Typically, records at the APS comes from either:
* '''[https://epics.anl.gov/base/index.php EPICS base]''': main core of EPICS
* '''[https://www.aps.anl.gov/BCDA/synApps synApps]''': a collection of EPICS modules that are commonly used at the APS
A module is similar to a python package: it is an already written set of databases and associated behavior you can load into your IOC to do certain things, for example talk to specific hardware. An IOC can link to several modules.

{| class="wikitable" style="margin-left: 100px; margin-right:auto"
|+ Records in synApps modules 

|-
! Module !! Records !! Module !! Records
|-
| alive || alive || motor || motor
|-
| asyn || asyn || optics || table
|-
| busy || busy || scaler || scaler
|-
| calc || acalcout, scalcout, sseq, swait, transform || sscan || sscan
|-
| camac || camac || std || epid, timestamp, throttle
|-
| lua || luascript || vac || digitel, vs
|-
| mca || mca || vme || vme
|-
| allenBradley || ab1771IFE, ab1771N, ab1771IX, ...|| allenBradley || ..., ab1791, abDcm
|}

If a record isn't listed here, it probably comes from EPICS base.
The advantage to using synApps over a collection of modules one assembles on their own is that some amount of BCDA testing has been done.

=== What is a record field? ===

Record fields are controllable or informative properties of a record. For a list of noteworthy fields, see the [[EPICS_Cheat_Sheet#Noteworthy_fields|EPICS cheat sheet]].

==== SCAN, PROC & FLNK fields ====

Records do nothing until they are processed. Records can be processed multiple ways:
* Manually:
** Write 1 (or any non-zero value) to a record’s '''PROC''' field
**Forward link ('''FLNK''') from another record
* Periodically by setting the '''SCAN''' field to something other than <code>Passive</code>

{| class="wikitable" style="margin-left: 100px; margin-right:auto"
|+ SCAN field options 

|-
! Index !! Option !! Index !! Option
|-
| 0 || <code>Passive</code> || 5 || <code>2 second</code>
|-
| 1 || <code>Event</code> || 6 || <code>1 second</code>
|-
| 2 || <code>I/O Intr</code> ||7 || <code>0.5 second</code>
|-
| 3 || <code>10 second</code> ||8 || <code>0.2 second</code>
|-
| 4 || <code>5 second</code> ||9 || <code>0.1 second</code>
|}

Note:
* If a record’s '''SCAN''' field is not specified in a database, it defaults to <code>Passive</code>.
* The '''SCAN''' field is common to ''all'' record types and the options can be selected by index or by string.
* <code>I/O Intr</code> processes a record when the driver gets a new value, but is device/driver dependent and isn’t guaranteed to work

=== Record link types ===

Much of the power of EPICS comes from the ability to link EPICS records together.
There are two link implementations: standard and dynamic.
* Dynamic links (pink in standard BCDA screens) are found on ''userCalc'' ('''swait''' records) and ''scan'' ('''sscan''' records) screens; this type of link accept only a PV name. The role of the process command (see below) is taken in a separate field.
* Standard links (purple in standard BCDA screens) are found everywhere else.
''inlinkHelp'' and ''outlinkHelp'' are accessible from userTransform screens and scaler screens (for the later only).

==== Input Links ====

Input links read a value from another PV. It accepts a PV name followed by a process command (e.g. <code>NPP</code>) and an alarm command (e.g. <code>NMS</code>, ignore this).
Here are the possible process commands for an input link:
* <code>NPP</code> - Just read the value; don't process the target record (i.e. read a potentially stale value); this is a default command.
* <code>PP</code> - Process the target record and then read a value from it.
* <code>CA</code> - Read a value without causing processing to occur.
* <code>CP</code> - Process linking record every time the target record processes.
* <code>CPP</code> - Same as CP, but linking record process only if its SCAN field has the value <code>Passive</code>.
Note: ''inlinkHelp'' is accessible from userTransform screens.

==== Output Links ====

Output links write a value to another PV. It accepts a PV name followed by a process command (e.g. <code>NPP</code>) and an alarm command (e.g. <code>NMS</code>, ignore this).
Here are the possible process commands for an output link:
* <code>NPP</code> - Just write a value; don't process the target record; this is a default command.
* <code>PP</code> - Write a value and then process the target record. To be used if you want EPICS to wait for completion. When the target record get processed, EPICS will trace it.
* <code>CA</code> - Write a value; let the target record decide whether or not to process (as it would if you typed the value by hand); if the processing does occur, EPICS will not trace it.

Note:
* The default link process command, <code>NPP</code>, is '''almost never the desired behavior''' for output links. Change <code>NPP</code> to <code>PP</code> immediately and revert if problems are discovered.
* If the target PV is itself a link field, you must specify the command <code>CA</code>; if you want to '''wait for completion''' in this case, you must use either the '''sseq''' or the '''sCalcout''' record, select <code>CA</code> and set the link's '''WAIT''' field to <code>Wait</code>
* ''outlinkHelp'' is accessible from userTransform and scaler screens.

==== Forward Links ====

Forward links look like output links; they are purple on user displays, but they don't auto-populate with process & alarm commands.
Forward links cause the linked record to process when the record containing the forward link is processed:
* Linked record is in the same IOC ('''PROC''' field for linked record is optional):
** <code>record_name.FLNK = linked_record_name</code>
** <code>record_name.FLNK = linked_record_name.PROC</code>
* Linked record is in a different IOC ('''PROC''' field for linked record is required):
** <code>record_name.FLNK = linked_record_name.PROC</code>

Note:
* Forwarding-linking records in the same IOC by only specifying the record name is an exception to the "'''VAL''' ''is the default field''" rule.
* When omitting the '''PROC''' field of a linked record in a different IOC, nothing will happen but you won't get any error message.
* Always specify the '''PROC''' field to avoid problems.

=== More about records ===

* Records vary significantly in complexity; one way to classify them would be (*):
** Simple records are associated with individual data types
*::;bi, bo, mbbi, mbbo, longin, longout, ai, ao, stringin, stringout, waveform
** Intermediate records often perform calculations or link multiple records
*::;calcout, scalcout, acalcout, swait, transform, fanout, dfanout
** Complex records are associated with devices or higher-level functionality
*::;motor, scaler, mca, sscan, table, digitel, vs
** Flexible records allow custom functionality to be implemented
*::;sub, aSub, luascript
(*) this is not an official classification, nor an exhaustive list of records. 
Note:
* areaDetector databases are composed of mostly simple record.

=== How to find more info about record fields ===

To find documentation for a specific field you need to know/find answers to the following questions:
* Is the field common to all record or only input/output record types?
* If not, does the record come from EPICS base or a different EPICS module (synApps)?
** The [https://epics.anl.gov/base/R7-0/7-docs/RecordReference.html EPICS base record reference documentation] is the best place to find information about record types that come from EPICS base. Field documentation appears in multiple locations:
*** Fields common to all record types
*** Fields common to input record types
*** Fields common to output record types
*** Record-specific documentation
** synApps record reference documentation
*** In each module's documentation directory
**** https://github.com/epics-modules
**** <code>/APSshare/epics/synApps_X_Y/support</code>

== What is a database? ==

A database is just a text file with a '''.db''' (or '''.template''') extension containing a collection of records that are related.

Databases are generalized with macros to allow multiple instances to be loaded (e.g. $(P)):
* Macros are variables containing strings to be substituted when the database is loaded.
* Macros without default values are required to load the database; there is no easy way to figure out what those are for a given database other than looking at the file itself.
* Macros required by the database should match the macros required by associated screens & autosave (.req) files.
* The P (prefix) macro usually includes a colon (:), which is the standard separator used in PV names.

=== How are databases loaded? ===

There are 2 ways to load databases which are strictly equivalent.

==== dbLoadRecords ====

<code>dbLoadRecords("$(IP)/db/ADAM_4018.db","P=kmp:,R=adam1,PORT=serial1,A=01")</code>
<code>dbLoadRecords("$(IP)/db/ADAM_4018.db","P=kmp:,R=adam2,PORT=serial1,A=02")</code>

==== dbLoadTemplate ====

<code>dbLoadTemplate("substitutions/ADAM_4018.substitutions","P=kmp:")</code>

where ADAM_4018.substitutions:

<code>file "$(IP)/db/ADAM_4018.db"
{
pattern
{R,PORT,A}
{adam1, serial1,01}
{adam2, serial1,02}
}</code>

== What's next? ==

To learn how to create your own IOC, go to [[IOC 101]].

== External links ==

* [https://epics.anl.gov/base/R3-16/2-docs/AppDevGuide/DatabaseDefinition.html#x7-2630006.6 https://epics.anl.gov/]
* [https://epics.anl.gov/base/R7-0/7-docs/RecordReference.html EPICS base record reference documentation]

Beamline Controls

2023-12-23T02:45:20Z

Tguruswamy: /* Beamline Controls Documentation */ add epics networking page

WIKI for the APS Beamline Controls Community, maintained by the Beamline Controls Technical Working Group. 
This wiki exists to collect the output of the working group, including documentation on beamline controls software and hardware useful for APS beamline scientists and resident users.

----

===Beamline Controls Documentation===

* [[EPICS Cheat Sheet]]: Channel Access command-lines, noteworthy record fields & IOC shell commands.
* [[EPICS 101]]: definition of basic EPICS components ([[EPICS_101#What_is_an_IOC.3F|IOC]], [[EPICS 101#What_is_a_PV.3F|PV]], [[EPICS_101#What_is_a_record.3F|record]], [[EPICS 101#What_is_a_databases.3F|database]]).
* [[IOC 101]]: learn how to create your own IOC.
* [[EPICS networking]]: basic overview of how to control the network paths used by EPICS IOCs and clients.
* [[Git_101|Git 101]]
* [[Controls Software Documentation]]
* [[Hardware and IT Configuration]]

===Practical Beamline Controls Training (2023)===

[[Practical_Beamline_Controls_Training#IOC_Basics|Session 1: IOC Basics]]

[[Practical_Beamline_Controls_Training#IOC_Deployment_.26_Troubleshooting|Session 2: IOC Deployment & Troubleshooting]]

===Beamline Controls Technical Working Group information===

The BC Technical Working (BCTWG) group meets monthly to review and gather feedback on possible changes in either the way BC delivers support or the technologies that BC supports. The BCTWG comprises the BC group plus interested stakeholders from XSD. In a nutshell, the main goal of the BCTWG is to improve communication between BC and its customers on topics of mutual interest.

* Box folder with slides and meeting notes (ANL only): [https://anl.box.com/s/chcn7xauil93inay8ipinbxtcckc9k9f]

* Current chairs:
** Kevin Peterson (XSD-BC)
** Tejas Guruswamy (XSD-DET)
** Andrew Chuang (XSD-MPE)

===Beamline Data Pipelines===

* https://git.aps.anl.gov/groups/bdp-public/-/wikis/home

EPICS networking

2023-12-23T02:40:52Z

Tguruswamy: Writeup of BDP presentation

= EPICS networking =

For basic terminology, see [[EPICS 101]].

An EPICS IOC is an application which serves EPICS clients (command-line programs like caget; applications using p4p, pyepics, pvapy, or other libraries; GUI display managers like MEDM or caQtDM) process variables over the network. They do so using either the Channel Access (CA) or PVAccess (PVA) network protocols.

By default, an EPICS IOC with CA and/or PVA support enabled will listen for clients on '''all available network interfaces.'''

== Connection Flow ==

* A client emits a request for particular PV names, e.g. ''13SIM1:cam1:Image''.
* Any IOC which receives the request and provides any PV name requested will respond to the client.
* The client and Server now know each other's IP addresses.
* A one-to-one communications channel between client and server is established.
* The opened Channel is used for either a one-off READ or WRITE, or a SUBSCRIPTION (monitor) to receive an update every time the PV value changes.

This flow is more or less the same for the CA or PVA protocols, and involves a mix of TCP/IP and UDP/IP transport with varying port numbers. Many client libraries (e.g. pyepics) will cache connections, or convert repeated READs into SUBSCRIPTIONS behind-the-scenes to save resources.

This system ensures that a newly-started IOC is immediately accessible to all clients without needing to register with a central server, and the only information you need to connect to a PV is its full name.

== The search request ==

EPICS clients follow a simple two-step process to construct the destination of a PV search request. This is performed separately for CA and PVA protocols (substitute ''either'' <code>CA</code> or <code>PVA</code> into the names of the environment variables described below).

# Start with an empty list.
# If the environment variable <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=yes</code>, then add all broadcast IP addresses for all available network interfaces to the list.
#* "yes" is the default value if unset.
#* To skip this step, set <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code>.
# Add the contents of the environment variable <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the list.
#* This should be a space-separated list of IP addresses or hostnames.
# Send the PV search request to every IP or host in the list.

Environment variables should be set prior to starting the client program to be effective, and may need to be "exported" to persist (depending on your shell). Example in bash:

export EPICS_CA_AUTO_ADDR_LIST=no EPICS_CA_ADDR_LIST=s1pvgate.xray.aps.anl.gov
caget 1BM:ExampleVariable

== Potential issues ==
=== Crossing subnet boundaries ===

Since by default only broadcast addresses are in the search request destination list, and broadcast packets are not routed between subnets, without extra configuration you will only be able to find PVs on the same subnet as your client.

;Solution 1: add the IP address of the IOC server directly to <code>EPICS_{CA,PVA}_ADDR_LIST</code>.
:This works across subnet boundaries as long as the required ports are available (no firewalls), but doesn't scale well if there are many IOCs you want to connect to.

;Solution 2: set up a PV Gateway server in the IOC subnet, and add the PV gateway IP address
:A [https://github.com/epics-extensions/ca-gateway PV gateway] is a dedicated server program which relays incoming requests on to the rest of the subnet. It can also apply security rules to the traffic it is forwarding, e.g. it can force a PV to be read-only. The PV gateway hostname or IP address must be added to the relevant <code>ADDR_LIST</code> for all clients.

; Solution 3: set up a PV nameserver.
:Similar to DNS, the [https://github.com/epics-extensions/ca-nameserver nameserver] is a server program that takes over responding to all client name search requests. In this setup, the nameserver must be added to the <code>ADDR_LIST</code> for all clients.

=== Multiple paths between client and server ===

An IOC will reply to '''all''' requests for PVs it servers, even multiple ones from the same client.

Therefore if there are multiple network paths from client to server, for example because the client and/or server have multiple network interfaces, the server may see multiple requests from the client and the client will get multiple responses, each with a different IP.

The same situation can occur if multiple IOCs are serving a PV with the exact same name. The client will usually pick the first response it receives and print a warning to the user, for example:

12-12-2023 01:36:01 Channel Access Exception Channel: "bdpSim0:cam1:Model_RBV", Connecting to: beryl-gpm.xray.aps.anl.gov:5064, Ignored: 10.0.0.1:5064 on ? (op=5 data_type=DBR_invalid count=0)

Which server IP is selected is essentially random each time a search request completes and a connection is created, and there is no guarantee it is the optimal or fastest path from client to server.

;Solution: limit EPICS traffic to only one interface and one network path.
:On the client, set <code>EPICS_{CA,PVA}_AUTO_ADDR_LIST=no</code> and manually set <code>EPICS_{CA,PVA}_ADDR_LIST</code> to the single broadcast address on the network interface you want to use.

It is also possible to set <code>EPICS_{CA,PVA}S_INTF_ADDR_LIST</code> on the server to limit the addresses the IOC listens on, but this must be done ''before'' starting the IOC.

The most common variant of this situation is when the client and server are on the same computer. In this case, all the available network interfaces (including the loopback interface ''lo'') present a valid path from client to server.

To limit all traffic to only travel over the loopback interface (which will be the fastest), before running your client set:

EPICS_CA_ADDR_LIST=127.255.255.255
EPICS_PVA_ADDR_LIST=127.255.255.255
EPICS_CA_AUTO_ADDR_LIST=no
EPICS_PVA_AUTO_ADDR_LIST=no

The difference between using only the loopback address (127.0.0.1) and using the loopback broadcast address (127.255.255.255) is that the broadcast address supports communicating with multiple IOCs on the same host on Linux. See the [https://docs.epics-controls.org/en/latest/sys-admin/channel-access-reach-multiple-soft-iocs-linux.html EPICS documentation] for more details and alternative solutions.

== References ==

* [https://epics.anl.gov/base/R3-14/12-docs/CAref.html EPICS Channel Access Reference Manual (v3.14)]
* [https://mdavidsaver.github.io/pvxs/netconfig.html PVXS (PVA) Network Configuration Information]

NVIDIA Data Processing Units

2023-05-17T18:50:31Z

Tguruswamy:

== Introduction ==

NVIDIA DPUs are expansion cards which allow for offloading of certain network-traffic related tasks from the host CPU. They comprise of an ARM CPU, memory, and high-speed ConnectX NIC on the same board.
Network packets to/from the host can be manipulated by the ARM CPU using programs written with the DPDK or DOCA SDKs.
The DPU ARM system runs its own OS; as shipped by NVIDIA, this is currently Ubuntu Linux.

== DPU host software setup ==

[https://docs.nvidia.com/networking/display/BlueField2DPUVPI/BlueField+DPU+Administrator+Quick+Start+Guide Bluefield DPU Administrator Quick Start Guide] (NVIDIA)

On a RHEL8 machine, first install the RPM package which contains the DOCA and DPU-related packages. This includes both a local copy of the necessary RPMs and enables a YUM repo for updates.

<pre>
$ wget https://www.mellanox.com/downloads/DOCA/DOCA_v1.5.1/doca-host-repo-rhel86-1.5.1-0.1.8.1.5.1007.1.el8.5.8.1.1.2.1.x86_64.rpm
# yum install ./doca-host-repo-rhel86-1.5.1-0.1.8.1.5.1007.1.el8.5.8.1.1.2.1.x86_64.rpm
</pre>

We find that the NVIDIA repos tend to timeout when accessed from the APS, so add to the end of <code>/etc/yum.conf</code>:

<pre>minrate=10
timeout=300</pre>

Then install the necessary RPMs, allowing for downgrades and package removals:
<pre># yum makecache
# yum install --allowerasing --nobest doca-runtime doca-tools pv</pre>

<code>rshim</code> is a userspace tool which allows for configuration of NVIDIA Mellanox cards.
Ensure rshim is running with <code>systemctl status rshim</code> (look for "loaded" and "enabled").

<code>mst</code>, or Mellanox Software Tools, is a userspace program which creates a device tree used for configuration.

<pre>
# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success
# mst status -v
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA
BlueField2(rev:1) /dev/mst/mt41686_pciconf0 ca:00.0 mlx5_0 net-ib0 1
</pre>

Get a new Bluefield OS system image:

<pre>$ wget https://content.mellanox.com/BlueField/BFBs/Ubuntu20.04/DOCA_1.5.1_BSP_3.9.3_Ubuntu_20.04-4.2211-LTS.signed.bfb</pre>

and install it:

<pre># bfb-install --bfb DOCA_1.5.1_BSP_3.9.3_Ubuntu_20.04-4.2211-LTS.signed.bfb --rshim rshim0
Collecting BlueField booting status. Press Ctrl+C to stop…
INFO[BL2]: start
INFO[BL2]: DDR POST passed
INFO[BL2]: UEFI loaded
INFO[BL31]: start
INFO[BL31]: runtime
INFO[UEFI]: UPVS valid
INFO[UEFI]: eMMC init
INFO[UEFI]: eMMC probed
INFO[UEFI]: PMI: updates started
INFO[UEFI]: PMI: boot image update
INFO[UEFI]: PMI: updates completed, status 0
INFO[UEFI]: PCIe enum start
INFO[UEFI]: PCIe enum end
INFO[MISC]: Ubuntu installation started
INFO[MISC]: Installing OS image
INFO[MISC]: Installation finished</pre>

Only if DPU remote access (LAN/internet) is required, enable ip routing on host: add to <code>/etc/sysctl.d/50-dpu.conf</code>

<pre>net.ipv4.conf.all.forwarding = 1
net.ipv6.conf.all.forwarding = 1</pre>

and setup IPv4 masquerading via nftables:

<pre># nft add table nat
# nft -- add chain nat prerouting { type nat hook prerouting priority -100 \; }
# nft -- add chain nat postrouting { type nat hook postrouting priority 100 \; }
# nft add rule nat postrouting oifname "ens6f0" snat to $(host IP)
# nft list ruleset > /etc/nftables/dpu_nat.nft
# echo "include "/etc/nftables/dpu_nat.nft" >> /etc/sysconfig/nftables.conf
# systemctl enable nftables.service</pre>

== DPU login and configuration ==

First set up an appropriate IP configuration for the tmfifo_net0 interface on the host. The DPU is factory-configured at 192.168.100.2.

<pre>
# nmcli conn add type tun mode tap con-name tmfifo_net0 ifname tmfifo_net0 autoconnect yes ip4 192.168.100.1/24 ipv4.never-default true
# nmci conn up tmfifo_net0
</pre>

then login via ssh:

<pre>$ ssh ubuntu@192.168.100.2</pre>

You should receive the Ubuntu OS login prompt, and will be prompted to update the user password.

You can now update the DPU firmware:

<pre># sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl</pre>

and configure it:

<pre>
# sudo mst start
# sudo mlxconfig -d /dev/mst/mt41686_pciconf0 -y reset # Reset all settings
# sudo mlxconfig -d /dev/mst/mt41686_pciconf0 s LINK_TYPE_P1=2 # Set port 1 to Ethernet mode (not Infiniband)
</pre>

Reboot the host and ensure settings persist.

== References ==

* [https://docs.nvidia.com/networking/display/BlueFieldDPUOSLatest/Modes+of+Operation BlueField Modes of Operation]
* [https://docs.nvidia.com/networking/display/BlueFieldDPUOSLatest/Functional+Diagram BlueField Functional Diagram]
* [https://docs.nvidia.com/networking/display/BlueFieldDPUOSLatest/Kernel+Representors+Model BlueField Kernel Representors] (names for the passthrough interfaces from the DPU side)

NVIDIA Data Processing Units

2023-05-17T18:49:43Z

Tguruswamy: /* References */

== Introduction ==

NVIDIA DPUs are expansion cards which allow for offloading of certain network-traffic related tasks from the host CPU. They comprise of an ARM CPU, memory, and high-speed ConnectX NIC on the same board.
Network packets to/from the host can be manipulated by the ARM CPU using programs written with the DPDK or DOCA SDKs.
The DPU ARM system runs its own OS; as shipped by NVIDIA, this is currently Ubuntu Linux.

== DPU host software setup ==

[https://docs.nvidia.com/networking/display/BlueField2DPUVPI/BlueField+DPU+Administrator+Quick+Start+Guide Bluefield DPU Administrator Quick Start Guide] (NVIDIA)

On a RHEL8 machine, first install the RPM package which contains the DOCA and DPU-related packages. This includes both a local copy of the necessary RPMs and enables a YUM repo for updates.

<pre>
$ wget https://www.mellanox.com/downloads/DOCA/DOCA_v1.5.1/doca-host-repo-rhel86-1.5.1-0.1.8.1.5.1007.1.el8.5.8.1.1.2.1.x86_64.rpm
# yum install ./doca-host-repo-rhel86-1.5.1-0.1.8.1.5.1007.1.el8.5.8.1.1.2.1.x86_64.rpm
</pre>

We find that the NVIDIA repos tend to timeout when accessed from the APS, so add to the end of <code>/etc/yum.conf</code>:

<pre>minrate=10
timeout=300</pre>

Then install the necessary RPMs, allowing for downgrades and package removals:
<pre># yum makecache
# yum install --allowerasing --nobest doca-runtime doca-tools pv</pre>

<code>rshim</code> is a userspace tool which allows for configuration of NVIDIA Mellanox cards.
Ensure rshim is running with <code>systemctl status rshim</code> (look for "loaded" and "enabled").

<code>mst</code>, or Mellanox Software Tools, is a userspace program which creates a device tree used for configuration.

<pre>
# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success
# mst status -v
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA
BlueField2(rev:1) /dev/mst/mt41686_pciconf0 ca:00.0 mlx5_0 net-ib0 1
</pre>

Get a new Bluefield OS system image:

<pre>$ wget https://content.mellanox.com/BlueField/BFBs/Ubuntu20.04/DOCA_1.5.1_BSP_3.9.3_Ubuntu_20.04-4.2211-LTS.signed.bfb</pre>

and install it:

<pre># bfb-install --bfb DOCA_1.5.1_BSP_3.9.3_Ubuntu_20.04-4.2211-LTS.signed.bfb --rshim rshim0
Collecting BlueField booting status. Press Ctrl+C to stop…
INFO[BL2]: start
INFO[BL2]: DDR POST passed
INFO[BL2]: UEFI loaded
INFO[BL31]: start
INFO[BL31]: runtime
INFO[UEFI]: UPVS valid
INFO[UEFI]: eMMC init
INFO[UEFI]: eMMC probed
INFO[UEFI]: PMI: updates started
INFO[UEFI]: PMI: boot image update
INFO[UEFI]: PMI: updates completed, status 0
INFO[UEFI]: PCIe enum start
INFO[UEFI]: PCIe enum end
INFO[MISC]: Ubuntu installation started
INFO[MISC]: Installing OS image
INFO[MISC]: Installation finished</pre>

Only if DPU remote access (LAN/internet) is required, enable ip routing on host: add to <code>/etc/sysctl.d/50-dpu.conf</code>

<pre>net.ipv4.conf.all.forwarding = 1
net.ipv6.conf.all.forwarding = 1</pre>

and setup IPv4 masquerading via nftables:

<pre># nft add table nat
# nft -- add chain nat prerouting { type nat hook prerouting priority -100 \; }
# nft -- add chain nat postrouting { type nat hook postrouting priority 100 \; }
# nft add rule nat postrouting oifname "ens6f0" snat to $(host IP)
# nft list ruleset > /etc/nftables/dpu_nat.nft
# echo "include "/etc/nftables/dpu_nat.nft" >> /etc/sysconfig/nftables.conf
# systemctl enable nftables.service</pre>

== DPU login and configuration ==

First set up an appropriate IP configuration for the tmfifo_net0 interface on the host. The DPU is factory-configured at 192.168.100.2.

<pre>
# nmcli conn add type tun mode tap con-name tmfifo_net0 ifname tmfifo_net0 autoconnect yes ip4 192.168.100.1/24 ipv4.never-default true
# nmci conn up tmfifo_net0
</pre>

then login via ssh:

<pre>$ ssh ubuntu@192.168.100.2</pre>

You should receive the Ubuntu OS login prompt, and will be prompted to update the user password.

You can now update the DPU firmware:

<pre>sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl</pre>

and configure it:

<pre>
sudo mst start
# Reset all settings
sudo mlxconfig -d /dev/mst/mt41686_pciconf0 -y reset
# Set port 1 to Ethernet mode (not Infiniband)
sudo mlxconfig -d /dev/mst/mt41686_pciconf0 s LINK_TYPE_P1=2
</pre>

Reboot the host and ensure settings persist.

== References ==

* [https://docs.nvidia.com/networking/display/BlueFieldDPUOSLatest/Modes+of+Operation BlueField Modes of Operation]
* [https://docs.nvidia.com/networking/display/BlueFieldDPUOSLatest/Functional+Diagram BlueField Functional Diagram]
* [https://docs.nvidia.com/networking/display/BlueFieldDPUOSLatest/Kernel+Representors+Model BlueField Kernel Representors] (names for the passthrough interfaces from the DPU side)

NVIDIA Data Processing Units

2023-05-17T18:49:27Z

Tguruswamy: DPU setup

== Introduction ==

NVIDIA DPUs are expansion cards which allow for offloading of certain network-traffic related tasks from the host CPU. They comprise of an ARM CPU, memory, and high-speed ConnectX NIC on the same board.
Network packets to/from the host can be manipulated by the ARM CPU using programs written with the DPDK or DOCA SDKs.
The DPU ARM system runs its own OS; as shipped by NVIDIA, this is currently Ubuntu Linux.

== DPU host software setup ==

[https://docs.nvidia.com/networking/display/BlueField2DPUVPI/BlueField+DPU+Administrator+Quick+Start+Guide Bluefield DPU Administrator Quick Start Guide] (NVIDIA)

On a RHEL8 machine, first install the RPM package which contains the DOCA and DPU-related packages. This includes both a local copy of the necessary RPMs and enables a YUM repo for updates.

<pre>
$ wget https://www.mellanox.com/downloads/DOCA/DOCA_v1.5.1/doca-host-repo-rhel86-1.5.1-0.1.8.1.5.1007.1.el8.5.8.1.1.2.1.x86_64.rpm
# yum install ./doca-host-repo-rhel86-1.5.1-0.1.8.1.5.1007.1.el8.5.8.1.1.2.1.x86_64.rpm
</pre>

We find that the NVIDIA repos tend to timeout when accessed from the APS, so add to the end of <code>/etc/yum.conf</code>:

<pre>minrate=10
timeout=300</pre>

Then install the necessary RPMs, allowing for downgrades and package removals:
<pre># yum makecache
# yum install --allowerasing --nobest doca-runtime doca-tools pv</pre>

<code>rshim</code> is a userspace tool which allows for configuration of NVIDIA Mellanox cards.
Ensure rshim is running with <code>systemctl status rshim</code> (look for "loaded" and "enabled").

<code>mst</code>, or Mellanox Software Tools, is a userspace program which creates a device tree used for configuration.

<pre>
# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success
# mst status -v
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA
BlueField2(rev:1) /dev/mst/mt41686_pciconf0 ca:00.0 mlx5_0 net-ib0 1
</pre>

Get a new Bluefield OS system image:

<pre>$ wget https://content.mellanox.com/BlueField/BFBs/Ubuntu20.04/DOCA_1.5.1_BSP_3.9.3_Ubuntu_20.04-4.2211-LTS.signed.bfb</pre>

and install it:

<pre># bfb-install --bfb DOCA_1.5.1_BSP_3.9.3_Ubuntu_20.04-4.2211-LTS.signed.bfb --rshim rshim0
Collecting BlueField booting status. Press Ctrl+C to stop…
INFO[BL2]: start
INFO[BL2]: DDR POST passed
INFO[BL2]: UEFI loaded
INFO[BL31]: start
INFO[BL31]: runtime
INFO[UEFI]: UPVS valid
INFO[UEFI]: eMMC init
INFO[UEFI]: eMMC probed
INFO[UEFI]: PMI: updates started
INFO[UEFI]: PMI: boot image update
INFO[UEFI]: PMI: updates completed, status 0
INFO[UEFI]: PCIe enum start
INFO[UEFI]: PCIe enum end
INFO[MISC]: Ubuntu installation started
INFO[MISC]: Installing OS image
INFO[MISC]: Installation finished</pre>

Only if DPU remote access (LAN/internet) is required, enable ip routing on host: add to <code>/etc/sysctl.d/50-dpu.conf</code>

<pre>net.ipv4.conf.all.forwarding = 1
net.ipv6.conf.all.forwarding = 1</pre>

and setup IPv4 masquerading via nftables:

<pre># nft add table nat
# nft -- add chain nat prerouting { type nat hook prerouting priority -100 \; }
# nft -- add chain nat postrouting { type nat hook postrouting priority 100 \; }
# nft add rule nat postrouting oifname "ens6f0" snat to $(host IP)
# nft list ruleset > /etc/nftables/dpu_nat.nft
# echo "include "/etc/nftables/dpu_nat.nft" >> /etc/sysconfig/nftables.conf
# systemctl enable nftables.service</pre>

== DPU login and configuration ==

First set up an appropriate IP configuration for the tmfifo_net0 interface on the host. The DPU is factory-configured at 192.168.100.2.

<pre>
# nmcli conn add type tun mode tap con-name tmfifo_net0 ifname tmfifo_net0 autoconnect yes ip4 192.168.100.1/24 ipv4.never-default true
# nmci conn up tmfifo_net0
</pre>

then login via ssh:

<pre>$ ssh ubuntu@192.168.100.2</pre>

You should receive the Ubuntu OS login prompt, and will be prompted to update the user password.

You can now update the DPU firmware:

<pre>sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl</pre>

and configure it:

<pre>
sudo mst start
# Reset all settings
sudo mlxconfig -d /dev/mst/mt41686_pciconf0 -y reset
# Set port 1 to Ethernet mode (not Infiniband)
sudo mlxconfig -d /dev/mst/mt41686_pciconf0 s LINK_TYPE_P1=2
</pre>

Reboot the host and ensure settings persist.

== References ==

[https://docs.nvidia.com/networking/display/BlueFieldDPUOSLatest/Modes+of+Operation BlueField Modes of Operation]
[https://docs.nvidia.com/networking/display/BlueFieldDPUOSLatest/Functional+Diagram BlueField Functional Diagram]
[https://docs.nvidia.com/networking/display/BlueFieldDPUOSLatest/Kernel+Representors+Model BlueField Kernel Representors] (names for the passthrough interfaces from the DPU side)

Hardware and IT Configuration

2023-05-17T16:32:57Z

Tguruswamy: /* IT and General Computing Resources */

This page exists to collect links and resources on controls hardware and related topics useful to beamline scientists. Examples might include information on certain control equipment (detectors, motors, FPGAs, etc.), correct setup and use of networking or storage IT infrastructure (without reference to specific ANL information), and low-level configuration of operating systems and device drivers for effective operation.

==Controls Hardware==
* [https://wiki.aps.anl.gov/bts/index.php?title=Main_Page Detector Pool wiki] (APS internal)
* [https://wiki.aps.anl.gov/bcda/index.php?title=Category:RECOMMENDED Beamline Controls group recommended hardware] (APS internal)

==IT and General Computing Resources==
* [[Linux network teaming]] -- aka bonding, combining multiple network interfaces for increased network bandwidth
* [[NVIDIA Data Processing Units]] -- BlueField2 series initial hardware setup
* [[NVIDIA GPUDirect]] -- transfer data to GPU memory bypassing the CPU

Hardware and IT Configuration

2023-05-17T15:23:45Z

Tguruswamy:

This page exists to collect links and resources on controls hardware and related topics useful to beamline scientists. Examples might include information on certain control equipment (detectors, motors, FPGAs, etc.), correct setup and use of networking or storage IT infrastructure (without reference to specific ANL information), and low-level configuration of operating systems and device drivers for effective operation.

==Controls Hardware==
* [https://wiki.aps.anl.gov/bts/index.php?title=Main_Page Detector Pool wiki] (APS internal)
* [https://wiki.aps.anl.gov/bcda/index.php?title=Category:RECOMMENDED Beamline Controls group recommended hardware] (APS internal)

==IT and General Computing Resources==
* [[Linux network teaming]] -- aka bonding, combining multiple network interfaces for increased network bandwidth
* [[NVIDIA DPU setup]] -- BlueField2 series initial hardware setup
* [[NVIDIA GPUDirect setup]] -- transfer data to GPU memory bypassing the CPU

Linux network teaming

2023-05-11T17:01:35Z

Tguruswamy: /* Maximum throughput (one-to-one) */

==Introduction==
Network teaming, or bonding, involves combining multiple physical network interfaces (NICs) into one virtual interface for increased reliability or throughput. It is well-supported on the APS network. However, there are many possible pitfalls and areas of confusion when configuring bonded interfaces.

==Streams or flows==
The aggregated bandwidth of a bonded link is not the same as the bandwidth available to a single "stream" or "flow" of network packets. What defines a single stream? Here are the usual possibilities (see the <code>runner.tx_hash</code> setting in Linux teamd):

* Layer 2 only: Any network packet with the same source and destination MAC is part of the same "stream".
* Layer 3: Any network packet with the same source and destination IP is part of the same "stream".
* Layer 4: Any network packet with the same source and destination IP and source and destination TCP/UDP port is part of the same "stream".

Depending on how streams are distributed among the available interfaces, the aggregated transmission bandwidth for all streams may be the same as, or different to, the transmission bandwidth available for a single stream.

==Bonding modes==
The bonding mode decides how each stream is handled. Using the teamd terminology (see <code>runner.name</code>):

; broadcast : every packet of every stream is sent over all bonded interfaces (duplicated).
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; roundrobin : the packets of every stream are cycled between interfaces in order. Packet 1 -> interface 1, Packet 2 -> interface 2, Packet 3 -> interface 1, etc.
: aggregate bandwidth = sum of all links. single-stream bandwidth = a bit less than the sum of all links (60-80% perhaps), because packets will sometimes arrive out-of-order.
; activebackup : the packets of every stream are sent over the designated "active" connection, if that fails they are sent over the next backup.
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; loadbalance : the transmitting computer hashes the stream info (Layer 2, 3, or 4 described above) modulo the number of interfaces, and uses that to assign it to an interface. For example if ''loadbalance+Layer 2'' is selected, any packet which satisfies <code>hash(src MAC, dest MAC) == 1</code> will be sent over interface 1.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, because a single stream will always hash to transmit over the same interface.
; 802.3ad / LACP : allows the transmitter to automatically notify the switch/gateway that the physical ports are bonded. After that all of the behavior is the same as loadbalance above.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, as for ''loadbalance''.

==Examples==
===Maximum throughput (one-to-one) ===
Consider a simple configuration where we are trying to get maximum throughput from a single detector PC to network storage. The connection from the detector PC is the bottleneck so we add an extra interface card.
How should we configure the two available interfaces?

[[File:Bonding_network_diagram.png|thumb|center|800px]]

There are only two ways to get more transmission bandwidth over the bonded link than a single link (10 Gb):

* Use '''roundrobin''' mode. No application support or cooperation required. Packets can arrive out-of-order, which may increase CPU load on the receiver or cause retransmissions -- leading to the usable bandwidth being perhaps 150% of a single link.
* Use '''loadbalance mode and multiple streams which hash to different interfaces'''. Requires application support to generate multiple streams and an appropriate stream definition/hashing algorithm chosen. For example, if Layer 4 hashing is chosen, the application can transmit on two separate TCP ports which hash to interface 1 and 2. The receiving application then must reconstruct the data into a single data stream, if required.

Another option is to avoid network teaming entirely, and configure each interface with separate IPs, guaranteeing both transmissions can happen in parallel.

==Testing==

<code>iperf3</code> is a common tool used to test network bandwidth between two points. Run <code>iperf3 -s</code> on one computer, and <code>iperf3 -c {hostname}</code> on another. By default, iperf3 will only create a single network stream. Adding the <code>--parallel {n}</code> flag will cause the iperf3 client to use multiple TCP/UDP ports to connect to the server and so, if ''loadbalance+Layer 4'' hashing is chosen on the bonded interfaces, allow for multi-stream testing with respect to the bonded network interfaces. Note this still runs iperf3 within a single thread and on a single CPU core.

Alternatively, start up entirely separate iperf3 server instances, listening on different ports, and run several clients in parallel. This can allow the use of multiple network streams (assuming Layer 4 hashing) and CPU cores.

==References==
* [https://www.kernel.org/doc/html/latest/networking/bonding.html Linux Ethernet Bonding Driver HOWTO]
* [https://manpages.ubuntu.com/manpages/jammy/man5/teamd.conf.5.html man(5) teamd.conf]
* [https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/

Linux network teaming

2023-05-11T17:01:09Z

Tguruswamy: /* Maximum throughput (one-to-one) */

==Introduction==
Network teaming, or bonding, involves combining multiple physical network interfaces (NICs) into one virtual interface for increased reliability or throughput. It is well-supported on the APS network. However, there are many possible pitfalls and areas of confusion when configuring bonded interfaces.

==Streams or flows==
The aggregated bandwidth of a bonded link is not the same as the bandwidth available to a single "stream" or "flow" of network packets. What defines a single stream? Here are the usual possibilities (see the <code>runner.tx_hash</code> setting in Linux teamd):

* Layer 2 only: Any network packet with the same source and destination MAC is part of the same "stream".
* Layer 3: Any network packet with the same source and destination IP is part of the same "stream".
* Layer 4: Any network packet with the same source and destination IP and source and destination TCP/UDP port is part of the same "stream".

Depending on how streams are distributed among the available interfaces, the aggregated transmission bandwidth for all streams may be the same as, or different to, the transmission bandwidth available for a single stream.

==Bonding modes==
The bonding mode decides how each stream is handled. Using the teamd terminology (see <code>runner.name</code>):

; broadcast : every packet of every stream is sent over all bonded interfaces (duplicated).
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; roundrobin : the packets of every stream are cycled between interfaces in order. Packet 1 -> interface 1, Packet 2 -> interface 2, Packet 3 -> interface 1, etc.
: aggregate bandwidth = sum of all links. single-stream bandwidth = a bit less than the sum of all links (60-80% perhaps), because packets will sometimes arrive out-of-order.
; activebackup : the packets of every stream are sent over the designated "active" connection, if that fails they are sent over the next backup.
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; loadbalance : the transmitting computer hashes the stream info (Layer 2, 3, or 4 described above) modulo the number of interfaces, and uses that to assign it to an interface. For example if ''loadbalance+Layer 2'' is selected, any packet which satisfies <code>hash(src MAC, dest MAC) == 1</code> will be sent over interface 1.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, because a single stream will always hash to transmit over the same interface.
; 802.3ad / LACP : allows the transmitter to automatically notify the switch/gateway that the physical ports are bonded. After that all of the behavior is the same as loadbalance above.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, as for ''loadbalance''.

==Examples==
===Maximum throughput (one-to-one) ===
Consider a simple configuration where we are trying to get maximum throughput from a single detector PC to network storage. The connection from the detector PC is the bottleneck so we add an extra interface card.
How should we configure the two available interfaces?

[[File:Bonding_network_diagram.png|thumb|center|800px]]

There are only two ways to get more transmission bandwidth over the bonded link than a single link (10 Gb):

* Use '''roundrobin''' mode. No application support or cooperation required. Packets can arrive out-of-order, which may increase CPU load on the receiver or cause retransmissions -- leading to the usable bandwidth being more like 150% of a single link.
* Use '''loadbalance mode and multiple streams which hash to different interfaces'''. Requires application support to generate multiple streams and an appropriate stream definition/hashing algorithm chosen. For example, if Layer 4 hashing is chosen, the application can transmit on two separate TCP ports which hash to interface 1 and 2. The receiving application then must reconstruct the data into a single data stream, if required.

Another option is to avoid network teaming entirely, and configure each interface with separate IPs, guaranteeing both transmissions can happen in parallel.

==Testing==

<code>iperf3</code> is a common tool used to test network bandwidth between two points. Run <code>iperf3 -s</code> on one computer, and <code>iperf3 -c {hostname}</code> on another. By default, iperf3 will only create a single network stream. Adding the <code>--parallel {n}</code> flag will cause the iperf3 client to use multiple TCP/UDP ports to connect to the server and so, if ''loadbalance+Layer 4'' hashing is chosen on the bonded interfaces, allow for multi-stream testing with respect to the bonded network interfaces. Note this still runs iperf3 within a single thread and on a single CPU core.

Alternatively, start up entirely separate iperf3 server instances, listening on different ports, and run several clients in parallel. This can allow the use of multiple network streams (assuming Layer 4 hashing) and CPU cores.

==References==
* [https://www.kernel.org/doc/html/latest/networking/bonding.html Linux Ethernet Bonding Driver HOWTO]
* [https://manpages.ubuntu.com/manpages/jammy/man5/teamd.conf.5.html man(5) teamd.conf]
* [https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/

Linux network teaming

2023-05-11T17:00:03Z

Tguruswamy: /* Maximum throughput (one-to-one) */

==Introduction==
Network teaming, or bonding, involves combining multiple physical network interfaces (NICs) into one virtual interface for increased reliability or throughput. It is well-supported on the APS network. However, there are many possible pitfalls and areas of confusion when configuring bonded interfaces.

==Streams or flows==
The aggregated bandwidth of a bonded link is not the same as the bandwidth available to a single "stream" or "flow" of network packets. What defines a single stream? Here are the usual possibilities (see the <code>runner.tx_hash</code> setting in Linux teamd):

* Layer 2 only: Any network packet with the same source and destination MAC is part of the same "stream".
* Layer 3: Any network packet with the same source and destination IP is part of the same "stream".
* Layer 4: Any network packet with the same source and destination IP and source and destination TCP/UDP port is part of the same "stream".

Depending on how streams are distributed among the available interfaces, the aggregated transmission bandwidth for all streams may be the same as, or different to, the transmission bandwidth available for a single stream.

==Bonding modes==
The bonding mode decides how each stream is handled. Using the teamd terminology (see <code>runner.name</code>):

; broadcast : every packet of every stream is sent over all bonded interfaces (duplicated).
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; roundrobin : the packets of every stream are cycled between interfaces in order. Packet 1 -> interface 1, Packet 2 -> interface 2, Packet 3 -> interface 1, etc.
: aggregate bandwidth = sum of all links. single-stream bandwidth = a bit less than the sum of all links (60-80% perhaps), because packets will sometimes arrive out-of-order.
; activebackup : the packets of every stream are sent over the designated "active" connection, if that fails they are sent over the next backup.
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; loadbalance : the transmitting computer hashes the stream info (Layer 2, 3, or 4 described above) modulo the number of interfaces, and uses that to assign it to an interface. For example if ''loadbalance+Layer 2'' is selected, any packet which satisfies <code>hash(src MAC, dest MAC) == 1</code> will be sent over interface 1.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, because a single stream will always hash to transmit over the same interface.
; 802.3ad / LACP : allows the transmitter to automatically notify the switch/gateway that the physical ports are bonded. After that all of the behavior is the same as loadbalance above.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, as for ''loadbalance''.

==Examples==
===Maximum throughput (one-to-one) ===
Consider a simple configuration where we are trying to get maximum throughput from a single detector PC to network storage '''using a single network stream'''. The connection from the detector PC is the bottleneck so we add an extra interface card.
How should we configure the two available interfaces?

[[File:Bonding_network_diagram.png|thumb|center|800px]]

There are only two ways to get more transmission bandwidth over the bonded link than a single link (10 Gb):

* Use '''roundrobin''' mode. No application support required. Packets can arrive out-of-order, which may increase CPU load on the receiver or cause retransmissions.
* Use '''loadbalance mode and multiple streams which hash to different interfaces'''. Requires application support to generate multiple streams and an appropriate stream definition/hashing algorithm chosen. For example, if Layer 4 hashing is chosen, the application can transmit on two separate TCP ports which hash to interface 1 and 2. The receiving application then must reconstruct the data into a single data stream, if required.

Another option is to avoid network teaming entirely, and configure each interface with separate IPs, guaranteeing both transmissions can happen in parallel.

==Testing==

<code>iperf3</code> is a common tool used to test network bandwidth between two points. Run <code>iperf3 -s</code> on one computer, and <code>iperf3 -c {hostname}</code> on another. By default, iperf3 will only create a single network stream. Adding the <code>--parallel {n}</code> flag will cause the iperf3 client to use multiple TCP/UDP ports to connect to the server and so, if ''loadbalance+Layer 4'' hashing is chosen on the bonded interfaces, allow for multi-stream testing with respect to the bonded network interfaces. Note this still runs iperf3 within a single thread and on a single CPU core.

Alternatively, start up entirely separate iperf3 server instances, listening on different ports, and run several clients in parallel. This can allow the use of multiple network streams (assuming Layer 4 hashing) and CPU cores.

==References==
* [https://www.kernel.org/doc/html/latest/networking/bonding.html Linux Ethernet Bonding Driver HOWTO]
* [https://manpages.ubuntu.com/manpages/jammy/man5/teamd.conf.5.html man(5) teamd.conf]
* [https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/

Controls Software Documentation

2023-03-15T17:43:00Z

Tguruswamy: /* areaDetector */ update url

This page exists to collect links to documentation about the instrument control system EPICS, its modules and related libraries (synApps, areaDetector), as used at the Advanced Photon Source.

Documentation referenced should be appropriate for beamline scientists, as opposed to only EPICS developers.
New deployments should prefer EPICSv7, the latest release series.

See also the extensive [https://epics-controls.org/resources-and-support/documents/training/ resources from past EPICS Collaboration meetings].

==EPICS Base==
* [https://docs.epics-controls.org/en/latest/guides/EPICS_Intro.html Intro to EPICS]
* [https://docs.epics-controls.org/en/latest/guides/EPICS_Process_Database_Concepts.html EPICS Database Concepts]
* [https://docs.epics-controls.org/projects/how-tos/en/latest/getting-started/installation.html EPICS Base Getting Started (official)]
* [https://cmd-response.readthedocs.io/en/latest/epics/rpi_epics.html Installing EPICS on RaspberryPi]

==synApps (aka EPICS Support)==
* [https://epics.anl.gov/bcda/synApps/synApps_docs_all.html synApps Collected Documentation]
* [https://epics.anl.gov/bcda/synApps/synApps.html synApps deployment instructions, including use of xxx template ioc]

==areaDetector==
* [https://areadetector.github.io/areaDetector/ areaDetector Official Documentation]

==Deploying, troubleshooting, and maintaining IOCs==
* [[Troubleshooting EPICS IOCs]]
* [https://docs2bm.readthedocs.io/en/latest/source/ops/item_000_0.html simDetector Install directions (2BM)]
* [https://epics-modules.github.io/xxx/overview.html The Basics of an IOC (XXX documentation)]
* [https://gist.github.com/keenanlang/e768774cbd0a367b96281939b8a2c9ba Creation of an IOC using mkioc]
* [https://wiki.aps.anl.gov/bcda/index.php/Category:TUTORIAL BCDA Tutorials (APS Internal)]

==Pre-built IOCs==
===Docker images===
* EPICS Base 7.0.5, synApps 6.2: [https://github.com/prjemian/epics-docker/tree/main/v1.1/n3_synApps#readmemd standard xxx], [https://github.com/prjemian/epics-docker/tree/main/v1.1/n5_custom_synApps#readmemd customized]
* EPICS Base 7.0.5, synApps 6.2, area detector 3.10: [https://github.com/prjemian/epics-docker/tree/main/v1.1/n4_areaDetector#readmemd standard], [https://github.com/prjemian/epics-docker/tree/main/v1.1/n6_custom_areaDetector#readmemd custom]
===Virtual Machines===
* [https://github.com/BCDA-APS/epics-bluesky-vm/blob/main/install_vm.md APS Bluesky 101 training VM image.]

==Maintaining Documentation==
* [https://bctwg.readthedocs.io/en/latest/index.html ReadTheDocs tutorial and example site]

==APS ESAF & Proposal - Read-only interface==
* [https://bcda-aps.github.io/apsbss/ apsbss - Read-only interface to information from APS Proposal and ESAF (experiment safety approval form) databases.]

==Bluesky==
* [https://blueskyproject.io Bluesky home page]
* [https://try.nsls2.bnl.gov Try Bluesky in a web browser (from NSLS-II)]
* [https://bcda-aps.github.io/apstools apstools - Various Python tools for use with Bluesky at the APS.], [https://bcda-aps.github.io/apstools/examples/index.html Guides, How-Tos, and other examples.] [https://github.com/BCDA-APS/bluesky_training/blob/main/APS_Bluesky_101.md APS Bluesky 101 (from 2021).]
* [https://blueskyproject.io/hklpy/ hklpy: Bluesky's diffractometer support], [https://blueskyproject.io/hklpy/examples/index.html Diffractometer examples.]
* [https://github.com/BCDA-APS/bluesky_training Bluesky training for APS users]
* [https://github.com/BCDA-APS/bluesky_training/tree/main/bluesky Template for new instrument description. (Includes configuration for IPython console, Jupyter notebook, and bluesky-queueserver.]
* [https://github.com/BCDA-APS/use_bluesky#use_bluesky Installation Steps for APS users]
* [https://blueskyproject.io/bluesky-queueserver/ queueserver - Server for queueing plans.], [https://blueskyproject.io/bluesky-queueserver-api/ API documentation]
* [https://blueskyproject.io/tiled/ tiled - Data server for databroker and directories with data files.], [https://github.com/BCDA-APS/tiled-template Template for creating a local tiled data server at APS.]
* [https://slaclab.github.io/pydm/ PyDM - Python Display Manager]
* [https://github.com/BCDA-APS/adl2pydm adl2pydm - Convert MEDM's .adl files to PyDM's .ui format.]

==Python support==
===Channel Access===
* [https://pyepics.github.io/pyepics/ PyEpics - Python interface for Channel Access based on libca and libCom.]
* [https://caproto.github.io/caproto/v0.8.1/ caproto - Python interface for Channel Access in pure Python.]
===PV Access===
* [https://mdavidsaver.github.io/p4p/ p4p - Python interface for PV Access]
* [https://github.com/epics-base/pvaPy pvapy - Python interface for PV Access, also provides CA interface]
===General===
* [https://bcdamenu.readthedocs.io/ bcdamenu - GUI menu to beam line software, such as synoptic display.]
* [https://github.com/EPICS-synApps/support/tree/master/utils/mdaPythonUtils mdaPythonUtils - Read .mda data files into Python.]
* [https://github.com/prjemian/pvMail PvMail - Watch an EPICS PV. Send email when it changes from 0 to 1.]
* [https://github.com/BCDA-APS/pvview pvView - Display one or more EPICS PVs in a PyDM GUI window as a table.]
* [https://github.com/BCDA-APS/pvWebMonitor pvWebMonitor - Post EPICS PVs to read-only (static) web site.]
* [https://prjemian.github.io/punx/ punx - Python Utilities for NeXus HDF5 files: validation, structure, hierarchy.]
* [https://prjemian.github.io/pyRestTable/ pyRestTable - Format a nice table in reST (reStructuredText) from Python.]
* [https://prjemian.github.io/pysumreg/ pysumreg - Statistics of list of (x,y) pairs from calculator-style summation registers.]
* [https://github.com/prjemian/spec2nexus/ spec2nexus - Read SPEC data files into Python, write NeXus data file.]

Linux network teaming

2023-03-15T06:55:43Z

Tguruswamy: /* Testing */

==Introduction==
Network teaming, or bonding, involves combining multiple physical network interfaces (NICs) into one virtual interface for increased reliability or throughput. It is well-supported on the APS network. However, there are many possible pitfalls and areas of confusion when configuring bonded interfaces.

==Streams or flows==
The aggregated bandwidth of a bonded link is not the same as the bandwidth available to a single "stream" or "flow" of network packets. What defines a single stream? Here are the usual possibilities (see the <code>runner.tx_hash</code> setting in Linux teamd):

* Layer 2 only: Any network packet with the same source and destination MAC is part of the same "stream".
* Layer 3: Any network packet with the same source and destination IP is part of the same "stream".
* Layer 4: Any network packet with the same source and destination IP and source and destination TCP/UDP port is part of the same "stream".

Depending on how streams are distributed among the available interfaces, the aggregated transmission bandwidth for all streams may be the same as, or different to, the transmission bandwidth available for a single stream.

==Bonding modes==
The bonding mode decides how each stream is handled. Using the teamd terminology (see <code>runner.name</code>):

; broadcast : every packet of every stream is sent over all bonded interfaces (duplicated).
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; roundrobin : the packets of every stream are cycled between interfaces in order. Packet 1 -> interface 1, Packet 2 -> interface 2, Packet 3 -> interface 1, etc.
: aggregate bandwidth = sum of all links. single-stream bandwidth = a bit less than the sum of all links (60-80% perhaps), because packets will sometimes arrive out-of-order.
; activebackup : the packets of every stream are sent over the designated "active" connection, if that fails they are sent over the next backup.
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; loadbalance : the transmitting computer hashes the stream info (Layer 2, 3, or 4 described above) modulo the number of interfaces, and uses that to assign it to an interface. For example if ''loadbalance+Layer 2'' is selected, any packet which satisfies <code>hash(src MAC, dest MAC) == 1</code> will be sent over interface 1.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, because a single stream will always hash to transmit over the same interface.
; 802.3ad / LACP : allows the transmitter to automatically notify the switch/gateway that the physical ports are bonded. After that all of the behavior is the same as loadbalance above.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, as for ''loadbalance''.

==Examples==
===Maximum throughput (one-to-one) ===
Consider a simple configuration where we are trying to get maximum throughput from a single detector PC to network storage. The connection from the detector PC is the bottleneck so we add an extra interface card.
How should we configure the two available interfaces?

[[File:Bonding_network_diagram.png|thumb|center|800px]]

There are only two ways to get more transmission bandwidth over the bonded link than a single link (10 Gb):

* Use '''roundrobin''' mode. No application support required. Packets can arrive out-of-order, which may increase CPU load on the receiver or cause retransmissions.
* Use '''loadbalance mode and multiple streams which hash to different interfaces'''. Requires application support to generate multiple streams and an appropriate stream definition/hashing algorithm chosen. For example, if Layer 4 hashing is chosen, the application can transmit on two separate TCP ports which hash to interface 1 and 2. The receiving application then must reconstruct the data into a single data stream, if required.

Another option is to avoid network teaming entirely, and configure each interface with separate IPs, guaranteeing both transmissions can happen in parallel.

==Testing==

<code>iperf3</code> is a common tool used to test network bandwidth between two points. Run <code>iperf3 -s</code> on one computer, and <code>iperf3 -c {hostname}</code> on another. By default, iperf3 will only create a single network stream. Adding the <code>--parallel {n}</code> flag will cause the iperf3 client to use multiple TCP/UDP ports to connect to the server and so, if ''loadbalance+Layer 4'' hashing is chosen on the bonded interfaces, allow for multi-stream testing with respect to the bonded network interfaces. Note this still runs iperf3 within a single thread and on a single CPU core.

Alternatively, start up entirely separate iperf3 server instances, listening on different ports, and run several clients in parallel. This can allow the use of multiple network streams (assuming Layer 4 hashing) and CPU cores.

==References==
* [https://www.kernel.org/doc/html/latest/networking/bonding.html Linux Ethernet Bonding Driver HOWTO]
* [https://manpages.ubuntu.com/manpages/jammy/man5/teamd.conf.5.html man(5) teamd.conf]
* [https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/

Linux network teaming

2023-03-15T06:55:26Z

Tguruswamy: /* Maximum throughput (one-to-one) */

==Introduction==
Network teaming, or bonding, involves combining multiple physical network interfaces (NICs) into one virtual interface for increased reliability or throughput. It is well-supported on the APS network. However, there are many possible pitfalls and areas of confusion when configuring bonded interfaces.

==Streams or flows==
The aggregated bandwidth of a bonded link is not the same as the bandwidth available to a single "stream" or "flow" of network packets. What defines a single stream? Here are the usual possibilities (see the <code>runner.tx_hash</code> setting in Linux teamd):

* Layer 2 only: Any network packet with the same source and destination MAC is part of the same "stream".
* Layer 3: Any network packet with the same source and destination IP is part of the same "stream".
* Layer 4: Any network packet with the same source and destination IP and source and destination TCP/UDP port is part of the same "stream".

Depending on how streams are distributed among the available interfaces, the aggregated transmission bandwidth for all streams may be the same as, or different to, the transmission bandwidth available for a single stream.

==Bonding modes==
The bonding mode decides how each stream is handled. Using the teamd terminology (see <code>runner.name</code>):

; broadcast : every packet of every stream is sent over all bonded interfaces (duplicated).
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; roundrobin : the packets of every stream are cycled between interfaces in order. Packet 1 -> interface 1, Packet 2 -> interface 2, Packet 3 -> interface 1, etc.
: aggregate bandwidth = sum of all links. single-stream bandwidth = a bit less than the sum of all links (60-80% perhaps), because packets will sometimes arrive out-of-order.
; activebackup : the packets of every stream are sent over the designated "active" connection, if that fails they are sent over the next backup.
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; loadbalance : the transmitting computer hashes the stream info (Layer 2, 3, or 4 described above) modulo the number of interfaces, and uses that to assign it to an interface. For example if ''loadbalance+Layer 2'' is selected, any packet which satisfies <code>hash(src MAC, dest MAC) == 1</code> will be sent over interface 1.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, because a single stream will always hash to transmit over the same interface.
; 802.3ad / LACP : allows the transmitter to automatically notify the switch/gateway that the physical ports are bonded. After that all of the behavior is the same as loadbalance above.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, as for ''loadbalance''.

==Examples==
===Maximum throughput (one-to-one) ===
Consider a simple configuration where we are trying to get maximum throughput from a single detector PC to network storage. The connection from the detector PC is the bottleneck so we add an extra interface card.
How should we configure the two available interfaces?

[[File:Bonding_network_diagram.png|thumb|center|800px]]

There are only two ways to get more transmission bandwidth over the bonded link than a single link (10 Gb):

* Use '''roundrobin''' mode. No application support required. Packets can arrive out-of-order, which may increase CPU load on the receiver or cause retransmissions.
* Use '''loadbalance mode and multiple streams which hash to different interfaces'''. Requires application support to generate multiple streams and an appropriate stream definition/hashing algorithm chosen. For example, if Layer 4 hashing is chosen, the application can transmit on two separate TCP ports which hash to interface 1 and 2. The receiving application then must reconstruct the data into a single data stream, if required.

Another option is to avoid network teaming entirely, and configure each interface with separate IPs, guaranteeing both transmissions can happen in parallel.

==Testing==

<code>iperf3</code> is a common tool used to test network bandwidth between two points. Run <code>iperf3 -s</code> on one computer, and <code>iperf3 -c {hostname}</code> on another. By default, iperf3 will only create a single network stream. Adding the <code>--parallel {n}</code> flag will cause the iperf3 client to use multiple ports to connect to the server and so, if ''loadbalance+Layer 4'' hashing is chosen on the bonded interfaces, allow for multi-stream testing with respect to the bonded network interfaces. Note this still runs iperf3 within a single thread and on a single CPU core.

Alternatively, start up entirely separate iperf3 server instances, listening on different ports, and run several clients in parallel. This can allow the use of multiple network streams (assuming Layer 4 hashing) and CPU cores.

==References==
* [https://www.kernel.org/doc/html/latest/networking/bonding.html Linux Ethernet Bonding Driver HOWTO]
* [https://manpages.ubuntu.com/manpages/jammy/man5/teamd.conf.5.html man(5) teamd.conf]
* [https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/

Linux network teaming

2023-03-15T06:55:08Z

Tguruswamy: /* Maximum throughput (one-to-one) */

==Introduction==
Network teaming, or bonding, involves combining multiple physical network interfaces (NICs) into one virtual interface for increased reliability or throughput. It is well-supported on the APS network. However, there are many possible pitfalls and areas of confusion when configuring bonded interfaces.

==Streams or flows==
The aggregated bandwidth of a bonded link is not the same as the bandwidth available to a single "stream" or "flow" of network packets. What defines a single stream? Here are the usual possibilities (see the <code>runner.tx_hash</code> setting in Linux teamd):

* Layer 2 only: Any network packet with the same source and destination MAC is part of the same "stream".
* Layer 3: Any network packet with the same source and destination IP is part of the same "stream".
* Layer 4: Any network packet with the same source and destination IP and source and destination TCP/UDP port is part of the same "stream".

Depending on how streams are distributed among the available interfaces, the aggregated transmission bandwidth for all streams may be the same as, or different to, the transmission bandwidth available for a single stream.

==Bonding modes==
The bonding mode decides how each stream is handled. Using the teamd terminology (see <code>runner.name</code>):

; broadcast : every packet of every stream is sent over all bonded interfaces (duplicated).
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; roundrobin : the packets of every stream are cycled between interfaces in order. Packet 1 -> interface 1, Packet 2 -> interface 2, Packet 3 -> interface 1, etc.
: aggregate bandwidth = sum of all links. single-stream bandwidth = a bit less than the sum of all links (60-80% perhaps), because packets will sometimes arrive out-of-order.
; activebackup : the packets of every stream are sent over the designated "active" connection, if that fails they are sent over the next backup.
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; loadbalance : the transmitting computer hashes the stream info (Layer 2, 3, or 4 described above) modulo the number of interfaces, and uses that to assign it to an interface. For example if ''loadbalance+Layer 2'' is selected, any packet which satisfies <code>hash(src MAC, dest MAC) == 1</code> will be sent over interface 1.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, because a single stream will always hash to transmit over the same interface.
; 802.3ad / LACP : allows the transmitter to automatically notify the switch/gateway that the physical ports are bonded. After that all of the behavior is the same as loadbalance above.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, as for ''loadbalance''.

==Examples==
===Maximum throughput (one-to-one) ===
Consider a simple configuration where we are trying to get maximum throughput from a single detector PC to network storage. The connection from the detector PC is the bottleneck so we add an extra interface card.
How should we configure the two available interfaces?

[[File:Bonding_network_diagram.png|thumb|center|800px]]

There are only two ways to get more transmission bandwidth over the bonded link than a single link (10 Gb):

* Use '''roundrobin''' mode. No application support required. Packets can arrive out-of-order, which may increase CPU load on the receiver or cause retransmissions.
* Use '''loadbalance mode and multiple streams which hash to different interfaces'''. Requires application support to generate multiple streams and an appropriate stream definition/hashing algorithm chosen. For example, if Layer 4 hashing is chosen, the application can transmit on two separate ports which hash to interface 1 and 2. The receiving application then must reconstruct the data into a single data stream, if required.

Another option is to avoid network teaming entirely, and configure each interface with separate IPs, guaranteeing both transmissions can happen in parallel.

==Testing==

<code>iperf3</code> is a common tool used to test network bandwidth between two points. Run <code>iperf3 -s</code> on one computer, and <code>iperf3 -c {hostname}</code> on another. By default, iperf3 will only create a single network stream. Adding the <code>--parallel {n}</code> flag will cause the iperf3 client to use multiple ports to connect to the server and so, if ''loadbalance+Layer 4'' hashing is chosen on the bonded interfaces, allow for multi-stream testing with respect to the bonded network interfaces. Note this still runs iperf3 within a single thread and on a single CPU core.

Alternatively, start up entirely separate iperf3 server instances, listening on different ports, and run several clients in parallel. This can allow the use of multiple network streams (assuming Layer 4 hashing) and CPU cores.

==References==
* [https://www.kernel.org/doc/html/latest/networking/bonding.html Linux Ethernet Bonding Driver HOWTO]
* [https://manpages.ubuntu.com/manpages/jammy/man5/teamd.conf.5.html man(5) teamd.conf]
* [https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/

Linux network teaming

2023-03-15T06:54:48Z

Tguruswamy: /* Maximum throughput (one-to-one) */

==Introduction==
Network teaming, or bonding, involves combining multiple physical network interfaces (NICs) into one virtual interface for increased reliability or throughput. It is well-supported on the APS network. However, there are many possible pitfalls and areas of confusion when configuring bonded interfaces.

==Streams or flows==
The aggregated bandwidth of a bonded link is not the same as the bandwidth available to a single "stream" or "flow" of network packets. What defines a single stream? Here are the usual possibilities (see the <code>runner.tx_hash</code> setting in Linux teamd):

* Layer 2 only: Any network packet with the same source and destination MAC is part of the same "stream".
* Layer 3: Any network packet with the same source and destination IP is part of the same "stream".
* Layer 4: Any network packet with the same source and destination IP and source and destination TCP/UDP port is part of the same "stream".

Depending on how streams are distributed among the available interfaces, the aggregated transmission bandwidth for all streams may be the same as, or different to, the transmission bandwidth available for a single stream.

==Bonding modes==
The bonding mode decides how each stream is handled. Using the teamd terminology (see <code>runner.name</code>):

; broadcast : every packet of every stream is sent over all bonded interfaces (duplicated).
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; roundrobin : the packets of every stream are cycled between interfaces in order. Packet 1 -> interface 1, Packet 2 -> interface 2, Packet 3 -> interface 1, etc.
: aggregate bandwidth = sum of all links. single-stream bandwidth = a bit less than the sum of all links (60-80% perhaps), because packets will sometimes arrive out-of-order.
; activebackup : the packets of every stream are sent over the designated "active" connection, if that fails they are sent over the next backup.
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; loadbalance : the transmitting computer hashes the stream info (Layer 2, 3, or 4 described above) modulo the number of interfaces, and uses that to assign it to an interface. For example if ''loadbalance+Layer 2'' is selected, any packet which satisfies <code>hash(src MAC, dest MAC) == 1</code> will be sent over interface 1.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, because a single stream will always hash to transmit over the same interface.
; 802.3ad / LACP : allows the transmitter to automatically notify the switch/gateway that the physical ports are bonded. After that all of the behavior is the same as loadbalance above.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, as for ''loadbalance''.

==Examples==
===Maximum throughput (one-to-one) ===
Consider a simple configuration where we are trying to get maximum throughput from a single detector PC to network storage. The connection from the detector PC is the bottleneck so we add an extra interface card.
How should we configure the two available interfaces?

[[File:Bonding_network_diagram.png|thumb|center|800px]]

There are only two ways to get more transmission bandwidth over the bonded link than a single link (10 Gb):

* Use '''roundrobin''' mode. No application support required. Packets may arrive out-of-order, which increases CPU load on the receiver or causes retransmissions.
* Use '''loadbalance mode and multiple streams which hash to different interfaces'''. Requires application support to generate multiple streams and an appropriate stream definition/hashing algorithm chosen. For example, if Layer 4 hashing is chosen, the application can transmit on two separate ports which hash to interface 1 and 2. The receiving application then must reconstruct the data into a single data stream, if required.

Another option is to avoid network teaming entirely, and configure each interface with separate IPs, guaranteeing both transmissions can happen in parallel.

==Testing==

<code>iperf3</code> is a common tool used to test network bandwidth between two points. Run <code>iperf3 -s</code> on one computer, and <code>iperf3 -c {hostname}</code> on another. By default, iperf3 will only create a single network stream. Adding the <code>--parallel {n}</code> flag will cause the iperf3 client to use multiple ports to connect to the server and so, if ''loadbalance+Layer 4'' hashing is chosen on the bonded interfaces, allow for multi-stream testing with respect to the bonded network interfaces. Note this still runs iperf3 within a single thread and on a single CPU core.

Alternatively, start up entirely separate iperf3 server instances, listening on different ports, and run several clients in parallel. This can allow the use of multiple network streams (assuming Layer 4 hashing) and CPU cores.

==References==
* [https://www.kernel.org/doc/html/latest/networking/bonding.html Linux Ethernet Bonding Driver HOWTO]
* [https://manpages.ubuntu.com/manpages/jammy/man5/teamd.conf.5.html man(5) teamd.conf]
* [https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/

Linux network teaming

2023-03-15T06:16:10Z

Tguruswamy: /* Maximum throughput (one-to-one) */

==Introduction==
Network teaming, or bonding, involves combining multiple physical network interfaces (NICs) into one virtual interface for increased reliability or throughput. It is well-supported on the APS network. However, there are many possible pitfalls and areas of confusion when configuring bonded interfaces.

==Streams or flows==
The aggregated bandwidth of a bonded link is not the same as the bandwidth available to a single "stream" or "flow" of network packets. What defines a single stream? Here are the usual possibilities (see the <code>runner.tx_hash</code> setting in Linux teamd):

* Layer 2 only: Any network packet with the same source and destination MAC is part of the same "stream".
* Layer 3: Any network packet with the same source and destination IP is part of the same "stream".
* Layer 4: Any network packet with the same source and destination IP and source and destination TCP/UDP port is part of the same "stream".

Depending on how streams are distributed among the available interfaces, the aggregated transmission bandwidth for all streams may be the same as, or different to, the transmission bandwidth available for a single stream.

==Bonding modes==
The bonding mode decides how each stream is handled. Using the teamd terminology (see <code>runner.name</code>):

; broadcast : every packet of every stream is sent over all bonded interfaces (duplicated).
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; roundrobin : the packets of every stream are cycled between interfaces in order. Packet 1 -> interface 1, Packet 2 -> interface 2, Packet 3 -> interface 1, etc.
: aggregate bandwidth = sum of all links. single-stream bandwidth = a bit less than the sum of all links (60-80% perhaps), because packets will sometimes arrive out-of-order.
; activebackup : the packets of every stream are sent over the designated "active" connection, if that fails they are sent over the next backup.
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; loadbalance : the transmitting computer hashes the stream info (Layer 2, 3, or 4 described above) modulo the number of interfaces, and uses that to assign it to an interface. For example if ''loadbalance+Layer 2'' is selected, any packet which satisfies <code>hash(src MAC, dest MAC) == 1</code> will be sent over interface 1.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, because a single stream will always hash to transmit over the same interface.
; 802.3ad / LACP : allows the transmitter to automatically notify the switch/gateway that the physical ports are bonded. After that all of the behavior is the same as loadbalance above.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, as for ''loadbalance''.

==Examples==
===Maximum throughput (one-to-one) ===
Consider a simple configuration where we are trying to get maximum throughput from a single detector PC to network storage. The connection from the detector PC is the bottleneck so we add an extra interface card.
How should we configure the two available interfaces?

[[File:Bonding_network_diagram.png|thumb|center|800px]]

There are only two ways to get more transmission bandwidth over the bonded link than a single link (10 Gb):

* Use '''roundrobin''' mode. No application support required. Packets may arrive out-of-order, which increases CPU load on the receiver or cause retransmissions.
* Use '''loadbalance mode and multiple streams which hash to different interfaces'''. Requires application support to generate multiple streams and an appropriate stream definition/hashing algorithm chosen. For example, if Layer 4 hashing is chosen, the application can transmit on two separate ports which hash to interface 1 and 2. The receiving application then must reconstruct the data into a single data stream, if required.

Another option is to avoid network teaming entirely, and configure each interface with separate IPs, guaranteeing both transmissions can happen in parallel.

==Testing==

<code>iperf3</code> is a common tool used to test network bandwidth between two points. Run <code>iperf3 -s</code> on one computer, and <code>iperf3 -c {hostname}</code> on another. By default, iperf3 will only create a single network stream. Adding the <code>--parallel {n}</code> flag will cause the iperf3 client to use multiple ports to connect to the server and so, if ''loadbalance+Layer 4'' hashing is chosen on the bonded interfaces, allow for multi-stream testing with respect to the bonded network interfaces. Note this still runs iperf3 within a single thread and on a single CPU core.

Alternatively, start up entirely separate iperf3 server instances, listening on different ports, and run several clients in parallel. This can allow the use of multiple network streams (assuming Layer 4 hashing) and CPU cores.

==References==
* [https://www.kernel.org/doc/html/latest/networking/bonding.html Linux Ethernet Bonding Driver HOWTO]
* [https://manpages.ubuntu.com/manpages/jammy/man5/teamd.conf.5.html man(5) teamd.conf]
* [https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/

Linux network teaming

2023-03-15T06:14:21Z

Tguruswamy: /* Bonding modes */

==Introduction==
Network teaming, or bonding, involves combining multiple physical network interfaces (NICs) into one virtual interface for increased reliability or throughput. It is well-supported on the APS network. However, there are many possible pitfalls and areas of confusion when configuring bonded interfaces.

==Streams or flows==
The aggregated bandwidth of a bonded link is not the same as the bandwidth available to a single "stream" or "flow" of network packets. What defines a single stream? Here are the usual possibilities (see the <code>runner.tx_hash</code> setting in Linux teamd):

* Layer 2 only: Any network packet with the same source and destination MAC is part of the same "stream".
* Layer 3: Any network packet with the same source and destination IP is part of the same "stream".
* Layer 4: Any network packet with the same source and destination IP and source and destination TCP/UDP port is part of the same "stream".

Depending on how streams are distributed among the available interfaces, the aggregated transmission bandwidth for all streams may be the same as, or different to, the transmission bandwidth available for a single stream.

==Bonding modes==
The bonding mode decides how each stream is handled. Using the teamd terminology (see <code>runner.name</code>):

; broadcast : every packet of every stream is sent over all bonded interfaces (duplicated).
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; roundrobin : the packets of every stream are cycled between interfaces in order. Packet 1 -> interface 1, Packet 2 -> interface 2, Packet 3 -> interface 1, etc.
: aggregate bandwidth = sum of all links. single-stream bandwidth = a bit less than the sum of all links (60-80% perhaps), because packets will sometimes arrive out-of-order.
; activebackup : the packets of every stream are sent over the designated "active" connection, if that fails they are sent over the next backup.
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; loadbalance : the transmitting computer hashes the stream info (Layer 2, 3, or 4 described above) modulo the number of interfaces, and uses that to assign it to an interface. For example if ''loadbalance+Layer 2'' is selected, any packet which satisfies <code>hash(src MAC, dest MAC) == 1</code> will be sent over interface 1.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, because a single stream will always hash to transmit over the same interface.
; 802.3ad / LACP : allows the transmitter to automatically notify the switch/gateway that the physical ports are bonded. After that all of the behavior is the same as loadbalance above.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, as for ''loadbalance''.

==Examples==
===Maximum throughput (one-to-one) ===
Consider a simple configuration where we are trying to get maximum throughput from a single detector PC to network storage. The connection from the detector PC is the bottleneck so we add an extra interface card.
How should we configure the two available interfaces?

[[File:Bonding_network_diagram.png|thumb|center|800px]]

There are only two ways to get more transmission bandwidth over the bonded link than a single link (10 Gb):

* Use '''roundrobin''' mode. No application support required. Packets may arrive out-of-order, which increases CPU load on the receiver or cause retransmissions.
* Use '''loadbalance mode and multiple streams which hash to different interfaces'''. Requires application support to generate multiple streams and an appropriate stream definition/hashing algorithm chosen. For example, if Layer 4 hashing is chosen, the application can transmit on two separate ports which hash to interface 1 and 2. The receiving application then must reconstruct the data into a single data stream, if required.

Another option is to avoid network teaming entirely, and configure each interface with separate IPs on different subnets, guaranteeing both transmissions can happen in parallel.

==Testing==

<code>iperf3</code> is a common tool used to test network bandwidth between two points. Run <code>iperf3 -s</code> on one computer, and <code>iperf3 -c {hostname}</code> on another. By default, iperf3 will only create a single network stream. Adding the <code>--parallel {n}</code> flag will cause the iperf3 client to use multiple ports to connect to the server and so, if ''loadbalance+Layer 4'' hashing is chosen on the bonded interfaces, allow for multi-stream testing with respect to the bonded network interfaces. Note this still runs iperf3 within a single thread and on a single CPU core.

Alternatively, start up entirely separate iperf3 server instances, listening on different ports, and run several clients in parallel. This can allow the use of multiple network streams (assuming Layer 4 hashing) and CPU cores.

==References==
* [https://www.kernel.org/doc/html/latest/networking/bonding.html Linux Ethernet Bonding Driver HOWTO]
* [https://manpages.ubuntu.com/manpages/jammy/man5/teamd.conf.5.html man(5) teamd.conf]
* [https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/

Linux network teaming

2023-03-15T05:11:06Z

Tguruswamy: /* Testing and troubleshooting */

==Introduction==
Network teaming, or bonding, involves combining multiple physical network interfaces (NICs) into one virtual interface for increased reliability or throughput. It is well-supported on the APS network. However, there are many possible pitfalls and areas of confusion when configuring bonded interfaces.

==Streams or flows==
The aggregated bandwidth of a bonded link is not the same as the bandwidth available to a single "stream" or "flow" of network packets. What defines a single stream? Here are the usual possibilities (see the <code>runner.tx_hash</code> setting in Linux teamd):

* Layer 2 only: Any network packet with the same source and destination MAC is part of the same "stream".
* Layer 3: Any network packet with the same source and destination IP is part of the same "stream".
* Layer 4: Any network packet with the same source and destination IP and source and destination TCP/UDP port is part of the same "stream".

Depending on how streams are distributed among the available interfaces, the aggregated transmission bandwidth for all streams may be the same as, or different to, the transmission bandwidth available for a single stream.

==Bonding modes==
The bonding mode decides how each stream is handled. Using the teamd terminology (see <code>runner.name</code>):

; broadcast : every packet of every stream is sent over all bonded interfaces.
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; roundrobin : the packets of every stream are cycled between interfaces in order. Packet 1 -> interface 1, Packet 2 -> interface 2, Packet 3 -> interface 1, etc.
: aggregate bandwidth = sum of all links. single-stream bandwidth = a bit less than the sum of all links (60-80% perhaps), because packets will sometimes arrive out-of-order.
; activebackup : the packets of every stream are sent over the designated "active" connection, if that fails they are sent over the next backup.
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; loadbalance : the transmitting computer hashes the stream info (Layer 2, 3, or 4 described above) modulo the number of interfaces, and uses that to assign it to an interface. For example if ''loadbalance+Layer 2'' is selected, any packet which satisfies <code>hash(src MAC, dest MAC) == 1</code> will be sent over interface 1.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, because a single stream will always hash to transmit over the same interface.
; 802.3ad / LACP : allows the transmitter to automatically notify the switch/gateway that the physical ports are bonded. After that all of the behavior is the same as loadbalance above.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, as for ''loadbalance''.

==Examples==
===Maximum throughput (one-to-one) ===
Consider a simple configuration where we are trying to get maximum throughput from a single detector PC to network storage. The connection from the detector PC is the bottleneck so we add an extra interface card.
How should we configure the two available interfaces?

[[File:Bonding_network_diagram.png|thumb|center|800px]]

There are only two ways to get more transmission bandwidth over the bonded link than a single link (10 Gb):

* Use '''roundrobin''' mode. No application support required. Packets may arrive out-of-order, which increases CPU load on the receiver or cause retransmissions.
* Use '''loadbalance mode and multiple streams which hash to different interfaces'''. Requires application support to generate multiple streams and an appropriate stream definition/hashing algorithm chosen. For example, if Layer 4 hashing is chosen, the application can transmit on two separate ports which hash to interface 1 and 2. The receiving application then must reconstruct the data into a single data stream, if required.

Another option is to avoid network teaming entirely, and configure each interface with separate IPs on different subnets, guaranteeing both transmissions can happen in parallel.

==Testing==

<code>iperf3</code> is a common tool used to test network bandwidth between two points. Run <code>iperf3 -s</code> on one computer, and <code>iperf3 -c {hostname}</code> on another. By default, iperf3 will only create a single network stream. Adding the <code>--parallel {n}</code> flag will cause the iperf3 client to use multiple ports to connect to the server and so, if ''loadbalance+Layer 4'' hashing is chosen on the bonded interfaces, allow for multi-stream testing with respect to the bonded network interfaces. Note this still runs iperf3 within a single thread and on a single CPU core.

Alternatively, start up entirely separate iperf3 server instances, listening on different ports, and run several clients in parallel. This can allow the use of multiple network streams (assuming Layer 4 hashing) and CPU cores.

==References==
* [https://www.kernel.org/doc/html/latest/networking/bonding.html Linux Ethernet Bonding Driver HOWTO]
* [https://manpages.ubuntu.com/manpages/jammy/man5/teamd.conf.5.html man(5) teamd.conf]
* [https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/

Linux network teaming

2023-03-15T05:10:46Z

Tguruswamy: Created page with "==Introduction== Network teaming, or bonding, involves combining multiple physical network interfaces (NICs) into one virtual interface for increased reliability or throughput..."

==Introduction==
Network teaming, or bonding, involves combining multiple physical network interfaces (NICs) into one virtual interface for increased reliability or throughput. It is well-supported on the APS network. However, there are many possible pitfalls and areas of confusion when configuring bonded interfaces.

==Streams or flows==
The aggregated bandwidth of a bonded link is not the same as the bandwidth available to a single "stream" or "flow" of network packets. What defines a single stream? Here are the usual possibilities (see the <code>runner.tx_hash</code> setting in Linux teamd):

* Layer 2 only: Any network packet with the same source and destination MAC is part of the same "stream".
* Layer 3: Any network packet with the same source and destination IP is part of the same "stream".
* Layer 4: Any network packet with the same source and destination IP and source and destination TCP/UDP port is part of the same "stream".

Depending on how streams are distributed among the available interfaces, the aggregated transmission bandwidth for all streams may be the same as, or different to, the transmission bandwidth available for a single stream.

==Bonding modes==
The bonding mode decides how each stream is handled. Using the teamd terminology (see <code>runner.name</code>):

; broadcast : every packet of every stream is sent over all bonded interfaces.
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; roundrobin : the packets of every stream are cycled between interfaces in order. Packet 1 -> interface 1, Packet 2 -> interface 2, Packet 3 -> interface 1, etc.
: aggregate bandwidth = sum of all links. single-stream bandwidth = a bit less than the sum of all links (60-80% perhaps), because packets will sometimes arrive out-of-order.
; activebackup : the packets of every stream are sent over the designated "active" connection, if that fails they are sent over the next backup.
: aggregate bandwidth = single link bandwidth. single-stream bandwidth = single link bandwidth.
; loadbalance : the transmitting computer hashes the stream info (Layer 2, 3, or 4 described above) modulo the number of interfaces, and uses that to assign it to an interface. For example if ''loadbalance+Layer 2'' is selected, any packet which satisfies <code>hash(src MAC, dest MAC) == 1</code> will be sent over interface 1.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, because a single stream will always hash to transmit over the same interface.
; 802.3ad / LACP : allows the transmitter to automatically notify the switch/gateway that the physical ports are bonded. After that all of the behavior is the same as loadbalance above.
: aggregate bandwidth = sum of all links. single stream bandwidth = single link bandwidth, as for ''loadbalance''.

==Examples==
===Maximum throughput (one-to-one) ===
Consider a simple configuration where we are trying to get maximum throughput from a single detector PC to network storage. The connection from the detector PC is the bottleneck so we add an extra interface card.
How should we configure the two available interfaces?

[[File:Bonding_network_diagram.png|thumb|center|800px]]

There are only two ways to get more transmission bandwidth over the bonded link than a single link (10 Gb):

* Use '''roundrobin''' mode. No application support required. Packets may arrive out-of-order, which increases CPU load on the receiver or cause retransmissions.
* Use '''loadbalance mode and multiple streams which hash to different interfaces'''. Requires application support to generate multiple streams and an appropriate stream definition/hashing algorithm chosen. For example, if Layer 4 hashing is chosen, the application can transmit on two separate ports which hash to interface 1 and 2. The receiving application then must reconstruct the data into a single data stream, if required.

Another option is to avoid network teaming entirely, and configure each interface with separate IPs on different subnets, guaranteeing both transmissions can happen in parallel.

==Testing and troubleshooting==

<code>iperf3</code> is a common tool used to test network bandwidth between two points. Run <code>iperf3 -s</code> on one computer, and <code>iperf3 -c {hostname}</code> on another. By default, iperf3 will only create a single network stream. Adding the <code>--parallel {n}</code> flag will cause the iperf3 client to use multiple ports to connect to the server and so, if ''loadbalance+Layer 4'' hashing is chosen on the bonded interfaces, allow for multi-stream testing with respect to the bonded network interfaces. Note this still runs iperf3 within a single thread and on a single CPU core.

Alternatively, start up entirely separate iperf3 server instances, listening on different ports, and run several clients in parallel. This can allow the use of multiple network streams (assuming Layer 4 hashing) and CPU cores.

==References==
* [https://www.kernel.org/doc/html/latest/networking/bonding.html Linux Ethernet Bonding Driver HOWTO]
* [https://manpages.ubuntu.com/manpages/jammy/man5/teamd.conf.5.html man(5) teamd.conf]
* [https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/

File:Bonding network diagram.png

2023-03-15T04:53:34Z

Tguruswamy: Tguruswamy uploaded a new version of File:Bonding network diagram.png

== Summary ==
Simple diagram used in explanation of network teaming/bonding

File:Bonding network diagram.png

2023-03-15T04:49:43Z

Tguruswamy: Simple diagram used in explanation of network teaming/bonding

== Summary ==
Simple diagram used in explanation of network teaming/bonding

Beamline Controls

2023-03-15T04:15:26Z

Tguruswamy: /* Beamline Controls Documentation */

WIKI for the APS Beamline Controls Community, maintained by the Beamline Controls Technical Working Group. 
This wiki exists to collect the output of the working group, including documentation on beamline controls software and hardware useful for APS beamline scientists and resident users.

----

===Beamline Controls Documentation===

* [[Controls Software Documentation]]
* [[EPICS_for_Dummies|EPICS 101]]
* [[Hardware and IT Configuration]]

===Practical Beamline Controls Training (2023)===

[[Practical_Beamline_Controls_Training#IOC_Basics|Session 1: IOC Basics]]

[[Practical_Beamline_Controls_Training#IOC_Deployment_.26_Troubleshooting|Session 2: IOC Deployment & Troubleshooting]]

===Beamline Controls Technical Working Group information===

The BC Technical Working (BCTWG) group meets monthly to review and gather feedback on possible changes in either the way BC delivers support or the technologies that BC supports. The BCTWG comprises the BC group plus interested stakeholders from XSD. In a nutshell, the main goal of the BCTWG is to improve communication between BC and its customers on topics of mutual interest.

* Box folder with slides and meeting notes (ANL only): [https://anl.box.com/s/chcn7xauil93inay8ipinbxtcckc9k9f]

* Current chairs:
** Kevin Peterson (XSD-BC)
** Tejas Guruswamy (XSD-DET)
** Andrew Chuang (XSD-MPE)

Hardware and IT Configuration

2023-03-15T03:49:54Z

Tguruswamy: Created page with "This page exists to collect links and resources on controls hardware and related topics useful to beamline scientists. Examples might include information on certain control eq..."

This page exists to collect links and resources on controls hardware and related topics useful to beamline scientists. Examples might include information on certain control equipment (detectors, motors, FPGAs, etc.), correct setup and use of networking or storage IT infrastructure (without reference to specific ANL information), and low-level configuration of operating systems and device drivers for effective operation.

==Controls Hardware==
* [https://wiki.aps.anl.gov/bts/index.php?title=Main_Page Detector Pool wiki] (APS internal)
* [https://wiki.aps.anl.gov/bcda/index.php?title=Category:RECOMMENDED Beamline Controls group recommended hardware] (APS internal)

==IT==
* [[Linux network teaming]] -- aka bonding, combining multiple network interfaces for increased network bandwidth

Beamline Controls

2023-03-15T03:39:16Z

Tguruswamy:

WIKI for the APS Beamline Controls Community, maintained by the Beamline Controls Technical Working Group. 
This wiki exists to collect the output of the working group, including documentation on beamline controls software and hardware useful for APS beamline scientists and resident users.

----

===Beamline Controls Documentation===

* [[Controls Software Documentation]]
* [[EPICS_for_Dummies|EPICS 101]]
* [[Hardware Configuration]]

===Practical Beamline Controls Training (2023)===

[[Practical_Beamline_Controls_Training#IOC_Basics|Session 1: IOC Basics]]

[[Practical_Beamline_Controls_Training#IOC_Deployment_.26_Troubleshooting|Session 2: IOC Deployment & Troubleshooting]]

===Beamline Controls Technical Working Group information===

The BC Technical Working (BCTWG) group meets monthly to review and gather feedback on possible changes in either the way BC delivers support or the technologies that BC supports. The BCTWG comprises the BC group plus interested stakeholders from XSD. In a nutshell, the main goal of the BCTWG is to improve communication between BC and its customers on topics of mutual interest.

* Box folder with slides and meeting notes (ANL only): [https://anl.box.com/s/chcn7xauil93inay8ipinbxtcckc9k9f]

* Current chairs:
** Kevin Peterson (XSD-BC)
** Tejas Guruswamy (XSD-DET)
** Andrew Chuang (XSD-MPE)

Documentation

2023-03-15T03:37:30Z

Tguruswamy: Tguruswamy moved page Documentation to Controls Software Documentation: make room for hardware page

#REDIRECT [[Controls Software Documentation]]

Controls Software Documentation

2023-03-15T03:37:29Z

Tguruswamy: Tguruswamy moved page Documentation to Controls Software Documentation: make room for hardware page

This page exists to collect links to documentation about the instrument control system EPICS, its modules and related libraries (synApps, areaDetector), as used at the Advanced Photon Source.

Documentation referenced should be appropriate for beamline scientists, as opposed to only EPICS developers.
New deployments should prefer EPICSv7, the latest release series.

See also the extensive [https://epics-controls.org/resources-and-support/documents/training/ resources from past EPICS Collaboration meetings].

==EPICS Base==
* [https://docs.epics-controls.org/en/latest/guides/EPICS_Intro.html Intro to EPICS]
* [https://docs.epics-controls.org/en/latest/guides/EPICS_Process_Database_Concepts.html EPICS Database Concepts]
* [https://docs.epics-controls.org/projects/how-tos/en/latest/getting-started/installation.html EPICS Base Getting Started (official)]
* [https://cmd-response.readthedocs.io/en/latest/epics/rpi_epics.html Installing EPICS on RaspberryPi]

==synApps (aka EPICS Support)==
* [https://epics.anl.gov/bcda/synApps/synApps_docs_all.html synApps Collected Documentation]
* [https://epics.anl.gov/bcda/synApps/synApps.html synApps deployment instructions, including use of xxx template ioc]

==areaDetector==
* [https://areadetector.github.io/master/index.html areaDetector Official Documentation]

==Deploying, troubleshooting, and maintaining IOCs==
* [[Troubleshooting EPICS IOCs]]
* [https://docs2bm.readthedocs.io/en/latest/source/ops/item_000_0.html simDetector Install directions (2BM)]
* [https://epics-modules.github.io/xxx/overview.html The Basics of an IOC (XXX documentation)]
* [https://gist.github.com/keenanlang/e768774cbd0a367b96281939b8a2c9ba Creation of an IOC using mkioc]
* [https://wiki.aps.anl.gov/bcda/index.php/Category:TUTORIAL BCDA Tutorials (APS Internal)]

==Pre-built IOCs==
===Docker images===
* EPICS Base 7.0.5, synApps 6.2: [https://github.com/prjemian/epics-docker/tree/main/v1.1/n3_synApps#readmemd standard xxx], [https://github.com/prjemian/epics-docker/tree/main/v1.1/n5_custom_synApps#readmemd customized]
* EPICS Base 7.0.5, synApps 6.2, area detector 3.10: [https://github.com/prjemian/epics-docker/tree/main/v1.1/n4_areaDetector#readmemd standard], [https://github.com/prjemian/epics-docker/tree/main/v1.1/n6_custom_areaDetector#readmemd custom]
===Virtual Machines===
* [https://github.com/BCDA-APS/epics-bluesky-vm/blob/main/install_vm.md APS Bluesky 101 training VM image.]

==Maintaining Documentation==
* [https://bctwg.readthedocs.io/en/latest/index.html ReadTheDocs tutorial and example site]

==APS ESAF & Proposal - Read-only interface==
* [https://bcda-aps.github.io/apsbss/ apsbss - Read-only interface to information from APS Proposal and ESAF (experiment safety approval form) databases.]

==Bluesky==
* [https://blueskyproject.io Bluesky home page]
* [https://try.nsls2.bnl.gov Try Bluesky in a web browser (from NSLS-II)]
* [https://bcda-aps.github.io/apstools apstools - Various Python tools for use with Bluesky at the APS.], [https://bcda-aps.github.io/apstools/examples/index.html Guides, How-Tos, and other examples.] [https://github.com/BCDA-APS/bluesky_training/blob/main/APS_Bluesky_101.md APS Bluesky 101 (from 2021).]
* [https://blueskyproject.io/hklpy/ hklpy: Bluesky's diffractometer support], [https://blueskyproject.io/hklpy/examples/index.html Diffractometer examples.]
* [https://github.com/BCDA-APS/bluesky_training Bluesky training for APS users]
* [https://github.com/BCDA-APS/bluesky_training/tree/main/bluesky Template for new instrument description. (Includes configuration for IPython console, Jupyter notebook, and bluesky-queueserver.]
* [https://github.com/BCDA-APS/use_bluesky#use_bluesky Installation Steps for APS users]
* [https://blueskyproject.io/bluesky-queueserver/ queueserver - Server for queueing plans.], [https://blueskyproject.io/bluesky-queueserver-api/ API documentation]
* [https://blueskyproject.io/tiled/ tiled - Data server for databroker and directories with data files.], [https://github.com/BCDA-APS/tiled-template Template for creating a local tiled data server at APS.]
* [https://slaclab.github.io/pydm/ PyDM - Python Display Manager]
* [https://github.com/BCDA-APS/adl2pydm adl2pydm - Convert MEDM's .adl files to PyDM's .ui format.]

==Python support==
===Channel Access===
* [https://pyepics.github.io/pyepics/ PyEpics - Python interface for Channel Access based on libca and libCom.]
* [https://caproto.github.io/caproto/v0.8.1/ caproto - Python interface for Channel Access in pure Python.]
===PV Access===
* [https://mdavidsaver.github.io/p4p/ p4p - Python interface for PV Access]
* [https://github.com/epics-base/pvaPy pvapy - Python interface for PV Access, also provides CA interface]
===General===
* [https://bcdamenu.readthedocs.io/ bcdamenu - GUI menu to beam line software, such as synoptic display.]
* [https://github.com/EPICS-synApps/support/tree/master/utils/mdaPythonUtils mdaPythonUtils - Read .mda data files into Python.]
* [https://github.com/prjemian/pvMail PvMail - Watch an EPICS PV. Send email when it changes from 0 to 1.]
* [https://github.com/BCDA-APS/pvview pvView - Display one or more EPICS PVs in a PyDM GUI window as a table.]
* [https://github.com/BCDA-APS/pvWebMonitor pvWebMonitor - Post EPICS PVs to read-only (static) web site.]
* [https://prjemian.github.io/punx/ punx - Python Utilities for NeXus HDF5 files: validation, structure, hierarchy.]
* [https://prjemian.github.io/pyRestTable/ pyRestTable - Format a nice table in reST (reStructuredText) from Python.]
* [https://prjemian.github.io/pysumreg/ pysumreg - Statistics of list of (x,y) pairs from calculator-style summation registers.]
* [https://github.com/prjemian/spec2nexus/ spec2nexus - Read SPEC data files into Python, write NeXus data file.]

EPICS for Dummies

2023-03-08T02:25:50Z

Tguruswamy: /* What is an IOC? */ resize image

== What is EPICS? ==

EPICS (Experimental Physics and Industrial Control System) is a distributed control system, which means that the IOC (software) can run on a different computer as the user interfaces.

The next 5 sections are dedicated to the definition of basic EPICS components (IOC, PV, record, database, add devices?). To create your own EPICS support, you can skip directly to [[#How does one create an IOC?]].

== What is an IOC? ==

[[File:EPICS_basics_PV_diagram.png|right|thumb|600px]]

IOC (Input/Output Controller) is a piece of software running on a computer:
* loads '''databases'''
* serves something called '''process variables''' (PVs) that you can read and write to
* used to run on VMEs, but nowadays it runs on anything.
* usually talks to hardware, but that isn’t a requirement.

IOCs are started with startup scripts. Those scripts tell the IOC what database files to load and where those files are. When the IOC is told to load a database, what it is doing is parsing that database file, identifying the records that are contained within, and then generating the actual epics PVs that a user will interact with.

'''Records''' and '''databases''' (lists of records) are configuration files describing the initial setup for '''PVs''', the '''IOC''' is the software that manages the behavior of PVs and communication to get/set their values.

== What is a PV? ==

PV (Process Variable) is a the field from a record:
''PV = record_name + . + field_name''

Most of the interactive parts of user displays have PVs associated with them.

== What is a record? ==

Records are the building blocks of EPICS:
* There is a lot of different types of record, and each type is designed to perform a specific function
* Records are customizable via fields that allow users to configure how they behave and monitor their state (see [[#More about record fields]] to know how to learn about record fields)
* Records do nothing until they are processed (see [[#SCAN, PROC & FLNK fields]])
* Records vary significantly in complexity (see [[#More about records]])
* It is rare for new records to be created, now that ''asyn'' device support exists add link to asyn?).

=== Where do records come from? ===

Typically, records at the APS comes from either:
* '''[https://epics.anl.gov/base/index.php EPICS base]''': main core of EPICS
* '''[https://www.aps.anl.gov/BCDA/synApps synApps]''': a collection of EPICS modules that are commonly used at the APS
A module is similar to a python package: it is an already written set of databases and associated behavior you can load into your IOC to do certain things, for example talk to specific hardware. An IOC can link to several modules.

{| class="wikitable" style="margin-left: 100px; margin-right:auto"
|+ Records in synApps modules 

|-
! Module !! Records !! Module !! Records
|-
| alive || alive || motor || motor
|-
| asyn || asyn || optics || table
|-
| busy || busy || scaler || scaler
|-
| calc || acalcout, scalcout, sseq, swait, transform || sscan || sscan
|-
| camac || camac || std || epid, timestamp, throttle
|-
| lua || luascript || vac || digitel, vs
|-
| mca || mca || vme || vme
|-
| allenBradley || ab1771IFE, ab1771N, ab1771IX, ...|| allenBradley || ..., ab1791, abDcm
|}

If a record isn't listed here, it probably comes from EPICS base.
The advantage to using synApps over a collection of modules one assembles on their own is that some amount of BCDA testing has been done.

=== What is a record field? ===

Record fields are controllable or informative properties of a record.

====Noteworthy fields====

*'''VAL''' - the default field (used if no field is specified in the vast majority of cases)
*'''SCAN''' - determines when a record processes
*'''NAME''' - the record name (helpful if records are aliased)
*'''DESC''' - record description
*'''RTYP''' - the record type
*'''DTYP''' - the device support type (not meaningful for all record types)
*'''DISP''' - disable puts (ca = channel access & pva = pv access) from outside the IOC (when non-zero); e.g. disable a motor from the motorx_all screen
*'''DISV/DISA/SDIS''' - disable record processing internally
**Record processing is disabled when '''DISA''' equals '''DISV''' (e.g. uses SDIS field to read in a value/mode, which will disable some control if a certain mode is active)
**'''DISA''' is set to the value of the PV in the '''SDIS''' field, if it isn't empty
*'''TPRO''' - trace processing: when set to 1 prints out all the records that get processed after the record is processed; used only for troubleshooting

Note: some of the fields of a record are actually record attributes (they return the same value for all instances of the record type; see [https://epics.anl.gov/base/R3-16/2-docs/AppDevGuide/DatabaseDefinition.html#x7-2630006.6 https://epics.anl.gov/])

==== SCAN, PROC & FLNK fields ====

Records do nothing until they are processed. Records can be processed multiple ways:
* Manually:
** Write 1 (or any non-zero value) to a record’s '''PROC''' field
**Forward link ('''FLNK''') from another record
* Periodically by setting the '''SCAN''' field to something other than <code>Passive</code>

{| class="wikitable" style="margin-left: 100px; margin-right:auto"
|+ SCAN field options 

|-
! Index !! Option !! Index !! Option
|-
| 0 || <code>Passive</code> || 5 || <code>2 second</code>
|-
| 1 || <code>Event</code> || 6 || <code>1 second</code>
|-
| 2 || <code>I/O Intr</code> ||7 || <code>0.5 second</code>
|-
| 3 || <code>10 second</code> ||8 || <code>0.2 second</code>
|-
| 4 || <code>5 second</code> ||9 || <code>0.1 second</code>
|}

Note:
* If a record’s SCAN field is not specified in a database, it defaults to <code>Passive</code>.
* The '''SCAN''' field is common to ''all'' record types and the options can be selected by index instead of by string.
* <code>I/O Intr</code> processes a record when the driver gets a new value, but is device/driver dependent and isn’t guaranteed to work

=== Record link types ===

Much of the power of EPICS comes from the ability to link EPICS records together.
There are two link implementations: standard and dynamic.
* Dynamic links (pink) are found on ''userCalc'' ('''swait''' records) and ''scan'' ('''sscan''' records) screens; this type of link accept only a PV name. THe role of the process command (see below) is taken in a separate field.
* Standard links (purple) are found everywhere else.
''inlinkHelp'' and ''outlinkHelp'' are accessible from userTransform screens and scaler screens (for the later only).

==== Input Links ====

Input links read a value from another PV. It accepts a PV name followed by a process command (e.g. <code>NPP</code>) and an alarm command (e.g. <code>NMS</code>, ignore this).
Here are the possible process commands for an input link:
* <code>NPP</code> - Just read the value; don't process the target record (i.e. read a potentially stale value); this is a default command.
* <code>PP</code> - Process the target record and then read a value from it.
* <code>CA</code> - Read a value without causing processing to occur.
* <code>CP</code> - Process linking record every time the target record processes.
* <code>CPP</code> - Same as CP, but linking record process only if its SCAN field has the value <code>Passive</code>.
Note: ''inlinkHelp'' is accessible from userTransform screens.

==== Output Links ====

Output links write a value to another PV. It accepts a PV name followed by a process command (e.g. <code>NPP</code>) and an alarm command (e.g. <code>NMS</code>, ignore this).
Here are the possible process commands for an output link:
* <code>NPP</code> - Just write a value; don't process the target record; this is a default command.
* <code>PP</code> - Write a value and then process the target record. To be use if you want EPICS to trace processing (= wait for completion?)
* <code>CA</code> - Write a value; let the target record decide whether or not to process (as it would if you typed the value by hand); if the processing does occur, EPICS will not trace it.

Note:
* The default link process command, <code>NPP</code>, is '''almost never the desired behavior''' for output links. Change <code>NPP</code> to <code>PP</code> immediately and revert if problems are discovered.
* If the target PV is itself a link field, you must specify the command <code>CA</code>; if you want to wait for completion in this case, you must use either the '''sseq''' or the '''sCalcout''' record, select <code>CA</code> and set the link's '''WAIT''' field to <code>Wait</code>
* ''outlinkHelp'' is accessible from userTransform and scaler screens.

==== Forward Links ====

Forward link look like output links; they are purple on user displays, but they don't auto-populate with process & alarm commands.
Forward link causes the linked record to process when the record containing the forward link is processed
* Linked record is in the same IOC ('''PROC''' field for linked record is optional):
** <code>record_name.FLNK = linked_record_name</code>
** <code>record_name.FLNK = linked_record_name.PROC</code>
* Linked record is in a different IOC ('''PROC''' field for linked record is required):
** <code>record_name.FLNK = linked_record_name.PROC</code>

Note:
* Forwarding-linking records in the same IOC by only specifying the record name is an exception to the "'''VAL''' ''is the default field''" rule.
* When omitting the '''PROC''' field of a linked record in a different IOC, nothing will happen but you won't get any error message.
* Always specify the '''PROC''' field to avoid problems.

=== More about records ===

* Records vary significantly in complexity; one way to classify them would be (*):
** Simple records are associated with individual data types
*::;bi, bo, mbbi, mbbo, longin, longout, ai, ao, stringin, stringout, waveform
** Intermediate records often perform calculations or link multiple records
*::;calcout, scalcout, acalcout, swait, transform, fanout, dfanout
** Complex records are associated with devices or higher-level functionality
*::;motor, scaler, mca, sscan, table, digitel, vs
** Flexible records allow custom functionality to be implemented
*::;sub, aSub, luascript
(*) this is not an official classification, nor an exhaustive list of records. 
Note:
* areaDetector databases are composed of mostly simple record.

=== More about record fields ===

To find documentation for a specific field you need to know/find answers to the following questions:
* Is the field common to all or input/output record types?
* If not, does the record come from EPICS base or a different EPICS module (synApps)?
** The [https://epics.anl.gov/base/R7-0/7-docs/RecordReference.html EPICS base record reference documentation] is the best place to find information about record types that come from EPICS base. Field documentation appears in multiple locations:
*** Fields common to all record types
*** Fields common to input record types
*** Fields common to output record types
*** Record-specific documentation
** synApps record reference documentation
*** In each module's documentation directory
**** [https://github.com/epics-modules https://github.com/epics-modules]
**** <code>/APSshare/epics/synApps_X_Y/support</code>

== What is a databases? ==

A database is just a text file with a '''.db''' (or '''.template''') extension containing a collection of records that are related.

Databases are generalized with macros to allow multiple instances to be loaded (e.g. $(P)):
* Macros are variables containing strings to be substituted when the database is loaded.
* Macros without default values are required to load the database; there is no easy way to figure out what those are for a given database other than looking at the file itself.
* Macros required by the database should match the macros required by associated screens & autosave (.req) files.
* The P (prefix) macro usually includes a colon (:), which is the standard separator used in PV names.

=== How are databases loaded? ===

There are 2 ways to load databases which are strictly equivalent.

==== dbLoadRecords ====

<code>dbLoadRecords("$(IP)/db/ADAM_4018.db","P=kmp:,R=adam1,PORT=serial1,A=01")</code>
<code>dbLoadRecords("$(IP)/db/ADAM_4018.db","P=kmp:,R=adam2,PORT=serial1,A=02")</code>

==== dbLoadTemplate ====

<code>dbLoadTemplate("substitutions/ADAM_4018.substitutions","P=kmp:")</code>

where ADAM_4018.substitutions:

<code>file "$(IP)/db/ADAM_4018.db"
{
pattern
{R,PORT,A}
{adam1, serial1,01}
{adam2, serial1,02}
}</code>

== What is a device? ==

What is a device?

== How does one create an IOC? ==

There are multiple ways to create an IOC, but no one does it from scratch:

=== The EPICS base approach ===
It creates a bare bones IOC but requires more knowledge of the EPICS build system to get started:
::'''<code>makeBaseApp.pl</code>'''
* Must be run multiple times
* Creates IOC contents in the current directory
* synApps support needs to be added manually

=== The synApps approach ===

It is based on the ''xxx'' module and designed to give you a usable IOC with as little effort as possible:
::'''<code>mkioc</code>'''

* A single command leaves you with an IOC you can build:
::<code>/APSshare/bin/mkioc -n -f -s 6-2-1 kmp</code>
* Creates a top-level IOC directory in the current directory
* Common synApps support is included by default
* mkioc will ask you questions if you run it without options

Note: The BCDA convention is to name development IOCs using the developer's initials (here ''kmp'').

== IOC Layout ==

=== Overview ===

The IOC layout can be overwhelming and confusing to those who aren’t yet familiar with it. The EPICS build system creates many directories, making it harder to find the important directories.
Important files are often buried multiple directories down from the top-level. It is helpful to look at subsets of the IOC directory.

{| class="wikitable"
!rowspan="17"; style="color:blue"|kmp
|-
|style="color:red"|bin
|colspan="3"; style="color:red"|rhel8-x86_64
|-
|rowspan="2"; style="color:orange"|configure
|colspan="3"; style="color:grey"|O.Common
|-
|colspan="3"; style="color:grey"|O.rhel8-x86_64
|-
|colspan="4"; style="color:red"|db
|-
|colspan="4"; style="color:red"|dbd
|-
|rowspan="4"|iocBoot
|rowspan="4"; style="color:green"|iockmp
|autosave
|-
| style="color:green"|iocsh
|-
| style="color:blue"| softioc
| commands
|-
| style="color:green"| substitutions
|-
|rowspan="6"|kmpApp
|rowspan="2"; style="color:orange"|Db
|colspan="2"; style="color:grey"|O.Common
|-
|colspan="2"; style="color:grey"|O.rhel8-x86_64
|-
|rowspan="2"|op
|adl
|-
|ui
| autoconvert
|-
|rowspan="2"; style="color:orange"|src
|colspan="2"; style="color:grey"|O.Common
|-
|colspan="2"; style="color:grey"|O.rhel8-x86_64
|-
|style="color:red"|lib
|colspan="3"; style="color:red"|rhel8-x86_64
|}

=== Build configuration ===

Orange folders:
* '''<code>kmp/configure</code>''':
:contains the files that specify which versions of EPICS base and EPICS modules should be should be used when building the IOC.
* '''<code>kmp/kmpApp/Db</code>''':
:contains local databases (and associated autosave files) and protocol files used by the IOC.
* '''<code>kmp/kmpApp/src</code>''':
:contains the Makefile that determines all of the software that gets built into the IOC binary. ; local sequence programs and other EPICS support that needs to be compiled go here.

=== Intermediate build directories ===

Grey folders: 
The EPICS Build system will generate '''<code>O.</code>''' files (Common, EPICS HOST and TARGET architectures) in the configure, Db, and src directory. These directories are not needed after the IOC is built and can be removed with the '''<code>make clean</code>''' command. If they are not removed, they can be safely ignored. It is highly unlikely that you will ever need to look at any of the files in them.

=== Build products ===

The EPICS build system creates top level bin, db, dbd, and lib directories (red folders):
* '''<code>kmp/bin/rehl8-x86_64</code>''':
:contains the architecture-specific IOC binary (or munch file for VxWorks IOCs).
* '''<code>kmp/db</code>''':
:contains installed database files which may or may not be used, since many IOCs reference the databases in the Db directory instead.
* '''<code>kmp/dbd</code>''':
:contains a single database definition file to be loaded by the IOC.
* '''<code>kmp/lib/rehl8-x86_64</code>''':
:contains architecture-specific files for EPICS modules, but should be empty for the IOC. 
The build system creates an envPaths file in the startup directory, which contains the locations of the modules defined in the RELEASE file.

=== Run-time configuration ===

Green folders: 

The iockmp directory is referred to as the ''startup'' directory. It contains the '''<code>st.cmd</code>''' file, which is the runtime configuration for the IOC what means runtime?
The IOC’s config is often broken into separate '''<code>.iocsh</code>''' (or '''<code>.cmd</code>''') files so that support can be more easily commented out and to improve readability.
Substitutions and iocsh files usually reside in subdirectories to reduce clutter in the startup directory.

=== Start scripts ===

Blue folders: 
The top-level IOC directory contains scripts that start medm/caQtDM.
The softioc subdirectory of the startup dir contains a bash script which simplifies managing the IOC on Linux.
Windows IOCs: the startup directory contains batch files to start the IOC.

Note: The startup directory can reside elsewhere.

== How does support get into an IOC? ==

[[File:EPICS_basics_linking_diagrams.png|right|thumb|800px]]

 
=== Quick overview ===

* '''<code>configure/RELEASE</code>''':
** Defines the locations of EPICS base and EPICS modules
** Often includes other RELEASE files
* '''<code>kmpApp/src/Makefile</code>''':
** Specifies which database definitions (dbd) to include
** Specifies which libraries should be included in the IOC binary
** Specifies which local code should be included in the IOC binary
** Libraries will not be included unless an associated dbd file is included
** The order of the libraries is important (single-pass linker)
* '''<code>iocBoot/iockmp/st.cmd</code>''':
** The primary IOC config file that is sourced when the IOC is started
** Name can vary slightly based on how the IOC was created

=== Startup script <code>st.cmd</code> ===

Legacy IOCs often have monolithic '''<code>st.cmd</code>''' files. Newer IOCs source more '''<code>.iocsh</code>''' files from EPICS modules, which makes it easier to keep an IOC's config up-to-date.

< envPaths

dbLoadDatabase("../../dbd/iockmpLinux.dbd")
iockmpLinux_registerRecordDeviceDriver(pdbbase)

### Databases are loaded
### Drivers are initialized/configured here

iocInit

### Sequence programs & autosave are started here

dbl > dbl-all.txt

date

We can break it up into 3 functional zones:
* Zone 1: before '''<code>dbLoadDatabase</code>'''. Usually this is where env vars and paths are defined.
* Zone 2: after '''<code>dbLoadDatabase</code>''' & before '''<code>iocInit</code>'''. Databases can only be loaded here.
* Zone 3: after '''<code>iocInit</code>'''. This zone is mostly empty thanks to '''<code>doAfterIocInit</code>''' from the '''<code>std</code>''' module (sequence programs and autosave get initialized here, but the calls that do the initializing appear much earlier in the startup)

Note: PVs are available during '''<code>iocInit</code>''', even though the IOC is still starting up.

=== Autosave ===

'''<code>autosave</code>''' is a synApps module that provides seamless reboot functionality. It is mostly automatic when '''mkioc''' is used.
PVs are saved while the IOC is running and loaded during '''<code>iocInit</code>'''.
Autosaved values overwrite database defaults.

Databases with associated req files ('''<code>_settings.req</code>''') are automatically added to '''<code>built_*.req</code>''' by autosaveBuild
Multiple '''<code>.sav</code>''' files can be created:
* Default:
** '''<code>.sav</code>''' & '''<code>.savB</code>'''
** actually restored
* Periodic:
** '''<code>.sav0</code>''', '''<code>.sav1</code>''', '''<code>.sav2</code>''', …
** overwritten cyclically
* Dated:
** '''<code>.sav-230109-134502</code>'''
** written once at boot time

The 1st half of '''<code>common.iocsh</code>''' sets up autosave:
[https://github.com/epics-modules/xxx/blob/master/iocBoot/iocxxx/common.iocsh#L1-L46 https://github.com/epics-modules/xxx/blob/master/iocBoot/iocxxx/common.iocsh#L1-L46]

=== How to build & clean a Linux IOC ===

* Confirm the RHEL version
** <code> '''uname -r''' → 3.10.0-1160.62.1.'''el7'''.x86_64</code>
** <code> '''cat /etc/redhat-release''' → Red Hat Enterprise Linux Server release '''7.9'''</code>
* Set <code> EPICS_HOST_ARCH</code> environment variable
** bash: <code>'''export EPICS_HOST_ARCH=rhel7-x86_64'''</code>
** tcsh: <code>'''setenv EPICS_HOST_ARCH rhel7-x86_64'''</code>
* Initiate the build in the top-level IOC directory
** <code>'''make'''</code>

* Remove intermediate build dirs (for the current <code> EPICS_HOST_ARCH</code> )
** <code>'''make clean'''</code>
* Remove all intermediate build dirs and top-level build directories
** <code>'''make distclean'''</code>
* Show all make options
** <code>'''make help'''</code>

== Running an IOC ==
 

=== Use the script ===

==== Create an alias ====

Create an alias for the script (assuming PWD = IOC’s top-level dir)
:<code>$ '''alias kmp=${PWD}/iocBoot/iockmp/softioc/kmp.sh'''</code>
:<code>$ '''kmp''' </code>
:<code>Usage: kmp.sh {console|restart|run|start|caqtdm|medm|status|stop|usage}</code>

Noteworthy arguments:
* '''<code>run</code>''': Runs the IOC in the current terminal (doesn't return the command prompt). Useful for troubleshooting. Problematic for normal operation.
* '''<code>start</code>''': Runs the IOC in the background (returns the command prompt) using screen or procServ
* '''<code>status</code>''': Tells you if the IOC is running and refuses to start if it is

Notes:
* Use the full path to the IOC’s start script so the alias can be run from any directory (bash syntax shown).
* <code>EPICS_HOST_ARCH</code> needs to still be set otherwise the IOC will not start (unless the <code>EPICS_HOST_ARCH</code> is hard-coded in the start script)
* The ampersand is needed for synApps_6_2_1 IOCs, but it won’t be needed in the future because it was added to the '''<code>start_caQtDM_xxx</code>''' script.
* Bugs in the deployed copy of '''<code>xxx.sh</code>''' may prevent it from detecting an IOC is running.

==== Start caQtDM ====

:'''<code>kmp caqtdm &</code>'''

==== Start the IOC ====

Starts the IOC in the background (using screen or procServ):

:'''<code>kmp start</code>'''

IOCs are usually run in screen or procServ so that:
* the window in which they’re started can be closed
* multiple people can connect to them
* [procServ] allows other user accounts to restart an IOC
* [procServ] allows users to connect to Windows IOCs

==== Confirm the IOC is running ====

:'''<code>kmp status</code>'''
:<code>kmp is running (pid=1281733) in a screen session (pid=1281732)</code>

=== Connecting to an IOC’s shell ===

There are many ways to do it:

* Manually connect to an IOC running in screen from the IOC’s host
::<code>$ '''screen -x kmp'''</code>
::The -x option will connect even if someone is already connected to the screen session.
::The -r option only succeeds if no one is connected to the session.

* Connect to an IOC running in screen or procServ from the IOC’s host
::<code>$ '''kmp console'''</code>

* Connect to an IOC running in screen or procServ with logging '''from any host'''
::<code>$ '''/APSshare/bin/iocConsole.py iockmp'''</code>
::Note: iocConsole.py requires ssh access which beamline accounts don't have by default

* Manually connect to an IOC running in procServ from the IOC’s subnet
::<code>$ '''telnet s100bcda 53127'''</code>

=== Disconnecting from an IOC’s shell ===

* '''Close the terminal''' that is connected to the screen or procServ session
** Disconnects from both screen & procServ
** Kills an IOC running outside of screen & procServ
* '''<code>Ctrl+a</code>''', '''<code>Ctrl+d</code>'''
** Disconnects from both screen & procServ
** Kills an IOC running outside of screen & procServ
* '''<code>Ctrl+a, d</code>'''
** Only disconnects from screen

=== IOC shell commands ===

:'''<code>help [command]</code>''' - shows available commands or syntax of specified command
:'''<code>dbpr record_name [0-9]</code>''' - displays fields and values for a given record
:'''<code>dbl</code>''' - lists all records in an IOC
:'''<code>dbl record_type</code>''' - lists all records of a specific type in an IOC
:'''<code>dbl record_type "field_list"</code>''' - lists specified fields (space-separated list) of records of a specific type in an IOC
:'''<code>dbpf pv_name value</code>''' - change a PV’s value – analogous to caput
:'''<code>dbgf pv_name</code>''' - read a PV’s value – analogous to caget
:'''<code>epicsEnvShow</code>''' - prints environment variables
:'''<code>seqShow</code>''' - shows running sequence programs
:'''<code>seqStop</code>''' - stops a running sequence program

== External links ==

* [https://www.mediawiki.org/wiki/Help:Formatting Help formatting]
* [https://epics.anl.gov/base/R3-16/2-docs/AppDevGuide/DatabaseDefinition.html#x7-2630006.6 https://epics.anl.gov/]
* [https://epics.anl.gov/base/R7-0/7-docs/RecordReference.html EPICS base record reference documentation]

EPICS for Dummies

2023-03-08T02:25:31Z

Tguruswamy: /* How does support get into an IOC? */ add linking diagram

== What is EPICS? ==

EPICS (Experimental Physics and Industrial Control System) is a distributed control system, which means that the IOC (software) can run on a different computer as the user interfaces.

The next 5 sections are dedicated to the definition of basic EPICS components (IOC, PV, record, database, add devices?). To create your own EPICS support, you can skip directly to [[#How does one create an IOC?]].

== What is an IOC? ==

[[File:EPICS_basics_PV_diagram.png|right|thumb]]

IOC (Input/Output Controller) is a piece of software running on a computer:
* loads '''databases'''
* serves something called '''process variables''' (PVs) that you can read and write to
* used to run on VMEs, but nowadays it runs on anything.
* usually talks to hardware, but that isn’t a requirement.

IOCs are started with startup scripts. Those scripts tell the IOC what database files to load and where those files are. When the IOC is told to load a database, what it is doing is parsing that database file, identifying the records that are contained within, and then generating the actual epics PVs that a user will interact with.

'''Records''' and '''databases''' (lists of records) are configuration files describing the initial setup for '''PVs''', the '''IOC''' is the software that manages the behavior of PVs and communication to get/set their values.

== What is a PV? ==

PV (Process Variable) is a the field from a record:
''PV = record_name + . + field_name''

Most of the interactive parts of user displays have PVs associated with them.

== What is a record? ==

Records are the building blocks of EPICS:
* There is a lot of different types of record, and each type is designed to perform a specific function
* Records are customizable via fields that allow users to configure how they behave and monitor their state (see [[#More about record fields]] to know how to learn about record fields)
* Records do nothing until they are processed (see [[#SCAN, PROC & FLNK fields]])
* Records vary significantly in complexity (see [[#More about records]])
* It is rare for new records to be created, now that ''asyn'' device support exists add link to asyn?).

=== Where do records come from? ===

Typically, records at the APS comes from either:
* '''[https://epics.anl.gov/base/index.php EPICS base]''': main core of EPICS
* '''[https://www.aps.anl.gov/BCDA/synApps synApps]''': a collection of EPICS modules that are commonly used at the APS
A module is similar to a python package: it is an already written set of databases and associated behavior you can load into your IOC to do certain things, for example talk to specific hardware. An IOC can link to several modules.

{| class="wikitable" style="margin-left: 100px; margin-right:auto"
|+ Records in synApps modules 

|-
! Module !! Records !! Module !! Records
|-
| alive || alive || motor || motor
|-
| asyn || asyn || optics || table
|-
| busy || busy || scaler || scaler
|-
| calc || acalcout, scalcout, sseq, swait, transform || sscan || sscan
|-
| camac || camac || std || epid, timestamp, throttle
|-
| lua || luascript || vac || digitel, vs
|-
| mca || mca || vme || vme
|-
| allenBradley || ab1771IFE, ab1771N, ab1771IX, ...|| allenBradley || ..., ab1791, abDcm
|}

If a record isn't listed here, it probably comes from EPICS base.
The advantage to using synApps over a collection of modules one assembles on their own is that some amount of BCDA testing has been done.

=== What is a record field? ===

Record fields are controllable or informative properties of a record.

====Noteworthy fields====

*'''VAL''' - the default field (used if no field is specified in the vast majority of cases)
*'''SCAN''' - determines when a record processes
*'''NAME''' - the record name (helpful if records are aliased)
*'''DESC''' - record description
*'''RTYP''' - the record type
*'''DTYP''' - the device support type (not meaningful for all record types)
*'''DISP''' - disable puts (ca = channel access & pva = pv access) from outside the IOC (when non-zero); e.g. disable a motor from the motorx_all screen
*'''DISV/DISA/SDIS''' - disable record processing internally
**Record processing is disabled when '''DISA''' equals '''DISV''' (e.g. uses SDIS field to read in a value/mode, which will disable some control if a certain mode is active)
**'''DISA''' is set to the value of the PV in the '''SDIS''' field, if it isn't empty
*'''TPRO''' - trace processing: when set to 1 prints out all the records that get processed after the record is processed; used only for troubleshooting

Note: some of the fields of a record are actually record attributes (they return the same value for all instances of the record type; see [https://epics.anl.gov/base/R3-16/2-docs/AppDevGuide/DatabaseDefinition.html#x7-2630006.6 https://epics.anl.gov/])

==== SCAN, PROC & FLNK fields ====

Records do nothing until they are processed. Records can be processed multiple ways:
* Manually:
** Write 1 (or any non-zero value) to a record’s '''PROC''' field
**Forward link ('''FLNK''') from another record
* Periodically by setting the '''SCAN''' field to something other than <code>Passive</code>

{| class="wikitable" style="margin-left: 100px; margin-right:auto"
|+ SCAN field options 

|-
! Index !! Option !! Index !! Option
|-
| 0 || <code>Passive</code> || 5 || <code>2 second</code>
|-
| 1 || <code>Event</code> || 6 || <code>1 second</code>
|-
| 2 || <code>I/O Intr</code> ||7 || <code>0.5 second</code>
|-
| 3 || <code>10 second</code> ||8 || <code>0.2 second</code>
|-
| 4 || <code>5 second</code> ||9 || <code>0.1 second</code>
|}

Note:
* If a record’s SCAN field is not specified in a database, it defaults to <code>Passive</code>.
* The '''SCAN''' field is common to ''all'' record types and the options can be selected by index instead of by string.
* <code>I/O Intr</code> processes a record when the driver gets a new value, but is device/driver dependent and isn’t guaranteed to work

=== Record link types ===

Much of the power of EPICS comes from the ability to link EPICS records together.
There are two link implementations: standard and dynamic.
* Dynamic links (pink) are found on ''userCalc'' ('''swait''' records) and ''scan'' ('''sscan''' records) screens; this type of link accept only a PV name. THe role of the process command (see below) is taken in a separate field.
* Standard links (purple) are found everywhere else.
''inlinkHelp'' and ''outlinkHelp'' are accessible from userTransform screens and scaler screens (for the later only).

==== Input Links ====

Input links read a value from another PV. It accepts a PV name followed by a process command (e.g. <code>NPP</code>) and an alarm command (e.g. <code>NMS</code>, ignore this).
Here are the possible process commands for an input link:
* <code>NPP</code> - Just read the value; don't process the target record (i.e. read a potentially stale value); this is a default command.
* <code>PP</code> - Process the target record and then read a value from it.
* <code>CA</code> - Read a value without causing processing to occur.
* <code>CP</code> - Process linking record every time the target record processes.
* <code>CPP</code> - Same as CP, but linking record process only if its SCAN field has the value <code>Passive</code>.
Note: ''inlinkHelp'' is accessible from userTransform screens.

==== Output Links ====

Output links write a value to another PV. It accepts a PV name followed by a process command (e.g. <code>NPP</code>) and an alarm command (e.g. <code>NMS</code>, ignore this).
Here are the possible process commands for an output link:
* <code>NPP</code> - Just write a value; don't process the target record; this is a default command.
* <code>PP</code> - Write a value and then process the target record. To be use if you want EPICS to trace processing (= wait for completion?)
* <code>CA</code> - Write a value; let the target record decide whether or not to process (as it would if you typed the value by hand); if the processing does occur, EPICS will not trace it.

Note:
* The default link process command, <code>NPP</code>, is '''almost never the desired behavior''' for output links. Change <code>NPP</code> to <code>PP</code> immediately and revert if problems are discovered.
* If the target PV is itself a link field, you must specify the command <code>CA</code>; if you want to wait for completion in this case, you must use either the '''sseq''' or the '''sCalcout''' record, select <code>CA</code> and set the link's '''WAIT''' field to <code>Wait</code>
* ''outlinkHelp'' is accessible from userTransform and scaler screens.

==== Forward Links ====

Forward link look like output links; they are purple on user displays, but they don't auto-populate with process & alarm commands.
Forward link causes the linked record to process when the record containing the forward link is processed
* Linked record is in the same IOC ('''PROC''' field for linked record is optional):
** <code>record_name.FLNK = linked_record_name</code>
** <code>record_name.FLNK = linked_record_name.PROC</code>
* Linked record is in a different IOC ('''PROC''' field for linked record is required):
** <code>record_name.FLNK = linked_record_name.PROC</code>

Note:
* Forwarding-linking records in the same IOC by only specifying the record name is an exception to the "'''VAL''' ''is the default field''" rule.
* When omitting the '''PROC''' field of a linked record in a different IOC, nothing will happen but you won't get any error message.
* Always specify the '''PROC''' field to avoid problems.

=== More about records ===

* Records vary significantly in complexity; one way to classify them would be (*):
** Simple records are associated with individual data types
*::;bi, bo, mbbi, mbbo, longin, longout, ai, ao, stringin, stringout, waveform
** Intermediate records often perform calculations or link multiple records
*::;calcout, scalcout, acalcout, swait, transform, fanout, dfanout
** Complex records are associated with devices or higher-level functionality
*::;motor, scaler, mca, sscan, table, digitel, vs
** Flexible records allow custom functionality to be implemented
*::;sub, aSub, luascript
(*) this is not an official classification, nor an exhaustive list of records. 
Note:
* areaDetector databases are composed of mostly simple record.

=== More about record fields ===

To find documentation for a specific field you need to know/find answers to the following questions:
* Is the field common to all or input/output record types?
* If not, does the record come from EPICS base or a different EPICS module (synApps)?
** The [https://epics.anl.gov/base/R7-0/7-docs/RecordReference.html EPICS base record reference documentation] is the best place to find information about record types that come from EPICS base. Field documentation appears in multiple locations:
*** Fields common to all record types
*** Fields common to input record types
*** Fields common to output record types
*** Record-specific documentation
** synApps record reference documentation
*** In each module's documentation directory
**** [https://github.com/epics-modules https://github.com/epics-modules]
**** <code>/APSshare/epics/synApps_X_Y/support</code>

== What is a databases? ==

A database is just a text file with a '''.db''' (or '''.template''') extension containing a collection of records that are related.

Databases are generalized with macros to allow multiple instances to be loaded (e.g. $(P)):
* Macros are variables containing strings to be substituted when the database is loaded.
* Macros without default values are required to load the database; there is no easy way to figure out what those are for a given database other than looking at the file itself.
* Macros required by the database should match the macros required by associated screens & autosave (.req) files.
* The P (prefix) macro usually includes a colon (:), which is the standard separator used in PV names.

=== How are databases loaded? ===

There are 2 ways to load databases which are strictly equivalent.

==== dbLoadRecords ====

<code>dbLoadRecords("$(IP)/db/ADAM_4018.db","P=kmp:,R=adam1,PORT=serial1,A=01")</code>
<code>dbLoadRecords("$(IP)/db/ADAM_4018.db","P=kmp:,R=adam2,PORT=serial1,A=02")</code>

==== dbLoadTemplate ====

<code>dbLoadTemplate("substitutions/ADAM_4018.substitutions","P=kmp:")</code>

where ADAM_4018.substitutions:

<code>file "$(IP)/db/ADAM_4018.db"
{
pattern
{R,PORT,A}
{adam1, serial1,01}
{adam2, serial1,02}
}</code>

== What is a device? ==

What is a device?

== How does one create an IOC? ==

There are multiple ways to create an IOC, but no one does it from scratch:

=== The EPICS base approach ===
It creates a bare bones IOC but requires more knowledge of the EPICS build system to get started:
::'''<code>makeBaseApp.pl</code>'''
* Must be run multiple times
* Creates IOC contents in the current directory
* synApps support needs to be added manually

=== The synApps approach ===

It is based on the ''xxx'' module and designed to give you a usable IOC with as little effort as possible:
::'''<code>mkioc</code>'''

* A single command leaves you with an IOC you can build:
::<code>/APSshare/bin/mkioc -n -f -s 6-2-1 kmp</code>
* Creates a top-level IOC directory in the current directory
* Common synApps support is included by default
* mkioc will ask you questions if you run it without options

Note: The BCDA convention is to name development IOCs using the developer's initials (here ''kmp'').

== IOC Layout ==

=== Overview ===

The IOC layout can be overwhelming and confusing to those who aren’t yet familiar with it. The EPICS build system creates many directories, making it harder to find the important directories.
Important files are often buried multiple directories down from the top-level. It is helpful to look at subsets of the IOC directory.

{| class="wikitable"
!rowspan="17"; style="color:blue"|kmp
|-
|style="color:red"|bin
|colspan="3"; style="color:red"|rhel8-x86_64
|-
|rowspan="2"; style="color:orange"|configure
|colspan="3"; style="color:grey"|O.Common
|-
|colspan="3"; style="color:grey"|O.rhel8-x86_64
|-
|colspan="4"; style="color:red"|db
|-
|colspan="4"; style="color:red"|dbd
|-
|rowspan="4"|iocBoot
|rowspan="4"; style="color:green"|iockmp
|autosave
|-
| style="color:green"|iocsh
|-
| style="color:blue"| softioc
| commands
|-
| style="color:green"| substitutions
|-
|rowspan="6"|kmpApp
|rowspan="2"; style="color:orange"|Db
|colspan="2"; style="color:grey"|O.Common
|-
|colspan="2"; style="color:grey"|O.rhel8-x86_64
|-
|rowspan="2"|op
|adl
|-
|ui
| autoconvert
|-
|rowspan="2"; style="color:orange"|src
|colspan="2"; style="color:grey"|O.Common
|-
|colspan="2"; style="color:grey"|O.rhel8-x86_64
|-
|style="color:red"|lib
|colspan="3"; style="color:red"|rhel8-x86_64
|}

=== Build configuration ===

Orange folders:
* '''<code>kmp/configure</code>''':
:contains the files that specify which versions of EPICS base and EPICS modules should be should be used when building the IOC.
* '''<code>kmp/kmpApp/Db</code>''':
:contains local databases (and associated autosave files) and protocol files used by the IOC.
* '''<code>kmp/kmpApp/src</code>''':
:contains the Makefile that determines all of the software that gets built into the IOC binary. ; local sequence programs and other EPICS support that needs to be compiled go here.

=== Intermediate build directories ===

Grey folders: 
The EPICS Build system will generate '''<code>O.</code>''' files (Common, EPICS HOST and TARGET architectures) in the configure, Db, and src directory. These directories are not needed after the IOC is built and can be removed with the '''<code>make clean</code>''' command. If they are not removed, they can be safely ignored. It is highly unlikely that you will ever need to look at any of the files in them.

=== Build products ===

The EPICS build system creates top level bin, db, dbd, and lib directories (red folders):
* '''<code>kmp/bin/rehl8-x86_64</code>''':
:contains the architecture-specific IOC binary (or munch file for VxWorks IOCs).
* '''<code>kmp/db</code>''':
:contains installed database files which may or may not be used, since many IOCs reference the databases in the Db directory instead.
* '''<code>kmp/dbd</code>''':
:contains a single database definition file to be loaded by the IOC.
* '''<code>kmp/lib/rehl8-x86_64</code>''':
:contains architecture-specific files for EPICS modules, but should be empty for the IOC. 
The build system creates an envPaths file in the startup directory, which contains the locations of the modules defined in the RELEASE file.

=== Run-time configuration ===

Green folders: 

The iockmp directory is referred to as the ''startup'' directory. It contains the '''<code>st.cmd</code>''' file, which is the runtime configuration for the IOC what means runtime?
The IOC’s config is often broken into separate '''<code>.iocsh</code>''' (or '''<code>.cmd</code>''') files so that support can be more easily commented out and to improve readability.
Substitutions and iocsh files usually reside in subdirectories to reduce clutter in the startup directory.

=== Start scripts ===

Blue folders: 
The top-level IOC directory contains scripts that start medm/caQtDM.
The softioc subdirectory of the startup dir contains a bash script which simplifies managing the IOC on Linux.
Windows IOCs: the startup directory contains batch files to start the IOC.

Note: The startup directory can reside elsewhere.

== How does support get into an IOC? ==

[[File:EPICS_basics_linking_diagrams.png|right|thumb|800px]]

 
=== Quick overview ===

* '''<code>configure/RELEASE</code>''':
** Defines the locations of EPICS base and EPICS modules
** Often includes other RELEASE files
* '''<code>kmpApp/src/Makefile</code>''':
** Specifies which database definitions (dbd) to include
** Specifies which libraries should be included in the IOC binary
** Specifies which local code should be included in the IOC binary
** Libraries will not be included unless an associated dbd file is included
** The order of the libraries is important (single-pass linker)
* '''<code>iocBoot/iockmp/st.cmd</code>''':
** The primary IOC config file that is sourced when the IOC is started
** Name can vary slightly based on how the IOC was created

=== Startup script <code>st.cmd</code> ===

Legacy IOCs often have monolithic '''<code>st.cmd</code>''' files. Newer IOCs source more '''<code>.iocsh</code>''' files from EPICS modules, which makes it easier to keep an IOC's config up-to-date.

< envPaths

dbLoadDatabase("../../dbd/iockmpLinux.dbd")
iockmpLinux_registerRecordDeviceDriver(pdbbase)

### Databases are loaded
### Drivers are initialized/configured here

iocInit

### Sequence programs & autosave are started here

dbl > dbl-all.txt

date

We can break it up into 3 functional zones:
* Zone 1: before '''<code>dbLoadDatabase</code>'''. Usually this is where env vars and paths are defined.
* Zone 2: after '''<code>dbLoadDatabase</code>''' & before '''<code>iocInit</code>'''. Databases can only be loaded here.
* Zone 3: after '''<code>iocInit</code>'''. This zone is mostly empty thanks to '''<code>doAfterIocInit</code>''' from the '''<code>std</code>''' module (sequence programs and autosave get initialized here, but the calls that do the initializing appear much earlier in the startup)

Note: PVs are available during '''<code>iocInit</code>''', even though the IOC is still starting up.

=== Autosave ===

'''<code>autosave</code>''' is a synApps module that provides seamless reboot functionality. It is mostly automatic when '''mkioc''' is used.
PVs are saved while the IOC is running and loaded during '''<code>iocInit</code>'''.
Autosaved values overwrite database defaults.

Databases with associated req files ('''<code>_settings.req</code>''') are automatically added to '''<code>built_*.req</code>''' by autosaveBuild
Multiple '''<code>.sav</code>''' files can be created:
* Default:
** '''<code>.sav</code>''' & '''<code>.savB</code>'''
** actually restored
* Periodic:
** '''<code>.sav0</code>''', '''<code>.sav1</code>''', '''<code>.sav2</code>''', …
** overwritten cyclically
* Dated:
** '''<code>.sav-230109-134502</code>'''
** written once at boot time

The 1st half of '''<code>common.iocsh</code>''' sets up autosave:
[https://github.com/epics-modules/xxx/blob/master/iocBoot/iocxxx/common.iocsh#L1-L46 https://github.com/epics-modules/xxx/blob/master/iocBoot/iocxxx/common.iocsh#L1-L46]

=== How to build & clean a Linux IOC ===

* Confirm the RHEL version
** <code> '''uname -r''' → 3.10.0-1160.62.1.'''el7'''.x86_64</code>
** <code> '''cat /etc/redhat-release''' → Red Hat Enterprise Linux Server release '''7.9'''</code>
* Set <code> EPICS_HOST_ARCH</code> environment variable
** bash: <code>'''export EPICS_HOST_ARCH=rhel7-x86_64'''</code>
** tcsh: <code>'''setenv EPICS_HOST_ARCH rhel7-x86_64'''</code>
* Initiate the build in the top-level IOC directory
** <code>'''make'''</code>

* Remove intermediate build dirs (for the current <code> EPICS_HOST_ARCH</code> )
** <code>'''make clean'''</code>
* Remove all intermediate build dirs and top-level build directories
** <code>'''make distclean'''</code>
* Show all make options
** <code>'''make help'''</code>

== Running an IOC ==
 

=== Use the script ===

==== Create an alias ====

Create an alias for the script (assuming PWD = IOC’s top-level dir)
:<code>$ '''alias kmp=${PWD}/iocBoot/iockmp/softioc/kmp.sh'''</code>
:<code>$ '''kmp''' </code>
:<code>Usage: kmp.sh {console|restart|run|start|caqtdm|medm|status|stop|usage}</code>

Noteworthy arguments:
* '''<code>run</code>''': Runs the IOC in the current terminal (doesn't return the command prompt). Useful for troubleshooting. Problematic for normal operation.
* '''<code>start</code>''': Runs the IOC in the background (returns the command prompt) using screen or procServ
* '''<code>status</code>''': Tells you if the IOC is running and refuses to start if it is

Notes:
* Use the full path to the IOC’s start script so the alias can be run from any directory (bash syntax shown).
* <code>EPICS_HOST_ARCH</code> needs to still be set otherwise the IOC will not start (unless the <code>EPICS_HOST_ARCH</code> is hard-coded in the start script)
* The ampersand is needed for synApps_6_2_1 IOCs, but it won’t be needed in the future because it was added to the '''<code>start_caQtDM_xxx</code>''' script.
* Bugs in the deployed copy of '''<code>xxx.sh</code>''' may prevent it from detecting an IOC is running.

==== Start caQtDM ====

:'''<code>kmp caqtdm &</code>'''

==== Start the IOC ====

Starts the IOC in the background (using screen or procServ):

:'''<code>kmp start</code>'''

IOCs are usually run in screen or procServ so that:
* the window in which they’re started can be closed
* multiple people can connect to them
* [procServ] allows other user accounts to restart an IOC
* [procServ] allows users to connect to Windows IOCs

==== Confirm the IOC is running ====

:'''<code>kmp status</code>'''
:<code>kmp is running (pid=1281733) in a screen session (pid=1281732)</code>

=== Connecting to an IOC’s shell ===

There are many ways to do it:

* Manually connect to an IOC running in screen from the IOC’s host
::<code>$ '''screen -x kmp'''</code>
::The -x option will connect even if someone is already connected to the screen session.
::The -r option only succeeds if no one is connected to the session.

* Connect to an IOC running in screen or procServ from the IOC’s host
::<code>$ '''kmp console'''</code>

* Connect to an IOC running in screen or procServ with logging '''from any host'''
::<code>$ '''/APSshare/bin/iocConsole.py iockmp'''</code>
::Note: iocConsole.py requires ssh access which beamline accounts don't have by default

* Manually connect to an IOC running in procServ from the IOC’s subnet
::<code>$ '''telnet s100bcda 53127'''</code>

=== Disconnecting from an IOC’s shell ===

* '''Close the terminal''' that is connected to the screen or procServ session
** Disconnects from both screen & procServ
** Kills an IOC running outside of screen & procServ
* '''<code>Ctrl+a</code>''', '''<code>Ctrl+d</code>'''
** Disconnects from both screen & procServ
** Kills an IOC running outside of screen & procServ
* '''<code>Ctrl+a, d</code>'''
** Only disconnects from screen

=== IOC shell commands ===

:'''<code>help [command]</code>''' - shows available commands or syntax of specified command
:'''<code>dbpr record_name [0-9]</code>''' - displays fields and values for a given record
:'''<code>dbl</code>''' - lists all records in an IOC
:'''<code>dbl record_type</code>''' - lists all records of a specific type in an IOC
:'''<code>dbl record_type "field_list"</code>''' - lists specified fields (space-separated list) of records of a specific type in an IOC
:'''<code>dbpf pv_name value</code>''' - change a PV’s value – analogous to caput
:'''<code>dbgf pv_name</code>''' - read a PV’s value – analogous to caget
:'''<code>epicsEnvShow</code>''' - prints environment variables
:'''<code>seqShow</code>''' - shows running sequence programs
:'''<code>seqStop</code>''' - stops a running sequence program

== External links ==

* [https://www.mediawiki.org/wiki/Help:Formatting Help formatting]
* [https://epics.anl.gov/base/R3-16/2-docs/AppDevGuide/DatabaseDefinition.html#x7-2630006.6 https://epics.anl.gov/]
* [https://epics.anl.gov/base/R7-0/7-docs/RecordReference.html EPICS base record reference documentation]

EPICS for Dummies

2023-03-08T02:23:45Z

Tguruswamy: /* What is an IOC? */ add diagram

== What is EPICS? ==

EPICS (Experimental Physics and Industrial Control System) is a distributed control system, which means that the IOC (software) can run on a different computer as the user interfaces.

The next 5 sections are dedicated to the definition of basic EPICS components (IOC, PV, record, database, add devices?). To create your own EPICS support, you can skip directly to [[#How does one create an IOC?]].

== What is an IOC? ==

[[File:EPICS_basics_PV_diagram.png|right|thumb]]

IOC (Input/Output Controller) is a piece of software running on a computer:
* loads '''databases'''
* serves something called '''process variables''' (PVs) that you can read and write to
* used to run on VMEs, but nowadays it runs on anything.
* usually talks to hardware, but that isn’t a requirement.

IOCs are started with startup scripts. Those scripts tell the IOC what database files to load and where those files are. When the IOC is told to load a database, what it is doing is parsing that database file, identifying the records that are contained within, and then generating the actual epics PVs that a user will interact with.

'''Records''' and '''databases''' (lists of records) are configuration files describing the initial setup for '''PVs''', the '''IOC''' is the software that manages the behavior of PVs and communication to get/set their values.

== What is a PV? ==

PV (Process Variable) is a the field from a record:
''PV = record_name + . + field_name''

Most of the interactive parts of user displays have PVs associated with them.

== What is a record? ==

Records are the building blocks of EPICS:
* There is a lot of different types of record, and each type is designed to perform a specific function
* Records are customizable via fields that allow users to configure how they behave and monitor their state (see [[#More about record fields]] to know how to learn about record fields)
* Records do nothing until they are processed (see [[#SCAN, PROC & FLNK fields]])
* Records vary significantly in complexity (see [[#More about records]])
* It is rare for new records to be created, now that ''asyn'' device support exists add link to asyn?).

=== Where do records come from? ===

Typically, records at the APS comes from either:
* '''[https://epics.anl.gov/base/index.php EPICS base]''': main core of EPICS
* '''[https://www.aps.anl.gov/BCDA/synApps synApps]''': a collection of EPICS modules that are commonly used at the APS
A module is similar to a python package: it is an already written set of databases and associated behavior you can load into your IOC to do certain things, for example talk to specific hardware. An IOC can link to several modules.

{| class="wikitable" style="margin-left: 100px; margin-right:auto"
|+ Records in synApps modules 

|-
! Module !! Records !! Module !! Records
|-
| alive || alive || motor || motor
|-
| asyn || asyn || optics || table
|-
| busy || busy || scaler || scaler
|-
| calc || acalcout, scalcout, sseq, swait, transform || sscan || sscan
|-
| camac || camac || std || epid, timestamp, throttle
|-
| lua || luascript || vac || digitel, vs
|-
| mca || mca || vme || vme
|-
| allenBradley || ab1771IFE, ab1771N, ab1771IX, ...|| allenBradley || ..., ab1791, abDcm
|}

If a record isn't listed here, it probably comes from EPICS base.
The advantage to using synApps over a collection of modules one assembles on their own is that some amount of BCDA testing has been done.

=== What is a record field? ===

Record fields are controllable or informative properties of a record.

====Noteworthy fields====

*'''VAL''' - the default field (used if no field is specified in the vast majority of cases)
*'''SCAN''' - determines when a record processes
*'''NAME''' - the record name (helpful if records are aliased)
*'''DESC''' - record description
*'''RTYP''' - the record type
*'''DTYP''' - the device support type (not meaningful for all record types)
*'''DISP''' - disable puts (ca = channel access & pva = pv access) from outside the IOC (when non-zero); e.g. disable a motor from the motorx_all screen
*'''DISV/DISA/SDIS''' - disable record processing internally
**Record processing is disabled when '''DISA''' equals '''DISV''' (e.g. uses SDIS field to read in a value/mode, which will disable some control if a certain mode is active)
**'''DISA''' is set to the value of the PV in the '''SDIS''' field, if it isn't empty
*'''TPRO''' - trace processing: when set to 1 prints out all the records that get processed after the record is processed; used only for troubleshooting

Note: some of the fields of a record are actually record attributes (they return the same value for all instances of the record type; see [https://epics.anl.gov/base/R3-16/2-docs/AppDevGuide/DatabaseDefinition.html#x7-2630006.6 https://epics.anl.gov/])

==== SCAN, PROC & FLNK fields ====

Records do nothing until they are processed. Records can be processed multiple ways:
* Manually:
** Write 1 (or any non-zero value) to a record’s '''PROC''' field
**Forward link ('''FLNK''') from another record
* Periodically by setting the '''SCAN''' field to something other than <code>Passive</code>

{| class="wikitable" style="margin-left: 100px; margin-right:auto"
|+ SCAN field options 

|-
! Index !! Option !! Index !! Option
|-
| 0 || <code>Passive</code> || 5 || <code>2 second</code>
|-
| 1 || <code>Event</code> || 6 || <code>1 second</code>
|-
| 2 || <code>I/O Intr</code> ||7 || <code>0.5 second</code>
|-
| 3 || <code>10 second</code> ||8 || <code>0.2 second</code>
|-
| 4 || <code>5 second</code> ||9 || <code>0.1 second</code>
|}

Note:
* If a record’s SCAN field is not specified in a database, it defaults to <code>Passive</code>.
* The '''SCAN''' field is common to ''all'' record types and the options can be selected by index instead of by string.
* <code>I/O Intr</code> processes a record when the driver gets a new value, but is device/driver dependent and isn’t guaranteed to work

=== Record link types ===

Much of the power of EPICS comes from the ability to link EPICS records together.
There are two link implementations: standard and dynamic.
* Dynamic links (pink) are found on ''userCalc'' ('''swait''' records) and ''scan'' ('''sscan''' records) screens; this type of link accept only a PV name. THe role of the process command (see below) is taken in a separate field.
* Standard links (purple) are found everywhere else.
''inlinkHelp'' and ''outlinkHelp'' are accessible from userTransform screens and scaler screens (for the later only).

==== Input Links ====

Input links read a value from another PV. It accepts a PV name followed by a process command (e.g. <code>NPP</code>) and an alarm command (e.g. <code>NMS</code>, ignore this).
Here are the possible process commands for an input link:
* <code>NPP</code> - Just read the value; don't process the target record (i.e. read a potentially stale value); this is a default command.
* <code>PP</code> - Process the target record and then read a value from it.
* <code>CA</code> - Read a value without causing processing to occur.
* <code>CP</code> - Process linking record every time the target record processes.
* <code>CPP</code> - Same as CP, but linking record process only if its SCAN field has the value <code>Passive</code>.
Note: ''inlinkHelp'' is accessible from userTransform screens.

==== Output Links ====

Output links write a value to another PV. It accepts a PV name followed by a process command (e.g. <code>NPP</code>) and an alarm command (e.g. <code>NMS</code>, ignore this).
Here are the possible process commands for an output link:
* <code>NPP</code> - Just write a value; don't process the target record; this is a default command.
* <code>PP</code> - Write a value and then process the target record. To be use if you want EPICS to trace processing (= wait for completion?)
* <code>CA</code> - Write a value; let the target record decide whether or not to process (as it would if you typed the value by hand); if the processing does occur, EPICS will not trace it.

Note:
* The default link process command, <code>NPP</code>, is '''almost never the desired behavior''' for output links. Change <code>NPP</code> to <code>PP</code> immediately and revert if problems are discovered.
* If the target PV is itself a link field, you must specify the command <code>CA</code>; if you want to wait for completion in this case, you must use either the '''sseq''' or the '''sCalcout''' record, select <code>CA</code> and set the link's '''WAIT''' field to <code>Wait</code>
* ''outlinkHelp'' is accessible from userTransform and scaler screens.

==== Forward Links ====

Forward link look like output links; they are purple on user displays, but they don't auto-populate with process & alarm commands.
Forward link causes the linked record to process when the record containing the forward link is processed
* Linked record is in the same IOC ('''PROC''' field for linked record is optional):
** <code>record_name.FLNK = linked_record_name</code>
** <code>record_name.FLNK = linked_record_name.PROC</code>
* Linked record is in a different IOC ('''PROC''' field for linked record is required):
** <code>record_name.FLNK = linked_record_name.PROC</code>

Note:
* Forwarding-linking records in the same IOC by only specifying the record name is an exception to the "'''VAL''' ''is the default field''" rule.
* When omitting the '''PROC''' field of a linked record in a different IOC, nothing will happen but you won't get any error message.
* Always specify the '''PROC''' field to avoid problems.

=== More about records ===

* Records vary significantly in complexity; one way to classify them would be (*):
** Simple records are associated with individual data types
*::;bi, bo, mbbi, mbbo, longin, longout, ai, ao, stringin, stringout, waveform
** Intermediate records often perform calculations or link multiple records
*::;calcout, scalcout, acalcout, swait, transform, fanout, dfanout
** Complex records are associated with devices or higher-level functionality
*::;motor, scaler, mca, sscan, table, digitel, vs
** Flexible records allow custom functionality to be implemented
*::;sub, aSub, luascript
(*) this is not an official classification, nor an exhaustive list of records. 
Note:
* areaDetector databases are composed of mostly simple record.

=== More about record fields ===

To find documentation for a specific field you need to know/find answers to the following questions:
* Is the field common to all or input/output record types?
* If not, does the record come from EPICS base or a different EPICS module (synApps)?
** The [https://epics.anl.gov/base/R7-0/7-docs/RecordReference.html EPICS base record reference documentation] is the best place to find information about record types that come from EPICS base. Field documentation appears in multiple locations:
*** Fields common to all record types
*** Fields common to input record types
*** Fields common to output record types
*** Record-specific documentation
** synApps record reference documentation
*** In each module's documentation directory
**** [https://github.com/epics-modules https://github.com/epics-modules]
**** <code>/APSshare/epics/synApps_X_Y/support</code>

== What is a databases? ==

A database is just a text file with a '''.db''' (or '''.template''') extension containing a collection of records that are related.

Databases are generalized with macros to allow multiple instances to be loaded (e.g. $(P)):
* Macros are variables containing strings to be substituted when the database is loaded.
* Macros without default values are required to load the database; there is no easy way to figure out what those are for a given database other than looking at the file itself.
* Macros required by the database should match the macros required by associated screens & autosave (.req) files.
* The P (prefix) macro usually includes a colon (:), which is the standard separator used in PV names.

=== How are databases loaded? ===

There are 2 ways to load databases which are strictly equivalent.

==== dbLoadRecords ====

<code>dbLoadRecords("$(IP)/db/ADAM_4018.db","P=kmp:,R=adam1,PORT=serial1,A=01")</code>
<code>dbLoadRecords("$(IP)/db/ADAM_4018.db","P=kmp:,R=adam2,PORT=serial1,A=02")</code>

==== dbLoadTemplate ====

<code>dbLoadTemplate("substitutions/ADAM_4018.substitutions","P=kmp:")</code>

where ADAM_4018.substitutions:

<code>file "$(IP)/db/ADAM_4018.db"
{
pattern
{R,PORT,A}
{adam1, serial1,01}
{adam2, serial1,02}
}</code>

== What is a device? ==

What is a device?

== How does one create an IOC? ==

There are multiple ways to create an IOC, but no one does it from scratch:

=== The EPICS base approach ===
It creates a bare bones IOC but requires more knowledge of the EPICS build system to get started:
::'''<code>makeBaseApp.pl</code>'''
* Must be run multiple times
* Creates IOC contents in the current directory
* synApps support needs to be added manually

=== The synApps approach ===

It is based on the ''xxx'' module and designed to give you a usable IOC with as little effort as possible:
::'''<code>mkioc</code>'''

* A single command leaves you with an IOC you can build:
::<code>/APSshare/bin/mkioc -n -f -s 6-2-1 kmp</code>
* Creates a top-level IOC directory in the current directory
* Common synApps support is included by default
* mkioc will ask you questions if you run it without options

Note: The BCDA convention is to name development IOCs using the developer's initials (here ''kmp'').

== IOC Layout ==

=== Overview ===

The IOC layout can be overwhelming and confusing to those who aren’t yet familiar with it. The EPICS build system creates many directories, making it harder to find the important directories.
Important files are often buried multiple directories down from the top-level. It is helpful to look at subsets of the IOC directory.

{| class="wikitable"
!rowspan="17"; style="color:blue"|kmp
|-
|style="color:red"|bin
|colspan="3"; style="color:red"|rhel8-x86_64
|-
|rowspan="2"; style="color:orange"|configure
|colspan="3"; style="color:grey"|O.Common
|-
|colspan="3"; style="color:grey"|O.rhel8-x86_64
|-
|colspan="4"; style="color:red"|db
|-
|colspan="4"; style="color:red"|dbd
|-
|rowspan="4"|iocBoot
|rowspan="4"; style="color:green"|iockmp
|autosave
|-
| style="color:green"|iocsh
|-
| style="color:blue"| softioc
| commands
|-
| style="color:green"| substitutions
|-
|rowspan="6"|kmpApp
|rowspan="2"; style="color:orange"|Db
|colspan="2"; style="color:grey"|O.Common
|-
|colspan="2"; style="color:grey"|O.rhel8-x86_64
|-
|rowspan="2"|op
|adl
|-
|ui
| autoconvert
|-
|rowspan="2"; style="color:orange"|src
|colspan="2"; style="color:grey"|O.Common
|-
|colspan="2"; style="color:grey"|O.rhel8-x86_64
|-
|style="color:red"|lib
|colspan="3"; style="color:red"|rhel8-x86_64
|}

=== Build configuration ===

Orange folders:
* '''<code>kmp/configure</code>''':
:contains the files that specify which versions of EPICS base and EPICS modules should be should be used when building the IOC.
* '''<code>kmp/kmpApp/Db</code>''':
:contains local databases (and associated autosave files) and protocol files used by the IOC.
* '''<code>kmp/kmpApp/src</code>''':
:contains the Makefile that determines all of the software that gets built into the IOC binary. ; local sequence programs and other EPICS support that needs to be compiled go here.

=== Intermediate build directories ===

Grey folders: 
The EPICS Build system will generate '''<code>O.</code>''' files (Common, EPICS HOST and TARGET architectures) in the configure, Db, and src directory. These directories are not needed after the IOC is built and can be removed with the '''<code>make clean</code>''' command. If they are not removed, they can be safely ignored. It is highly unlikely that you will ever need to look at any of the files in them.

=== Build products ===

The EPICS build system creates top level bin, db, dbd, and lib directories (red folders):
* '''<code>kmp/bin/rehl8-x86_64</code>''':
:contains the architecture-specific IOC binary (or munch file for VxWorks IOCs).
* '''<code>kmp/db</code>''':
:contains installed database files which may or may not be used, since many IOCs reference the databases in the Db directory instead.
* '''<code>kmp/dbd</code>''':
:contains a single database definition file to be loaded by the IOC.
* '''<code>kmp/lib/rehl8-x86_64</code>''':
:contains architecture-specific files for EPICS modules, but should be empty for the IOC. 
The build system creates an envPaths file in the startup directory, which contains the locations of the modules defined in the RELEASE file.

=== Run-time configuration ===

Green folders: 

The iockmp directory is referred to as the ''startup'' directory. It contains the '''<code>st.cmd</code>''' file, which is the runtime configuration for the IOC what means runtime?
The IOC’s config is often broken into separate '''<code>.iocsh</code>''' (or '''<code>.cmd</code>''') files so that support can be more easily commented out and to improve readability.
Substitutions and iocsh files usually reside in subdirectories to reduce clutter in the startup directory.

=== Start scripts ===

Blue folders: 
The top-level IOC directory contains scripts that start medm/caQtDM.
The softioc subdirectory of the startup dir contains a bash script which simplifies managing the IOC on Linux.
Windows IOCs: the startup directory contains batch files to start the IOC.

Note: The startup directory can reside elsewhere.

== How does support get into an IOC? ==

 
=== Quick overview ===

* '''<code>configure/RELEASE</code>''':
** Defines the locations of EPICS base and EPICS modules
** Often includes other RELEASE files
* '''<code>kmpApp/src/Makefile</code>''':
** Specifies which database definitions (dbd) to include
** Specifies which libraries should be included in the IOC binary
** Specifies which local code should be included in the IOC binary
** Libraries will not be included unless an associated dbd file is included
** The order of the libraries is important (single-pass linker)
* '''<code>iocBoot/iockmp/st.cmd</code>''':
** The primary IOC config file that is sourced when the IOC is started
** Name can vary slightly based on how the IOC was created

=== Startup script <code>st.cmd</code> ===

Legacy IOCs often have monolithic '''<code>st.cmd</code>''' files. Newer IOCs source more '''<code>.iocsh</code>''' files from EPICS modules, which makes it easier to keep an IOC's config up-to-date.

< envPaths

dbLoadDatabase("../../dbd/iockmpLinux.dbd")
iockmpLinux_registerRecordDeviceDriver(pdbbase)

### Databases are loaded
### Drivers are initialized/configured here

iocInit

### Sequence programs & autosave are started here

dbl > dbl-all.txt

date

We can break it up into 3 functional zones:
* Zone 1: before '''<code>dbLoadDatabase</code>'''. Usually this is where env vars and paths are defined.
* Zone 2: after '''<code>dbLoadDatabase</code>''' & before '''<code>iocInit</code>'''. Databases can only be loaded here.
* Zone 3: after '''<code>iocInit</code>'''. This zone is mostly empty thanks to '''<code>doAfterIocInit</code>''' from the '''<code>std</code>''' module (sequence programs and autosave get initialized here, but the calls that do the initializing appear much earlier in the startup)

Note: PVs are available during '''<code>iocInit</code>''', even though the IOC is still starting up.

=== Autosave ===

'''<code>autosave</code>''' is a synApps module that provides seamless reboot functionality. It is mostly automatic when '''mkioc''' is used.
PVs are saved while the IOC is running and loaded during '''<code>iocInit</code>'''.
Autosaved values overwrite database defaults.

Databases with associated req files ('''<code>_settings.req</code>''') are automatically added to '''<code>built_*.req</code>''' by autosaveBuild
Multiple '''<code>.sav</code>''' files can be created:
* Default:
** '''<code>.sav</code>''' & '''<code>.savB</code>'''
** actually restored
* Periodic:
** '''<code>.sav0</code>''', '''<code>.sav1</code>''', '''<code>.sav2</code>''', …
** overwritten cyclically
* Dated:
** '''<code>.sav-230109-134502</code>'''
** written once at boot time

The 1st half of '''<code>common.iocsh</code>''' sets up autosave:
[https://github.com/epics-modules/xxx/blob/master/iocBoot/iocxxx/common.iocsh#L1-L46 https://github.com/epics-modules/xxx/blob/master/iocBoot/iocxxx/common.iocsh#L1-L46]

=== How to build & clean a Linux IOC ===

* Confirm the RHEL version
** <code> '''uname -r''' → 3.10.0-1160.62.1.'''el7'''.x86_64</code>
** <code> '''cat /etc/redhat-release''' → Red Hat Enterprise Linux Server release '''7.9'''</code>
* Set <code> EPICS_HOST_ARCH</code> environment variable
** bash: <code>'''export EPICS_HOST_ARCH=rhel7-x86_64'''</code>
** tcsh: <code>'''setenv EPICS_HOST_ARCH rhel7-x86_64'''</code>
* Initiate the build in the top-level IOC directory
** <code>'''make'''</code>

* Remove intermediate build dirs (for the current <code> EPICS_HOST_ARCH</code> )
** <code>'''make clean'''</code>
* Remove all intermediate build dirs and top-level build directories
** <code>'''make distclean'''</code>
* Show all make options
** <code>'''make help'''</code>

== Running an IOC ==
 

=== Use the script ===

==== Create an alias ====

Create an alias for the script (assuming PWD = IOC’s top-level dir)
:<code>$ '''alias kmp=${PWD}/iocBoot/iockmp/softioc/kmp.sh'''</code>
:<code>$ '''kmp''' </code>
:<code>Usage: kmp.sh {console|restart|run|start|caqtdm|medm|status|stop|usage}</code>

Noteworthy arguments:
* '''<code>run</code>''': Runs the IOC in the current terminal (doesn't return the command prompt). Useful for troubleshooting. Problematic for normal operation.
* '''<code>start</code>''': Runs the IOC in the background (returns the command prompt) using screen or procServ
* '''<code>status</code>''': Tells you if the IOC is running and refuses to start if it is

Notes:
* Use the full path to the IOC’s start script so the alias can be run from any directory (bash syntax shown).
* <code>EPICS_HOST_ARCH</code> needs to still be set otherwise the IOC will not start (unless the <code>EPICS_HOST_ARCH</code> is hard-coded in the start script)
* The ampersand is needed for synApps_6_2_1 IOCs, but it won’t be needed in the future because it was added to the '''<code>start_caQtDM_xxx</code>''' script.
* Bugs in the deployed copy of '''<code>xxx.sh</code>''' may prevent it from detecting an IOC is running.

==== Start caQtDM ====

:'''<code>kmp caqtdm &</code>'''

==== Start the IOC ====

Starts the IOC in the background (using screen or procServ):

:'''<code>kmp start</code>'''

IOCs are usually run in screen or procServ so that:
* the window in which they’re started can be closed
* multiple people can connect to them
* [procServ] allows other user accounts to restart an IOC
* [procServ] allows users to connect to Windows IOCs

==== Confirm the IOC is running ====

:'''<code>kmp status</code>'''
:<code>kmp is running (pid=1281733) in a screen session (pid=1281732)</code>

=== Connecting to an IOC’s shell ===

There are many ways to do it:

* Manually connect to an IOC running in screen from the IOC’s host
::<code>$ '''screen -x kmp'''</code>
::The -x option will connect even if someone is already connected to the screen session.
::The -r option only succeeds if no one is connected to the session.

* Connect to an IOC running in screen or procServ from the IOC’s host
::<code>$ '''kmp console'''</code>

* Connect to an IOC running in screen or procServ with logging '''from any host'''
::<code>$ '''/APSshare/bin/iocConsole.py iockmp'''</code>
::Note: iocConsole.py requires ssh access which beamline accounts don't have by default

* Manually connect to an IOC running in procServ from the IOC’s subnet
::<code>$ '''telnet s100bcda 53127'''</code>

=== Disconnecting from an IOC’s shell ===

* '''Close the terminal''' that is connected to the screen or procServ session
** Disconnects from both screen & procServ
** Kills an IOC running outside of screen & procServ
* '''<code>Ctrl+a</code>''', '''<code>Ctrl+d</code>'''
** Disconnects from both screen & procServ
** Kills an IOC running outside of screen & procServ
* '''<code>Ctrl+a, d</code>'''
** Only disconnects from screen

=== IOC shell commands ===

:'''<code>help [command]</code>''' - shows available commands or syntax of specified command
:'''<code>dbpr record_name [0-9]</code>''' - displays fields and values for a given record
:'''<code>dbl</code>''' - lists all records in an IOC
:'''<code>dbl record_type</code>''' - lists all records of a specific type in an IOC
:'''<code>dbl record_type "field_list"</code>''' - lists specified fields (space-separated list) of records of a specific type in an IOC
:'''<code>dbpf pv_name value</code>''' - change a PV’s value – analogous to caput
:'''<code>dbgf pv_name</code>''' - read a PV’s value – analogous to caget
:'''<code>epicsEnvShow</code>''' - prints environment variables
:'''<code>seqShow</code>''' - shows running sequence programs
:'''<code>seqStop</code>''' - stops a running sequence program

== External links ==

* [https://www.mediawiki.org/wiki/Help:Formatting Help formatting]
* [https://epics.anl.gov/base/R3-16/2-docs/AppDevGuide/DatabaseDefinition.html#x7-2630006.6 https://epics.anl.gov/]
* [https://epics.anl.gov/base/R7-0/7-docs/RecordReference.html EPICS base record reference documentation]

File:EPICS basics linking diagrams.png

2023-03-08T02:17:00Z

Tguruswamy: Diagram showing runtime and compile time links between IOC applications, modules, and EPICS base.

== Summary ==
Diagram showing runtime and compile time links between IOC applications, modules, and EPICS base.

File:EPICS basics PV diagram.png

2023-03-08T02:16:18Z

Tguruswamy: Diagram illustrating PVs, databases, records, and fields, as well as distinguishing between IOC applications, modules, and clients.

== Summary ==
Diagram illustrating PVs, databases, records, and fields, as well as distinguishing between IOC applications, modules, and clients.

Controls Software Documentation

2022-12-08T19:54:03Z

Tguruswamy: add link to troubleshooting page

Troubleshooting EPICS IOCs

2022-12-08T19:53:54Z

Tguruswamy: new page

Understanding the structure of a deployed EPICS IOC and environment, determining configuration, and parsing error messages.

== Linux-specific ==

=== Examine the environment with <code>env</code> ===

EPICS reads several "environment variables" during operation.
You can examine the current list of environment variables in a terminal (either bash or csh) using <code>env</code>.

To get the value of a particular environment variable, try
<code>env | grep VARIABLE</code> or <code>echo $VARIABLE </code>.

Particularly relevant ones are:

; PATH : list of folders which are searched for executable scripts and binaries. If you are seeing "command not found"-type errors, consider if $PATH is set correctly.
; EPICS_CA_ADDR_LIST : IP addresses that are contacted when searching for a CA (channel-access) variable. Important for any program which is accessing CA variables provided by a different program, possibly on a different host.
; EPICS_CA_AUTO_ADDR_LIST : (yes | no). Whether to also include the broadcast address for all configured network interfaces in the CA_ADDR_LIST. This automatically searches all computers on the same subnet for variables.
; EPICS_PVA_ADDR_LIST : As above, but for PVaccess variables.
; EPICS_PVA_AUTO_ADDR_LIST : As above, but for automatic use of broadcast addresses for PVaccess variables.

For example, when examining a GUI which should only connect to the channel access variables of a local instance of an IOC, you might expect to see a configuration like:

<pre>EPICS_CA_AUTO_ADDR_LIST=no
EPICS_CA_ADDR_LIST=127.0.0.255</pre>

=== Find library dependencies with ldd ===

Sometimes it is necessary to determine which versions of a shared library are linked into the IOC binary. On other occasions, an IOC binary will not run (typically, an immediate segfault) because a necessary linked library is missing.
<code>ldd</code> is a Linux tool which will list all linked dynamic (shared) libraries of a binary, to answer these questions. Execute <code>ldd -r</code> against the IOC '''binary''' (not startup script) and examine its output.

Example:

<pre>
$ cd ~/epics/synApps_6_2_1/ioc/IocSpinnaker/
$ ldd -r bin/rhel8-x86_64/SpinnakerApp
linux-vdso.so.1 (0x00007ffffb195000)
libADSpinnaker.so => /net/s100dserv/APSshare/epics/synApps_6_2_1/support/areaDetector-R3-12-1/ADSpinnaker/lib/rhel8-x86_64/libADSpinnaker.so (0x00007f149e6f9000)
libADGenICam.so => /net/s100dserv/APSshare/epics/synApps_6_2_1/support/areaDetector-R3-12-1/ADGenICam/lib/rhel8-x86_64/libADGenICam.so (0x00007f149e4e6000)
...
</pre>

The location of all found libraries will be listed in full, while any missing libraries will be noted:

<pre>
libfoo.so => not found
</pre>

=== View the full IOC output when running in screen ===

By default, IOCs built from the XXX template are controlled via a startup shell script:

<pre>ExampleIOC.sh (start|stop|status)</pre>

When you start an IOC this way, no output is immediately visible, and control returns to your terminal. It is possible to connect to the IOC console via

<pre>ExampleIOC.sh console</pre>

which opens a connection to <code>screen</code>, the Linux terminal multiplexer. Type <code>exit</code> at the IOC prompt to exit, or use <code>Ctrl-a Ctrl-d</code> (detach) to leave the IOC running in the background and return to the terminal. <code>Ctrl-a [</code> will change screen to scrollback-mode, where you will be able to examine the stored history of IOC output. Use <code>/</code> to search forward through the text, and <code>?</code> to search backwards, for example to find errors.

Controls Software Documentation

2022-11-07T21:05:34Z

Tguruswamy: /* synApps (support) */