How To Use Posix Thread Priority Scheduling under Linux

What is this about?

The linux scheduler supports the SCHED_FIFO scheduling policy defined by POSIX.1-2001. Threads scheduled with this "real-time" policy can be assigned a priority (under linux) in the range 1..99 with 99 representing the highest priority. Since ordinary, "non-real time" processes execute at priority 0 (nice(1) modifies a "dynamic priority" which only affects processes with real-time priority 0 and implements fair timesharing among such processes) SCHED_FIFO threads are always capable to preempt "ordinary processes" (these use the policy SCHED_OTHER).

Note, however, that unless the linux kernel is built with the RT_PREEMPT patch and measures are taken to lock the EPICS process in memory etc., no strictly deterministic latencies can be expected even under the SCHED_FIFO policy which thus is to be considered soft-real time.

How can I let epicsThreads use the SCHED_FIFO policy?

In order to let EPICS use the SCHED_FIFO real-time policy you first need to check that the following option is set to "YES" in epics-base/configure/CONFIG_SITE

# Use POSIX thread priority scheduling (YES or NO)
USE_POSIX_THREAD_PRIORITY_SCHEDULING = YES

If you find that the current setting is "NO" then you need to rebuild EPICS base (make clean uninstall; make) after changing to 'YES'.

Since engaging the SCHED_FIFO policy gives any thread created under that policy a higher priority than any normal process'es using SCHED_FIFO requires special privileges. Under a reasonably recent linux equipped with the pam_security module it is not necessary to execute EPICS as root. The system administrator can define the maximum priority that may be used by specific users and/or groups in a file under /etc/security/limits.d/, see the manpage for limits.conf(5).

E.g., if a file /etc/security.limits.d/epics.conf is created with the following contents (it may be necessary for a user to log out and log back on in order to obtain the new privileges):

someuser hard rtprio 20
someuser soft rtprio  0

then processes created by 'someuser' still have priority 0 by default (this is the 'soft' value) but 'someuser' may increase the resource limit up to 20 either from a running program (using the system call setrlimit(2)) or within a shell e.g., with bash's ulimit -r utility.

Note that a user process may set and/or lower its hard limit but never increase it. RTM.

Summary

You need to do three things:

• Build EPICS base with USE_POSIX_THREAD_PRIORITY_SCHEDULING=YES
• Make sure your EPICS application process has sufficient privileges to use SCHED_FIFO and the desired priority range.
• Set the soft and hard RTPRIO limits for your EPICS application process to enable it to actually make use of the privilege. This step is required because it is usually a bad idea to run all processes created by a given user or group at real-time priorities.

End of the story?

Unfortunately, this is not the end of the story. Linux' sched_get_priority_max(2) system call which is used by EPICS to map EPICS priorities from/to linux/pthread priorities always reports the maximal supported priority (99) without considering any limits that the system enforces (as discussed above). This has the consequence (see bug #835138) that threads which are created with a desired priority that is greater than the resource limit eventually end up with the lowest priority 0.

Work-around

• Either apply the bugfix (AFAIK the problem is still present in 3.14.12.1, may be fixed thereafter)
• or make sure the RTPRIO limits are set to the maximally possible value (99)