How To Use Posix Thread Priority Scheduling under Linux

From EPICSWIKI
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

What is this about?

The linux scheduler supports the SCHED_FIFO scheduling policy defined by POSIX.1-2001. Threads scheduled with this "real-time" policy can be assigned a priority (under linux) in the range 1..99 with 99 representing the highest priority. Since ordinary, "non-real time" processes execute at priority 0 (nice(1) modifies a "dynamic priority" which only affects processes with real-time priority 0 and implements fair timesharing among such processes) SCHED_FIFO threads are always capable to preempt "ordinary processes" (these use the policy SCHED_OTHER).

Note, however, that unless the linux kernel is built with the RT_PREEMPT patch and measures are taken to lock the EPICS process in memory etc., no strictly deterministic latencies can be expected even under the SCHED_FIFO policy which thus is to be considered soft-real time.

How can I let epicsThreads use the SCHED_FIFO policy?

In order to let EPICS use the SCHED_FIFO real-time policy you first need to check that the following option is set to "YES" in epics-base/configure/CONFIG_SITE 

# Use POSIX thread priority scheduling (YES or NO)
USE_POSIX_THREAD_PRIORITY_SCHEDULING = YES

If you find that the current setting is "NO" then you need to rebuild EPICS base (make clean uninstall; make) after changing to 'YES'.

Since engaging the SCHED_FIFO policy gives any thread created under that policy a higher priority than any normal process'es using SCHED_FIFO requires special privileges. Under a reasonably recent linux equipped with the pam_security module it is not necessary to execute EPICS as root. The system administrator can define the maximum priority that may be used by specific users and/or groups in a file under /etc/security/limits.d/, see the manpage for limits.conf(5).

E.g., if a file /etc/security.limits.d/epics.conf is created with the following contents (it may be necessary for a user to log out and log back on in order to obtain the new privileges): 

someuser hard rtprio 20
someuser soft rtprio  0

then processes created by 'someuser' still have priority 0 by default (this is the 'soft' value) but 'someuser' may increase the resource limit up to 20 either from a running program (using the system call setrlimit(2)) or within a shell e.g., with bash's ulimit -r utility.

Note that a user process may set and/or lower its hard limit but never increase it. RTM.

Summary

You need to do three things:

  • Build EPICS base with USE_POSIX_THREAD_PRIORITY_SCHEDULING=YES
  • Make sure your EPICS application process has sufficient privileges to use SCHED_FIFO and the desired priority range.
  • Set the soft and hard RTPRIO limits for your EPICS application process to enable it to actually make use of the privilege. This step is required because it is usually a bad idea to run all processes created by a given user or group at real-time priorities.

End of the story?

Unfortunately, this is not the end of the story. Linux' sched_get_priority_max(2) system call which is used by EPICS to map EPICS priorities from/to linux/pthread priorities always reports the maximal supported priority (99) without considering any limits that the system enforces (as discussed above). This has the consequence (see bug #835138) that threads which are created with a desired priority that is greater than the resource limit eventually end up with the lowest priority 0.

Work-around

  • Either apply the bugfix (AFAIK the problem is still present in 3.14.12.1, may be fixed thereafter)
  • or make sure the RTPRIO limits are set to the maximally possible value (99)
Retrieved from "https://wiki-ext.aps.anl.gov/epics/index.php?title=How_To_Use_Posix_Thread_Priority_Scheduling_under_Linux&oldid=2939"