Difference between revisions of "V4 Design: Record Processing"

From EPICSWIKI
Line 1: Line 1:
= EPICS V4: Record Processing =
= EPICS V4: Record Processing =
June 22 2005  
July 06 2005  


----
----
Line 8: Line 8:
  </center>
  </center>


NOTE: Preliminary thoughts.
NOTE: The comments from Tim Mooney about my previous thoughts on V4
record processing show that what I proposed will not work.
This is a new proposal for V4 record processing.


This document describes how records are processed.
Lets first state some goals for linking to other records:


The record processing model is different than the V3 model.
* Input Links - This means input from sources outside the record.
** get current value
** process record then get value
** wait until next process then get value
* Output Links
** put value
** put value then process record
** put value then process record and wait for completion


The model consists of the following features:
For both input and output links Application Developer should be able to
specify the following options for asynchronous links:
* Allow simultaneous requests
* processon link at a time.


* provides synchronous and asynchronous record processing
This is done separately for input and for output links.
** provides callbacks when an asynchronous record completes processing
* provides alarms
* provides posting of modifications
 
The asynchronous processing model is designed so that both control
and data acquisition database applications can be implemented.
Data acquisition applications need a way to delay record processing
until linked records complete processing.
In V3 several special record types, in particular the wait record,
were created to do the waiting.
The V4 asynchronous processing model supports data acquisition applications
without requiring special record types.


----
----
<center>
<center>
== Synchronous vs Asynchronous processing ==
== Processing States ==
  </center>
  </center>


Instead of the V3 PROC field, V4 will have a field processState.
This field will have the states:
; idle
: record is not being processed
; baseInputActive
: iocRecord is waiting for input to be done
; baseInputDone
: iocRecord has completed all input
; usetInputActive
: user defined fields are active.
; userInputDone
: user defined fields are done.
; inputActive
: input link processing is active
; inputDone
: all input links have completed
; deviceActive
: device support is active
; deviceDone
: device support has completed
; active
: record support is waiting for something to complete
; activeDone
: record support has completed all record related processing
; outputActive
: record support is waiting for output links to complete
; outputDone
: all output links have completed
; baseOutputActive
: iocRecord is waiting for input to be done
; baseOutputDone
:  iocRecord has completed all output
; userOutputActive
: user defined fields are active.
; userOutputDone


Each record instance is either synchronous, i.e. it will not block during
Some comments:
record processing, or it is asynchronous, i.e. it can block.
* iocRecord is what is called dbCommon in V4. For V4 it will have associated record support that will handle all the fields in iocRecord including an array of forward links. It will handle record completion.
 
* Each state may have substates. For example if a record has multiple record specific input links that are to be processed sequentially, it can keep state describing which link is being processed.
*COMMENT: Permission to block could hugely simplify the sscan record code, for example, but I think there must be some sort of back door. All the records I know of that currently wait for completion (excepting those that do it via asyn device support calls) ''must'' do it in a way that permits them still to respond to writes to their fields, so their processing can be aborted -- or, in some cases, redirected -- by a client. - TimMooney
* Individual record types may have only a subset of the above states.
 
----
It is the responsibility of record support and associated link or device support
<center>
to declare if a record instance is asynchronous. A field in dbCommon
== dbProcess ==
states if the record is asynchronous.
</center>


If a record is synchronous than the record is processed via normal scan threads.
dbProcess does the following based on state:
For example if the record is periodically scanned than it is processed by
a periodic scan thread.


*COMMENT: Several of the existing records that wait for completion also process periodically while they are waiting, so the user can get in-progress data while acquisition (which may last for many minutes) is ongoing. The scaler record does this by having device support schedule wakeups which process the record; the motor and MCA records rely on an external periodically scanned record to tell them to read and display current data. - TimMooney
; idle
: call iocRecord:process, which will handle all the input fields in iocRecord.
; baseInputActive
: nothing
; baseInputDone
: starts a loop to call process for each user defined field
; usetInputActive
: nothing
; userInputDone
: If not done with user fields continue loop. When user fields are done calls RecordSupport:process
; inputActive
: nothing
; inputDone
: nothing. Let record support decide when it is done
; deviceActive
: nothing
; deviceDone
: nothing
; active
: call RecordSupport:process. It can do what it wants.
; activeDone
: nothing
; outputActive
: nothing
; outputDone
: call iocRecord:process
; baseOutputActive
: nothing
; baseOutputDone
: starts a loop to call process for each user defined field
; userOutputActive
: nothing
; userOutputDone
: If not done with user fields continue loop. When user fields are done post monitprs for record done. Set state to idle.


A pool of threads are provided to process asynchronous records.
Some comments:
The maximum number of threads in the pool is configurable.
* It has to be determined who/what calls dbProcess
 
* It has to be decided when to post monitors. Tim's comments show that monitors can be issued while a record is active. These are different than monitors at the end of record processing.
A pool has an associated queue of records to process.
When a process request is received it is assigned to a thread
with the shortest queue, ideally one with an empty queue.
 
An asynchronous record is allowed to block.
 
The rest of this section describes additional features related to
asynchronous processing.


=== Notification of Process Completion ===
=== Notification of Process Completion ===


The request to process an asynchronous record can specify a callback
The request to process a record can specify a callback
that is called when the record completes processing.
that is called when the record completes processing.
It is also possible to make a process callback request over the network.
It is also possible to make a process callback request over the network.


*COMMENT: The requesting record doesn't know if the record it's calling is synchronous or asynchronous, so I guess this implies the request to process ''any'' record is permitted to specify a completion callback.  What happens if the called record is synchronous, and a callback routine is specified? - TimMooney


=== Wait for intermediate results ===
A record instance has an associated list of epicsEvents.
Instead of blocking, link and or device support can add an epicsEvent
to the list. Record support can make a call to wait for all events
on the list before proceding.
As an example the calcOut record could do the following:
* call the link support for each input link. Any link support that makes an asynchrous request to obtain data adds an entry to the event list.
* after all the input link support has been called, record support waits for all the events to be signaled.
* record support does the calculation and puts the result in the value field.
* record support calls the support for the output link. If the output link makes an asynchronous call it adds a entry to the event list.
* record support waits for the event before completing record processing.
COMMENT:  This is '''cool!'''  The ability of an input link to trigger processing, wait for completion, and know that the data it receives are the results of completed processing would really simplify database development and open new ground for run-time programming.  In V3, to achieve this effect, the record whose completion is required must drive all listeners who need completion-qualified data, and this means that it must know who those listeners are and be reconfigured when those listeners change.  V3 is very awkward in this respect.  Worse, V3 requires clients that want a completion-qualified value to know what record must be processed to generate that value. This new ability of V4 would allow clients merely to ask for the value, and let the record they're asking figure out how to acquire it. - TimMooney
=== flavors of link support ===
Link support for database and/or channel access links
should support the following:
* get
** get value
** process first then get data
* put
** put only
** put and request processing
** put, request processing, and wait for completion
COMMENT: Two possibilities for CP links:
# In V3, a record with a CP link, or any other CA client that puts a monitor on a PV, has no way of determining whether the value it receives is the result of completed processing, or merely an intermediate posting of work in progress.  Also, if a monitored record processes and the result is the same value it had before processing started, the monitoring client typically doesn't get any indication that something has happened.  This is a problem if the reason the client put a monitor on the PV was to discover that the record had processed, or to trigger some processing that should only be performed on completion-qualified values.  It would be nice if CA clients, including records, could say "send me all values that are the results of completed processing (whether or not they differ from previously posted values) and send me only such values."
# In V3, processing that results from a CP link (or any CA client's monitor) is outside of the putNotify execution trace.  A workable kludge (the 'busy' record) permits clients (typically, SNL programs) to participate by allowing them to control the calling of recGblFwdLink() with a caput().  Whatever new completion-detection scheme is used in V4, some form of this capability will still be needed.
ENDCOMMENT - TimMooney
----
----
<center>
<center>
== Alarm Processing ==
== Alarm Processing ==
  </center>
  </center>

Revision as of 19:06, 6 July 2005

EPICS V4: Record Processing

July 06 2005

Overview

NOTE: The comments from Tim Mooney about my previous thoughts on V4 record processing show that what I proposed will not work. This is a new proposal for V4 record processing.

Lets first state some goals for linking to other records:

  • Input Links - This means input from sources outside the record.
    • get current value
    • process record then get value
    • wait until next process then get value
  • Output Links
    • put value
    • put value then process record
    • put value then process record and wait for completion

For both input and output links Application Developer should be able to specify the following options for asynchronous links:

  • Allow simultaneous requests
  • processon link at a time.

This is done separately for input and for output links.

Processing States

Instead of the V3 PROC field, V4 will have a field processState. This field will have the states:

idle
record is not being processed
baseInputActive
iocRecord is waiting for input to be done
baseInputDone
iocRecord has completed all input
usetInputActive
user defined fields are active.
userInputDone
user defined fields are done.
inputActive
input link processing is active
inputDone
all input links have completed
deviceActive
device support is active
deviceDone
device support has completed
active
record support is waiting for something to complete
activeDone
record support has completed all record related processing
outputActive
record support is waiting for output links to complete
outputDone
all output links have completed
baseOutputActive
iocRecord is waiting for input to be done
baseOutputDone
iocRecord has completed all output
userOutputActive
user defined fields are active.
userOutputDone

Some comments:

  • iocRecord is what is called dbCommon in V4. For V4 it will have associated record support that will handle all the fields in iocRecord including an array of forward links. It will handle record completion.
  • Each state may have substates. For example if a record has multiple record specific input links that are to be processed sequentially, it can keep state describing which link is being processed.
  • Individual record types may have only a subset of the above states.

dbProcess

dbProcess does the following based on state:

idle
call iocRecord:process, which will handle all the input fields in iocRecord.
baseInputActive
nothing
baseInputDone
starts a loop to call process for each user defined field
usetInputActive
nothing
userInputDone
If not done with user fields continue loop. When user fields are done calls RecordSupport:process
inputActive
nothing
inputDone
nothing. Let record support decide when it is done
deviceActive
nothing
deviceDone
nothing
active
call RecordSupport:process. It can do what it wants.
activeDone
nothing
outputActive
nothing
outputDone
call iocRecord:process
baseOutputActive
nothing
baseOutputDone
starts a loop to call process for each user defined field
userOutputActive
nothing
userOutputDone
If not done with user fields continue loop. When user fields are done post monitprs for record done. Set state to idle.

Some comments:

  • It has to be determined who/what calls dbProcess
  • It has to be decided when to post monitors. Tim's comments show that monitors can be issued while a record is active. These are different than monitors at the end of record processing.

Notification of Process Completion

The request to process a record can specify a callback that is called when the record completes processing. It is also possible to make a process callback request over the network.


Alarm Processing

Are changes needed from the V3 model?

Posting Modifications

Whenever code modifies a field that is acessable outside the record it must call ????

When record support wants the modifications to be made available to code that has set monitors on fields in the record it calls ???

Retrieved from "https://wiki-ext.aps.anl.gov/epics/index.php?title=V4_Design:_Record_Processing&oldid=460"