V4 Design: Record Processing
EPICS V4: Record Processing
August 16 2005
NOTE: UNDER CONSTRUCTION
Overview
This version is based of feedback from the EPICS core developer's meeting at ANL/APS on July 11th through July 14th. It is also a companion to the V4 Design: dbdInterfaces wiki.
The V3 record processing semantics do not work well for data acquisition applications. An example of data acquisition is:
- Move sample to new position.
- Several motors may be involved
- While moving post monitors to show current position, etc.
- Sample data
- Wait until CA client has fetched the data before next move.
SynAPPS provides the Scan, motor, etc records which help with data processing applications. These records process as follows:
- Record does not complete processing until all inputs, outputs, etc done. Not completing means that they do not call recGblFwdLink until all proccessing is complete. Note that recGblFwdLink is what causes the V3 ioc to complete record processing.
- While waiting for asynchronous events these records have PACT=0 but keep state that shows it is still active
- Can be scanned and issue monitors while internal state is active
Leaving PACT 0 but not calling recGblFwdLink was not anticipated when the V3 record processing semantics were created. For V4 it must be posssible for a record to be processed while it is waiting for asynchronous processing to complete.
In V3, if an input link is asynchronous, Process Passive does not wait for asynchronous processing to complete before fetching data. This also complicated data acquisition applications.
V4 record processing is designed to make data acquisition easier. Both input and output links can wait for record processing. This must be done, however, without record support blocking. If blocking makes it easier to implement record support, then the support can spawn a separate thread that does block. Record support can communicate with the separate thread to decide when to complete record processing.
V4 Link Semantics
Lets first state some goals for linking to other records:
Instead of refering to forward links the name process links will be used. The features described next will apply to links to records in the same or in different IOCs, i.e. similar semantics will be available for both database and channel access links.
- Input Link from another record
- get current value independent of processing state.
- Ask the record to process and wait for completion before fetching value.
- wait until next time record processes then get value. No request is made to process the record.
- Output Link to another record
- put value without requesting that record be processed.
- put value then process record but don't wait for processing to complete.
- put value, process record, and wait for completion
- process link to another record
- request processing but do not wait for completion
- request processing and wait for completion
- all links to other records
- Allow simultaneous requests, i.e. process links in parallel
- process one link at a time, i.e. process links serially.
- Record is unlocked while waiting for links to complete.
V4 Processing States
Instead of the V3 PROC field, V4 will have a field processState. This field will have the states:
- idle
- record is not being processed
- inputActive
- waiting for input links to complete
- active
- record is waiting for completion of record specific activity
- outputActive
- waiting for output or process links to complete
- processDone
- process has completed.
NOTES:
- Each state may have substates. For example if a record has multiple record specific input links that are to be processed sequentially, it can keep state describing which link is being processed.
- Individual record types may have only a subset of the above states.
- V3 link instances had attributes like CA, CP, CPP. For V4 these should be something like:
- process - a request to process the record before gets or after puts.
- wait - a request to wait until the next time the record processes
- parallel - OK to allow other gets/puts to operate simultaneously.
V4 iocRecord
iocRecord is the V4 successor to the V3 dbCommon. For V4 it is a separate software component.
- FLNK (forward link) is replaced by an array of process links.
dbProcess
The functionality described in this section will be divided between iocCore and iocRecord. iocRecord is the V4 successor to the V3 dbCommon. For V4 it is a separate software component. An important note is that FLNK (forward link) is replaced by an array of process links.
processState idle
This means that a new request is being made to process the record. The first step is to prepare the record.
- set newSeverity to noAlarm and newStatus to empty string.
- Is this really what we want? Perhaps just set severity and status?
- What else?
The next step is to obtain inputs for all input links in iocRecord. This may involve dbProcess being called multiple times if wait is specified for one or more links.
After all input from iocRecord fields is complete, dbProcess calls the record support process with state = idle. Record support sets state to one of inputActive, active, outputActive, or done. Each of these is discussed in greater detail below.
When record support return with state done, the ouput and process links in iocRecord are handled. This again may take several calls to dbProcess. When all the output is processed the steps outlined in subsection done below are performed. After this the record again becomes idle and the record is ready to again start processing.
The remaining subsections provide more details about each state. Keep in mind that some actions are performed by dbProcess, some by iocRecord, some by record support, and some by link support.
processState inputActive
This state means that one or more input links to other records have not completed. Links can be processed in parallel, sequencially, or a combination of these. iocRecord handles the links in iocRecord and record support handles record specific input links. The support is responsible for implementing sequential or parallel link processing. It calls link support to do the actual I/O. The Link support methods provide options specifying if the linked record should be processed and provide a callback to call when link processing completes. It is up to iocRecord or record support to again call dbProcess when all input links complete.
If is OK for dbProcess to be called multiple times while input links are being processed. When iocRecord detects that all it's input links have completed, it sets state = active and returns. dbProcess then sets state = idle and calls record support so that it starts processing it's input links.
processState active
This state means that record support and associated device support are waiting for something to complete, e.g. for a motor to reach some position.
The record, however, can be processed so that it can sample and report intermediate data values, e.g. the current motor position. For example the record can be periodically scanned.
When record support detects that it is done, it sets the state to either outputActive or done.
processState outputActive
This state means that one or more output links to other records have not completed. Links can be processed in parallel, sequencially, or a combination of these. Record support handles the record specific output links and then iocRecord handles the output and process links in iocRecord. When record support detects that all it's output links are done, it returns with processState done. dbProcess then calls iocRecord which starts processing it's output and process links. Until it is done it sets processState to outputActive. When it is done it sets the state to done.
processState done
When both record support and iocRecord have reported that they are done, dbProcess does some final steps and then sets the state to idle.
The final steps are:
- If the request to process the record included a callback, the callback is called.
- A notification is issued that states that the record has completed
processing.
- The request to process a record can specify a callback that is called when the record completes processing. It a callback was specified, ,it is called.
- ???
Alarm Processing
Are changes needed from the V3 model?
Is it possible to do the following?
- When dbProcess is called with processState idle then the severity is set to noAlarm and the status to an empty string.
- Alarms can raised between the beginning of record processing and processState done. Like V3 if the new alarm has a greater severity than the current then the new alarm determines the severity and status.
get/put database fields
This section discusses how IOC database fields are accessed by channel access and also via database links.
The semantics differ depending on the DbfType. Note that the discussion
of array and struct is saved until last.
primitive types
This includes DbfBool, DbfOctet, DbfInt16, ... , DbfFloat64. These do not present any problems. The scalar value is transfered.
string
A string is just a UTF_8 encoded character string
DbfMenu
The index value is handled just like int16. Since the choices for a menu can't change, only the changes to the index can occur.
DbfEnum
The index value is handled just like DbfInt16.
The choices are handled as an array of strings. See array below for details.
DbfLink and DbfDevice
Both of these have the following:
- choiceName A string that selects the link or device support.
- dataStruct Each support has an associated struct for configuration.
How should the be made available? Perhaps:
The field appears as a string with the following syntax:
choiceName(structAssignmentList)
where structAssignmentList has the same syntax as defined in "V4 DB Record Instance Syntax"
MDArray
TBD
timeStamp
This is just transfered a a struct with two fields:
- int64 secondsSinceEpoch
- int32 nanoSeconds
struct
The database definition syntax allow a struct to be composed of fields of any DbfType. How should this be made available to clients?
The individual fields can be accessed directly so this discussion only involves access to the entire structure atomically.
Perhaps two forms of access should be made available:
- Access by clients that have knowledge of the structure
- Access by clients that do not know about the structure
structure access
This access will only be implemented if all fields types are either primitive or string. In this case the data is transfered as a sequence of field values.
non-structure access
The structure value is presented as a string that has the same syntax as structAssignmentList as defined in "V4 DB Record Instance Syntax"
array
There are two major issues:
- memory
- data type
memory
Database Access is implemented so that record/link/device support can transfer an array in segments. Two example are:
- If the array is a circular buffer, it is presented in two segements
- If the array is stored in hardware, e.g. a transient recorder, the array can be read from the hardware and passed to the client in segments.
data types
The database definition syntax allow an array to be any of the basic types, i.e. DbfBool,...,DbfFloat64, DbfString, DbfArray, DbfStruct.
For primitive types, the data should be passed as a sequence of scalar values. Question? Should it be possible to pass a subarray, i.e. offset, length?
For DbfString it should also be possible to pass that array as a sequence of string values.
What about DbfArray? Perhaps this should not be supported?
What about DbfStruct?
Associated Data
The typical example is a client that asks for timeStamp, severity, status, and data.
Perhaps these can be handled as four synchronized requests to the same record.
How is this implemented?
- What does dbProcess do?
- How does a client make the request
Posting Modifications
The document "V4 Design: dbdIterfaces" provides a design that allows dbAccess to detect all changes to fields of an IOC record.
Hopefully dbAccess can handle all posting on monitors without requiring any help from code that modifies database fields. Instead, when dbAccess detects a modification it handles the monitors.
The following is the initial attempt to describe how monitors are handled.
The following definitions are used:
- monitor - A client has asked to be notified when a field value changes.
- post - Create storage, copy the current value of the field into the storage, and pass the storage to the monitor subsystem.
Database fields are either
- static - These fields change only because something external to the record modifies the field.
- dynamic - These are fields that change because of record processing.
Each type has different semantics for monitors.
static fields
The semantics differ depending on the DbfType.
primitive types
This includes DbfBool, DbfOctet, DbfInt16, ..., DbfFloat64 When a static primitive field is modified, a monitor is immediately posted.
DbfMenu
Whenever the index is modified, a monitor is immediately posted.
Since the choices for a menu can't change, only the changes to the index can occur.
DbfEnum
Whenever the index is modified, a monitor is immediately posted.
What about when a choice changes?
- Should only the choice that changes be posted? If so what is the syntax?
- Should the complete new set of choices be posted?
- Should the user just be notified that one or more choices changed and the user must issue a new read request?
DbfLink and DbfDevice
Both of these have the following:
- choiceName A string that selects the link or device support.
- dataStruct Each support has an associated struct for configuration.
What should be made available?
string
When a static string field is modified, a monitor is immediately posted.
Is this correct? For long strings it could cause excessive amounts of memory to be consumed. Probably not a big problem.
array
Now memory becomes a big issue. Perhaps the following semantics could be used.
- The array is posted when the value changes.
- When the value changes, the client is notified and must issue a read request to retrieve the data.
Should both or only one be supported?
If both who decides which semantics to use?
- client?
- database developer?
- ???
struct
Since structures can contain fields of any DbfType
dynamic fields
For each dynamic database field, dbAccess keeps an associated bit that is set to 0 when the record starts processing and is set to 1 whenever the field is modified.
Whenever record support returns with processState active or iocRecord returns with processState done, then the following is done:
- The monitor subsystem is notified about the beginning of synchronized monitors.
- Each field with the associated bit set to 1 is posted.
- The associated bit is set back to 0.
- Since status, severity, and timeStamp are dynamic fields in iocRecord, they follow the same logic.
- The monitor subsystem is notified about the end of synchronized monitors.
Separate discussions are required for the following: 1) primitive fields, 2) string fields, 3) structured fields, and 4) array fields. Each is discussed in a separate subsection.
primitive fields
No problems.
string fields
Memory is an issue. Should there be a way to postpone posting until the monitor is ready to retrieve the value? If so who decides?
structured fields
Memory is again an issue. Structure with only primitive fields should be posted immediately but perhaps complicated structures should not be. Who decides? How?
array fields
It should be possible to post small arrays of primitive type. Large arrays should not be posted until monitor system is ready to accept the data. It should also be possible to just notify the client that the array changed but the client must issue a read request to retrieve the data.
