V4 Design: Record Processing

EPICS V4: Record Processing

August 16 2005

---

Overview

This version is based on feedback from the EPICS core developer's meeting at ANL/APS on July 11th through July 14th. It is also a companion to the V4 Design: dbdInterfaces wiki.

The V3 record processing semantics do not work well for data acquisition applications. An example of data acquisition is:

• Move sample to new position.
  • Several motors may be involved
  • While moving post monitors to show current position, etc.
• Sample data
• Wait until CA client has fetched the data before next move.

SynAPPS provides the Scan, motor, etc records which help with data processing applications. These records process as follows:

• Record does not complete processing until all inputs, outputs, etc done. Not completing means that they do not call recGblFwdLink until all proccessing is complete. Note that recGblFwdLink is what causes the V3 ioc to complete record processing.
• While waiting for asynchronous events these records have PACT=0 but keep state that shows it is still active
• Can be scanned and issue monitors while internal state is active

Leaving PACT 0 but not calling recGblFwdLink was not anticipated when the V3 record processing semantics were created. For V4 it must be posssible for a record to be processed while it is waiting for asynchronous processing to complete.

In V3, if an input link is asynchronous, Process Passive does not wait for asynchronous processing to complete before fetching data. This also complicated data acquisition applications.

V4 record processing is designed to make data acquisition easier. Both input and output links can wait for record processing. This must be done, however, without record support blocking. If blocking makes it easier to implement record support, then the support can spawn a separate thread that does block. Record support can communicate with the separate thread to decide when to complete record processing.

---

V4 Link Semantics

Lets first state some goals for linking to other records:

Instead of refering "forward link" the name "process link" will be used. The features described next apply to both database and channel access links.

• Input Link from another record
  • get current value independent of processing state.
  • Ask the record to process and wait for completion before fetching value.
  • wait until next time record processes then get value. No request is made to process the record.
• Output Link to another record
  • put value without requesting that record be processed.
  • put value then process record but don't wait for processing to complete.
  • put value, process record, and wait for completion
• process link to another record
  • request processing but do not wait for completion
  • request processing and wait for completion
• all links to other records
  • Allow simultaneous requests, i.e. process links in parallel
  • process one link at a time, i.e. process links serially.
  • Record is unlocked while waiting for links to complete.

---

V4 Processing States

Instead of the V3 PROC field, V4 will have a field processState. This field has the states:

idle

record is not being processed

inputActive

waiting for input links to complete

active

record is waiting for completion of record specific activity

outputActive

waiting for output and process links to complete

done

process has completed.

NOTES:

• Each state may have substates. For example if a record has multiple input links that are to be processed sequentially, it can keep state describing which link is being processed.
• Individual record types may use only a subset of the above states.
• V3 link instances had attributes like CA, CP, CPP. For V4 these should be something like:
  • process - a request to process the record before gets or after puts.
  • wait - a request to wait until it has processed all it's input links
  • parallel - OK to allow simultaneous gets or puts

---

dbProcess

The functionality described in this section will be divided between iocCore and iocRecord. iocRecord is the V4 successor to the V3 dbCommon. For V4 it is a separate software component. An important note is that FLNK (forward link) is replaced by an array of process links.

dbProcess implements the following:

• If state is idle prepare record for processing and call iocRecord
• While state is inputActive call iocRecord
• Set state to idle and call record support
• While state is inputActive call record support
• While state is active call record support
• While stated is outputActive call record support
• Set state to done and call iocRecord
• While stated is outputActive call iocRecord
• State should be done. Perform final processing and set state idle

The remaining subsections provide more details about each state. Keep in mind that some actions are performed by dbProcess, some by iocRecord, some by record support, and some by link support.

processState idle

If dbProcess is called with processState idle then it:

• initializes severity
  • if it is in alarm set modify bit
  • sets severity to noAlarm
• initializes status
  • If it is not an empty set the modify bit
  • sets status to an empty string
• Other stuff?????
• calls iocReport with state idle

When iocRecord is called with state idle it:

• starts processing it's input links. Details are given below.
• returns with state one of the following:
  • inputActive - dbProcess will continually call iocRecord.
  • active - dbProcess will set state to idle and call record support.

When record support is called with state idle it:

• starts processing it's input links. Details are given below.
• returns with state one of the following:
  • inputActive - dbProcess will continually call record support
  • active - dbProcess will continually call record support
  • outputActive - dbProcess will continually call record support
  • done - dbProcess will call iocRecord to process it's output links

processState inputActive

This state means that one or more input links to other records have not completed. Links can be processed in parallel, sequentially, or a combination of these. iocRecord handles the links in iocRecord and record support handles record specific input links. The support is responsible for implementing sequential or parallel link processing. It calls link support to do the actual I/O. The Link support methods provide options specifying if the linked record should be processed and provide a callback to call when link processing completes. It is up to iocRecord or record support to again call dbProcess when all input links complete.

If is OK for dbProcess to be called multiple times while input links are being processed. When iocRecord detects that all it's input links have completed, it sets state = active and returns. dbProcess then sets state = idle and calls record support so that it starts processing it's input links.

processState active

This state means that record support and associated device support are waiting for something to complete, e.g. for a motor to reach some position.

The record, however, can be processed so that it can report intermediate data values, e.g. the current motor position. For example the record can be periodically scanned.

When record support detects that it is done, it sets the state to either outputActive or done.

While the state is active dbProcess will post dynamic monitors.

processState outputActive

This state means that one or more output links to other records have not completed. Links can be processed in parallel, sequencially, or a combination of these. Record support handles the record specific output links and then iocRecord handles the output and process links in iocRecord. When record support detects that all it's output links are done, it returns with processState done. dbProcess then calls iocRecord which starts processing it's output and process links. Until it is done it sets processState to outputActive. When it is done it sets the state to done.

processState done

When both record support and iocRecord have reported that they are done, dbProcess does some final steps and then sets the state to idle.

The final steps are:

• If the request to process the record included a callback, the callback is called.
• Dynamic monitors are posted
• ???
• The state is set to idle.

---

Alarm Processing

Note the changes since V3

• When the record begins processing severity is set to noAlarm and status to an empty string.
• recGblResetSeverity no longer exists.
• nsta and nsev no longer exist. Instead if the record was in alarm when processing begins the modify bits for status and severity are set.
• While the processState is active the status and/or severity may change. Any dynamic monitors will have the latest values of status and severity.

---

get/put database fields

This section discusses how IOC database fields are accessed by channel access and also via database links.

The semantics differ depending on the DbfType.

primitive types

This includes DbfBool, DbfOctet, DbfInt16, ... , DbfFloat64. These do not present any problems. The scalar value is transfered.

string

A string is just a UTF_8 encoded character string

DbfMenu

The index value is handled just like DbfInt16. Since the choices for a menu can't change, only the changes to the index can occur.

DbfEnum

The index value is handled just like DbfInt16.

The choices are handled as an array of strings. See array below for details.

DbfLink and DbfDevice

Both of these have the following:

• choiceName A string that selects the link or device support.
• dataStruct Each support has an associated struct for configuration.

How should the be made available? Perhaps:

The field appears as a string with the following syntax:

choiceName(structAssignmentList)

where structAssignmentList has the same syntax as defined in "V4 DB Record Instance Syntax"

MDArray

TBD

timeStamp

This is just transfered a struct with two fields:

• int64 secondsSinceEpoch
• int32 nanoSeconds

struct

The database definition syntax allow a struct to be composed of fields of any DbfType. How should this be made available to clients?

The individual fields can be accessed directly so this discussion only involves access to the entire structure atomically.

One possibility is to just make each field a member of a synchronous set of data.

Another possibility is to make the following two forms of access available.

1. Access by clients that have knowledge of the structure -
   • This access will only be implemented if all fields types are either primitive or string.
   • In this case the data is transfered as a sequence of field values.
2. Access by clients that do not know about the structure -
   • The structure value is presented as a string that has the structAssignmentList syntax defined in "V4 DB Record Instance Syntax"

array

• segmented memory - Database Access is implemented so that record/link/device support can transfer an array in segments. Examples:
  • If the array is a circular buffer, it is presented in two segements
  • If the array is stored in hardware, e.g. a transient recorder, the array can be read from the hardware and passed to the client in segments.
• data type - The database definition syntax allow an array to be any of the basic types, i.e. DbfBool,...,DbfFloat64, DbfString, DbfArray, DbfStruct.
  • For primitive types, the data should be passed as a sequence of scalar values. Question? Should it be possible to pass a subarray, i.e. offset, length?
  • For DbfString it should also be possible to pass that array as a sequence of string values.
  • What about DbfArray? Perhaps this should not be supported?
  • What about DbfStruct?

Associated Data

The typical example is a client that asks for timeStamp, severity, status, and data.

Perhaps these can be handled as four synchronized requests to the same record.

How is this implemented?

• What does dbProcess do?
• How does a client make the request

---

Posting Modifications

The document "V4 Design: dbdIterfaces" provides a design that allows dbAccess to detect all changes to fields of an IOC record.

Hopefully dbAccess can handle all posting on monitors without requiring any help from the code that modifies database fields.

The following is the initial attempt to describe how monitors are handled.

The following definitions are used:

• monitor - A client has asked to be notified when a field value changes.
• post - Create storage, copy the current value of the field into the storage, and pass the storage to the monitor subsystem.
• static database field - A field that changes only because something external to the record modifies the field.
• dynamic database field - A field that changes because of record processing.

Static and dynamic fields have different semantics for monitors.

static fields

• primitive types - This includes DbfBool, DbfOctet, DbfInt16, ..., DbfFloat64. When a static primitive field is modified, a monitor is immediately posted.
• DbfMenu - Whenever the index is modified, a monitor is immediately posted. Since the choices for a menu can't change, only the changes to the index can occur.
• DbfEnum - Whenever the index is modified, a monitor is immediately posted. What about when a choice changes?
  • Should only the choice that changes be posted? If so what is the syntax?
  • Should the complete new set of choices be posted?
  • Should the user just be notified that one or more choices changed and the user must issue a new read request?
• DbfLink and DbfDevice
  • Both of these have the following:
    • choiceName A string that selects the link or device support.
    • dataStruct Each support has an associated struct for configuration.
  • What should be made available?
    • Perhaps just notification that a change has occcured. Client must issue new read request
• string - Perhaps
  • When a static string field is modified, a monitor is immediately posted.
  • Is this correct? For long strings it could cause excessive amounts of memory to be consumed. Probably not a big problem.
• array - Memory becomes a big issue.
  • Perhaps the following semantics could be used.
    • The array is posted when the value changes.
    • When the value changes, the client is notified and must issue a read request to retrieve the data.
  • Should both or only one be supported? If both who decides which semantics to use?
    • client?
    • database developer?
    • ???
• struct -
  • Since structures can contain fields of any DbfType what to do???
    • At the minumum restrict what types of struct support monitors

dynamic fields

For each dynamic database field, dbAccess keeps a modify bit that is set to 0 when the record starts processing and is set to 1 whenever the field is modified.

Whenever record support returns with processState active or iocRecord returns with processState done, then the following is done:

• The monitor subsystem is notified about the beginning of synchronized monitors.
• Each field with the associated bit set to 1 is posted.
• The associated bit is set back to 0.
• Since status, severity, and timeStamp are dynamic fields in iocRecord, they follow the same logic.
• The monitor subsystem is notified about the end of synchronized monitors.

Separate discussions are required for the following: 1) primitive fields, 2) string fields, 3) structured fields, and 4) array fields.

• primitive fields - No problems.
• string fields - Memory is an issue.
  • Should there be a way to postpone posting until the monitor is ready to retrieve the value?
  • If so who decides?
• struct fields - Memory is again an issue.
  • Structure with only primitive fields should be posted immediately but perhaps complicated structures should not be.
  • Who decides?
  • How?
• array fields - memory is an issue
  • It should be possible to post small arrays of primitive type.
  • Large arrays should not be posted until monitor system is ready to accept the data.
    • It should also be possible to just notify the client that the array changed but the client must issue a read request to retrieve the data.